Tools for Growth at WordPress.com

I tell anybody that will listen that much of the work I do on growth and analytics for WordPress.com wasn’t really possible until the last couple of years; and that’s almost entirely due to a couple of applications that we use. Of course, some of these have been around for much longer than that. But as a group, these applications (sometimes interoperable) help us to understand our users and their engagement, and the impact that our new features and experiments have.

KISSmetrics

Kissmetrics is a user-level measurement and analytics tool that tells you what things users do on your site, and lets you report on those activities. Internally we’ve been referred to as the ‘Kissmetrics team’, and with good reason: it’s the hub of our data for experimentation. We record dozens of individual key user events, adding up to hundreds of millions every month. It’s also the place that we record properties about users: everything from the device they’re using to the amount of money that they’ve spent on upgrades; it’s how we record details about cohorts that we assign them to.

Screenshot_2_20_13_5_06_AM-3

I think that Kissmetrics solves two big problems that it would be hard for a company of our size to solve ourselves: first, large-scale data storage and retrieval. Second, a great reporting interface that allows us to query and analyze our user data.

Screenshot_2_20_13_5_08_AM

Something of a distant third, but still incredibly useful, is the integration with a handful of other testing tools. Which brings me nicely to…

Optimizely

Optimizely is a front-end a/b testing tool that we use for homepage and signup flow tests. It’s great for experiments with logged-out WordPress.com visitors and really easy to use. If you’re scared of code, the easiest way to use Optimizely is to use its drag and drop editing interface. You drop a line of javascript into your page, and Optimizely pretty much figures out all the rest.

Because we’re not scared of code(!) and prefer to keep the code variations in our own files, we just use Optimizely to set cookies for users and then determine which variation (controlled with javascript) to show a given user. Then Optimizely lets Kissmetrics know which variation of an experiment was shown to a given user.

We rarely use Optimizely’s own results for a test. I tend to much more interested in how a change to the homepage affects a user’s likelihood to do something much further down the line: it could be signing up; publishing their first post; maybe even whether or not they’re still engaged a month later. Optimizely can track events besides clicks on that page, but it’s much easier to have the data all in one place, so we use Kissmetrics to determine the winner instead. Because of this, we created one Kissmetrics report that tracked some of our key metrics for early users and we were able to use this same report to measure the impact of successive tests on the homepage.

Qualaroo (formerly KISSinsights)

If you’ve never seen a little popup survey appear on a website, you’ve not traveled the world wide web very much. Many of them are the work of the team at Qualaroo, which makes it incredibly easy to create these little one-off polls that can help to elicit qualitative data from users at specific points in an application.

Screenshot_2_20_13_4_57_AM

One of our current uses is for asking users immediately after they sign up for a little detail about what they plan to write about. Just like Optimizely, you can have those responses sent to Kissmetrics. We’ve found pretty interesting insights by cohorting users based on their responses to the survey, and then comparing publishing and other engagement metrics across the cohorts.

Recently, this helped us to understand the increase in signups that we see every new year: we’ve always had to guess what kind of users make up that early January bump; this year we had much richer data on the users, and were able to compare publishing and spending rates across the different cohorts.

Mission Control

MC (for short) is where we keep our internal data. It’s the one true source that we rely on for our internal metrics. I like that the app we use for our experiments (Kissmetrics) is different from where the canonical record is kept. It lets us be fast and scrappy, and not worry too much if we record the wrong piece of data every now and again, or decide (as happened around six months ago) to empty all of our data from the account and start over. Evan provided a more in-depth look at how some of the MC data is visualized, if you’re curious.

Google Analytics

The original… but no longer the best. Google Analytics serves a handful of intermittently useful functions for us:

  • Provides great data on audience demographics and technology (both for our publishers and our consumers).
  • Allows us to track trends in visitor referral data.
  • Provides a sanity check for internal and other app measurements.

These days it’s used mostly as a reference tool: the per-user based tracking that Kissmetrics provides us makes Google Analytics’s Goal Tracking functionality seem pretty old!

Excel

The original… and amazingly, still the best. Kissmetrics reports cover lots of important bases, and are particularly good for at-a-glance trends and understanding funnels. But my most frequent workflow involves exporting their People or Power Report data into csv and then manipulating it in Excel. It’s by far the most comprehensive tool for things like Pivot Tables and quick charts. Hey, if it’s good enough for Nate Silver…

Other websites

isvalid.org is a project built by my former colleague, Evan Solomon. It’s a great tool for measuring significance in test results, especially when combined with the bookmarklet which lets you grab your experiment data from the page you’re on and sends it to isvalid.org.

Tableizer lets you paste data from Excel and then turns it into an html table. Automattic is a completely distributed company, so we communicate with writing a lot (on blogs, of course). But however you work, you probably write up your results, so you may find this handy too.

Clicktale might be my favorite thing for gathering qualitative user data. It allows you to record user sessions and watch them back in real time. The user never knows, so they’re acting under entirely normal conditions.

UserTesting.com I prefer in-person user testing, but if that’s not an option, directed online testing with an app like UserTesting.com is really efficient.

We’ve been playing with some SEO tools as well, I’ll maybe write up a separate post on those another time.

Other products are available

Of course, there are alternatives to all of these, and I’m sure there will be new products in the coming months that may well end up displacing some of the ones that I’ve written about here. It’s also likely that over time (and as the company appreciates more the value of the data in product development) that we’ll build more of this capability ourselves.

Skype update fail

Every time I restarted Skype it promised a new release of 6.0 that it would download but then fail to update:

Skype was unable to install the update because the old application can’t be overwritten.

Pretty unhelpful error message.

It was an ownership problem: I finally found the solution today in this thread in the Skype forum:

1. Open terminal

2. sudo bash

3. ls -l /Applications

    check which non root owns most apps (third column)

4. ls -ld /Applications/Skype.app/

    check which user owns the Skype app

5. If Skype.app is not owned by the usual user then

    chown -R whateveruser /Applications/Skype.app/

The WordPress user pyramid

There are very many WordPress sites on the internet. At the last independent count, 17.4% of the web’s top one million sites (according to Alexa) are running WordPress. But that only  accounts only for 174,000 site, and we know of least a couple of hundred times this number. We estimate the current total to be nearly 60 million.

At the recent Pressnomics Conference I gave a presentation in which I talked about the (vast) numbers of WordPress websites, and how I try to think about these when it comes to the business opportunity around WordPress. It’s basically a rough user segmentation:

I don’t have any great numbers about how many sites fall into each group. But some kind of order of magnitude scale starting at 10,000 ‘Big Enterprise’ sites might not be that far off. What I do see is the patterns that work alongside the pyramid. These users have varying needs that frequently scale up or down depending on whether they’re at the top or bottom of the pyramid:

Top Bottom
Price Sensitivity Low High
Security needs High Low
Sales Process Extensive Quick (to non-existent)
Support required Lots Still need some
Payment terms Months Minutes

It’s pretty crazy that WordPress is able to serve all these different segments so well. It can also be a big distraction (especially for businesses): I do not believe that it’s possible for another product to serve all these markets and be a viable business at the same time.

Focusing on just one segment is helpful: you can learn about its needs and build your business around it. For example, if you hate the sales process, you probably shouldn’t be in an Enterprise business; if you’re into providing rock-tight security, you should be focusing on high end customers that are willing to pay for it.

How to run a small country using the P2 theme

At WordCamp San Francisco I gave one of the last presentations of the (long) day, about the P2 theme. We use it a lot at Automattic: every team and project has a blog based on one. We actually have so many that there are more of these blogs than there are people at the company.

Knowing that the audience would be pretty tired, and likely looking forward to the next event (on open bar), I took an unusual approach to this presentation, and read a story.

Learn more about the P2 theme at p2theme.com.