Data: curious

In an HBR article from last year, Shvetank Shah, Andrew Horne, and Jaime Capellá discuss the idea of the Informed Skeptic:

We evaluated 5,000 employees at 22 global companies and sorted them into three groups. “Unquestioning empiricists” trust analysis over judgment, and “visceral decision makers” go exclusively with their gut. “Informed skeptics”—the employees best equipped to make good decisions—effectively balance judgment and analysis, possess strong analytic skills, and listen to others’ opinions but are willing to dissent. They’re the kind of data-savvy workers every company should try to cultivate.

I like the concept a lot, I’m less keen on the name (and not just because Brits spell ‘skeptical’ with a ‘c’). Terms like ‘skeptical’, ‘cynical’, ‘doubtful’ and ‘suspicious’ are bad ways to describe valuable employees, and imply mistrustfulness of others’ data. Instead, when people ask me what qualities they should look for in a Growth Hacker, I tell them that the number one requirement should be being Data Curious.

Data Curious is my shorthand for someone who has an intrinsic need to understand a data point. They understand that a number on its own very rarely tells the full story. Rather than simply being skeptical, the Data Curious person looks for more information and context. It helps her to understand, and increase her trust in, that number.

In question two of their quiz to find out which of their data-oriented groups you would fit into, Shah, Horne and Capellá ask the following question:

Reviewing recent sales figures, you notice a spike in a division that’s been struggling. You:

1. Look up some data, run some numbers, and make a couple of calls to figure out why sales are up
2. Are suspicious about the increase.
3. Congratulate the division manager for turning things around.

Can you figure out which is the Data Curious (or Informed Skeptic)? It’s the one that responds with (1) of course: they quietly check a few things themselves and reassure themselves that the data makes sense. The Visceral Decision Maker (2) simply stays suspicious and the Unquestioning Empiricist (3) just accepts everything at face value.

Here’s how the mind of the Data Curious person works when she sees the quarterly sales number compared to forecast:

• A little above or below forecast: one raised eyebrow.Does the number make sense in context: is it consistent with its recent trend; consistent with the other products or divisions; consistent with the overall market?
• Significantly above/below forecast: furrowed brow.Something went wrong. Was there a problem with the forecast, or the actual sales figure (or both)?
• Spot on forecast: two raised eyebrows.Some things are too good to be true!

The best science publications publish the results of studies that are both peer-reviewed and reproduced. It’s not skepticism, it’s just part of a good expriment process. It comes naturally to the Data Curious.

KISSmetrics

Kissmetrics is a user-level measurement and analytics tool that tells you what things users do on your site, and lets you report on those activities. Internally we’ve been referred to as the ‘Kissmetrics team’, and with good reason: it’s the hub of our data for experimentation. We record dozens of individual key user events, adding up to hundreds of millions every month. It’s also the place that we record properties about users: everything from the device they’re using to the amount of money that they’ve spent on upgrades; it’s how we record details about cohorts that we assign them to.

I think that Kissmetrics solves two big problems that it would be hard for a company of our size to solve ourselves: first, large-scale data storage and retrieval. Second, a great reporting interface that allows us to query and analyze our user data.

Something of a distant third, but still incredibly useful, is the integration with a handful of other testing tools. Which brings me nicely to…

Optimizely

Optimizely is a front-end a/b testing tool that we use for homepage and signup flow tests. It’s great for experiments with logged-out WordPress.com visitors and really easy to use. If you’re scared of code, the easiest way to use Optimizely is to use its drag and drop editing interface. You drop a line of javascript into your page, and Optimizely pretty much figures out all the rest.

Because we’re not scared of code(!) and prefer to keep the code variations in our own files, we just use Optimizely to set cookies for users and then determine which variation (controlled with javascript) to show a given user. Then Optimizely lets Kissmetrics know which variation of an experiment was shown to a given user.

We rarely use Optimizely’s own results for a test. I tend to much more interested in how a change to the homepage affects a user’s likelihood to do something much further down the line: it could be signing up; publishing their first post; maybe even whether or not they’re still engaged a month later. Optimizely can track events besides clicks on that page, but it’s much easier to have the data all in one place, so we use Kissmetrics to determine the winner instead. Because of this, we created one Kissmetrics report that tracked some of our key metrics for early users and we were able to use this same report to measure the impact of successive tests on the homepage.

Qualaroo (formerly KISSinsights)

If you’ve never seen a little popup survey appear on a website, you’ve not traveled the world wide web very much. Many of them are the work of the team at Qualaroo, which makes it incredibly easy to create these little one-off polls that can help to elicit qualitative data from users at specific points in an application.

One of our current uses is for asking users immediately after they sign up for a little detail about what they plan to write about. Just like Optimizely, you can have those responses sent to Kissmetrics. We’ve found pretty interesting insights by cohorting users based on their responses to the survey, and then comparing publishing and other engagement metrics across the cohorts.

Recently, this helped us to understand the increase in signups that we see every new year: we’ve always had to guess what kind of users make up that early January bump; this year we had much richer data on the users, and were able to compare publishing and spending rates across the different cohorts.

Mission Control

MC (for short) is where we keep our internal data. It’s the one true source that we rely on for our internal metrics. I like that the app we use for our experiments (Kissmetrics) is different from where the canonical record is kept. It lets us be fast and scrappy, and not worry too much if we record the wrong piece of data every now and again, or decide (as happened around six months ago) to empty all of our data from the account and start over. Evan provided a more in-depth look at how some of the MC data is visualized, if you’re curious.

Google Analytics

The original… but no longer the best. Google Analytics serves a handful of intermittently useful functions for us:

• Provides great data on audience demographics and technology (both for our publishers and our consumers).
• Allows us to track trends in visitor referral data.
• Provides a sanity check for internal and other app measurements.

These days it’s used mostly as a reference tool: the per-user based tracking that Kissmetrics provides us makes Google Analytics’s Goal Tracking functionality seem pretty old!

Excel

The original… and amazingly, still the best. Kissmetrics reports cover lots of important bases, and are particularly good for at-a-glance trends and understanding funnels. But my most frequent workflow involves exporting their People or Power Report data into csv and then manipulating it in Excel. It’s by far the most comprehensive tool for things like Pivot Tables and quick charts. Hey, if it’s good enough for Nate Silver…

Other websites

isvalid.org is a project built by my former colleague, Evan Solomon. It’s a great tool for measuring significance in test results, especially when combined with the bookmarklet which lets you grab your experiment data from the page you’re on and sends it to isvalid.org.

Tableizer lets you paste data from Excel and then turns it into an html table. Automattic is a completely distributed company, so we communicate with writing a lot (on blogs, of course). But however you work, you probably write up your results, so you may find this handy too.

Clicktale might be my favorite thing for gathering qualitative user data. It allows you to record user sessions and watch them back in real time. The user never knows, so they’re acting under entirely normal conditions.

UserTesting.com I prefer in-person user testing, but if that’s not an option, directed online testing with an app like UserTesting.com is really efficient.

We’ve been playing with some SEO tools as well, I’ll maybe write up a separate post on those another time.

Other products are available

Of course, there are alternatives to all of these, and I’m sure there will be new products in the coming months that may well end up displacing some of the ones that I’ve written about here. It’s also likely that over time (and as the company appreciates more the value of the data in product development) that we’ll build more of this capability ourselves.

A great post about how to think about pricing, the demand curve, and a user’s willingness to pay.

Skype was unable to install the update because the old application can’t be overwritten.

Pretty unhelpful error message.

It was an ownership problem: I finally found the solution today in this thread in the Skype forum:

1. Open terminal

2. sudo bash

3. ls -l /Applications

    check which non root owns most apps (third column)

4. ls -ld /Applications/Skype.app/

    check which user owns the Skype app

5. If Skype.app is not owned by the usual user then

    chown -R whateveruser /Applications/Skype.app/

At the recent Pressnomics Conference I gave a presentation in which I talked about the (vast) numbers of WordPress websites, and how I try to think about these when it comes to the business opportunity around WordPress. It’s basically a rough user segmentation:

I don’t have any great numbers about how many sites fall into each group. But some kind of order of magnitude scale starting at 10,000 ‘Big Enterprise’ sites might not be that far off. What I do see is the patterns that work alongside the pyramid. These users have varying needs that frequently scale up or down depending on whether they’re at the top or bottom of the pyramid:

It’s pretty crazy that WordPress is able to serve all these different segments so well. It can also be a big distraction (especially for businesses): I do not believe that it’s possible for another product to serve all these markets and be a viable business at the same time.

Focusing on just one segment is helpful: you can learn about its needs and build your business around it. For example, if you hate the sales process, you probably shouldn’t be in an Enterprise business; if you’re into providing rock-tight security, you should be focusing on high end customers that are willing to pay for it.

Knowing that the audience would be pretty tired, and likely looking forward to the next event (on open bar), I took an unusual approach to this presentation, and read a story.

Learn more about the P2 theme at p2theme.com.

Taken together, the conversations demonstrate something weird going on… Silicon Valley people in the know have no fucking clue what to make of Twitter. What’s more, they haven’t for years.

http://pandodaily.com/2012/09/04/will-twitters-uncanny-luck-ever-run-out/

Starring alongside Live Theme Previews, Custom Headers and Better Captions is a very good looking golden retriever… who just happens to be our very own Darcy.

We adopted Darcy from Pets Lifeline in Sonoma County, CA when she was a little over a year old (she’s now three). She had been a puppy with a family that had kept her in a back yard and never let her see other dogs. Thankfully they eventually realised that they couldn’t keep her like that forever and we were lucky to be looking to adopt a dog at the same time.

Pete and Evan with Darcy and Stella on Blithedale Summit

Evan and Chelsea are particularly frequent visitors. The picture above (and the one on the welcome screen taken in the back of a car) is from early February, when they brought Stella for a hike with me and Darcy on Blithedale Summit and Chelsea (as always) had her camera at the ready.

Chelsea’s been looking after Darcy for the last couple of weeks while I’ve been in Europe and has some fabulous new pictures of her:

Thanks to her new internet stardom (WordPress 3.4 has already been downloaded nearly 2 million times), Darcy has a whole load of new fans now, and has been (unknowingly) receiving appreciation from lots of strangers:

I like the dog on the new WordPress welcome screen —
Rob Cubbon (@RobCubbon) June 13, 2012

And I loved the caption added to the car picture in this post. It seems some people figured Darcy out straight away!

If you want to keep up with Darcy’s antics, then you should probably follow the (misleadingly named) Daily Darcy.

The project has a pretty cool interactive map on which you can see the work they’re doing, as well as archived webcams (click through a few days over last weekend during the demolition).

In the photo gallery I found this picture. The tunnel (bottom left) is new, but before it could be used (and accessible) they needed to remove the elevated roadway just above it. That all happened in just one weekend. There’s a lot of rubble there now!

Bank on openness: [...] the best example may be nearly invisible, even to a dedicated user of the Internet: blogging platforms. Less than a decade ago there were a multitude of services competing for the emerging legion of bloggers: Movable Type, TypePad, Blogger, WordPress. Today, only the last two remain relevant, and of these, the small, scrappy WordPress is the champ. WordPress prevailed for several reasons. For one, it was free and fantastically easy to install, allowing an aspiring blogger (or blogging company) to get off the ground in hours. Users who wanted a more robust design or additional features could turn to a community of fellow users who had created tools to meet their own needs. And that community didn’t just use WordPress—many made money on it by selling their designs and plug-ins. Their investment of time and resources emboldened others, and soon the WordPress community was stronger than any top-down business model forged inside the walls of their competition.