AARRR! Pirate metrics vs survey fatigue – what’s a business to do?

18 Mar

Pirate Metrics Cat

I spent much of the past three weeks trying to convince people – friends, random strangers, even people in LinkedIn Groups (I know, I know) – to fill in a survey about their news consumption habits. I cajoled. I implored. I offered dinner, drinks, lunch, babysitting. I received more than one “will I win the new iPad?” query.

And I felt utterly hypocritical throughout.

Because I loathe surveys. I despise forms. I’ve abandoned sign-up processes because the data requirements were too onerous. I angry-click closed those cheery pop-up boxes asking for my feedback. I deeply resent every time I get an email from an online retailer asking, “How did we do?” [In a not unrelated development, I tend to fill in surveys only when I am really, really pissed off about the quality of service received.]

A plague of surveys. An outbreak of requests for feedback.

But why? The NY Times offered the following explanation for the survey epidemic in a piece on March 16:

One reason is that software companies like SurveyGizmo and QuestionPro have made it possible for small companies to create customer surveys at a fraction of the cost of traditional surveys done by established research companies. Businesses of all sizes, desperate to lock in customer loyalty, see surveys as a window into the emotional world of their customers and a database that will offer guidance on how to please them.

I could blame Eric Ries and the rise of the “lean startup” philosophy, one of the central tenets of which is eschewing so-called vanity metrics (like page views and followers) in favour of actionable metrics. I could also blame Dave McClure, who in 2007 unleashed pirate metrics onto the world.

Here’s more from one of McClure’s blog posts in 2007:

The basic concept is based on 5 types of measurements of user behavior:

A: Acquisition – where / what channels do users come from?
A: Activation – what % have a “happy” initial experience?
R: Retention – do they come back & re-visit over time?
R: Referral – do they like it enough to tell their friends?
R: Revenue – can you monetize any of this behavior?
(… otherwise known as “AARRR!”, and thus the “Pirate” designation 😉

But I shall save the blame for one Frederick F. Reicheld and his 2003 Harvard Business Review article, “One Number You Need to Grow“.

Cue dramatic irony:

Dramatic Irony

Reicheld argued that a single survey question – “would you recommend this company to a friend?” – was a company’s best predictor of revenue growth. This so-called “net promoter score” has been embraced by corporate behemoths (including Apple) and lean startups alike.

The third R of McClure’s model – “referral” – is just net promoter score with an eyepatch and a peg-leg.

Cue relevant anecdotal evidence:

Grub Hub hopes I like them (I do)

Companies love this kind of data. But the consumers who provide it? Not so much.

Here’s the NY Times again:

Consumer patience may be fraying under the onslaught. The constant nagging has led to a condition known as survey fatigue and declining response rates over the last decade.

The NY Times said companies are throwing money at the problem:

On their register receipts, stores like Walmart, Petco and Rite Aid include a Web address and an invitation to fill out a survey, with the chance to win a prize. At Staples, the prize is a $5,000 store card.At Staples, the prize [for filling out a survey] is a $5,000 store card.

From AARRR to aargh.

Would You Recommend Us? – Businessweek, 2006
What’s Wrong With the Net Promoter Score – ClickZ, 2009
What Do Companies Like Amazon, Virgin America, Apple and Trader Joe’s Have in Common? – Daily Disruption
The Order of AARRR – Market By Numbers

Elsewhere on Ent!
qualitative feedback is most effective when it’s overwhelmingly negative.
in-home research visits are key to Facebook’s continued success.


3 Responses to “AARRR! Pirate metrics vs survey fatigue – what’s a business to do?”

  1. dopeambition March 18, 2012 at 22:38 #

    I just thought about this the other day as I opted out of taking a “how are we doing?” survey for an app I actually LIKE. I think Reicheld’s argument in favor of the simple, single single survey question “would you recommend this company to a friend?” is most likely to garner a response. Our brains only have to process two options: yes or no. This is sad, especially when we’re on the asking end and want more in-depth responses than yes or no.

    I’m willing to give away my shitty iPad1 for a well-rounded survey.

  2. Ashley Milne-Tyte March 21, 2012 at 14:09 #

    Great read. I was horrified when Barbara was talking to us about market research and mentioned surveys because I loathe them too and couldn’t bear to inflict one on anyone I know. I am inundated with the things and increasingly hit ‘delete’ when they arrive. The main reason I hate them is that the questions are stupid/unanswerable. They are far too general and do not encapsulate my experience accurately. The only way the store or whatever it is will know how I really feel is if they leave a box at the end for comments, in which I generally blast them. But I resent filling in a survey by clicking on option after option that doesn’t speak to what I actually went through. If there’s no way to accurately state my experience, I’d rather opt out.

  3. Denise Cheng March 22, 2012 at 17:42 #

    I agree with both Brianne and Ashley, and I also have to wonder whether they’re that much more solid than those television and online polls (The Rapidian is guilty of this, too) but without the internal acknowledgement that only the most fanatical users tend to respond (cue: anger or immense satisfaction).

Comments are closed.

%d bloggers like this: