Tag Archives: nytimes

AARRR! Pirate metrics vs survey fatigue – what’s a business to do?

18 Mar

Pirate Metrics Cat

I spent much of the past three weeks trying to convince people – friends, random strangers, even people in LinkedIn Groups (I know, I know) – to fill in a survey about their news consumption habits. I cajoled. I implored. I offered dinner, drinks, lunch, babysitting. I received more than one “will I win the new iPad?” query.

And I felt utterly hypocritical throughout.

Because I loathe surveys. I despise forms. I’ve abandoned sign-up processes because the data requirements were too onerous. I angry-click closed those cheery pop-up boxes asking for my feedback. I deeply resent every time I get an email from an online retailer asking, “How did we do?” [In a not unrelated development, I tend to fill in surveys only when I am really, really pissed off about the quality of service received.]

A plague of surveys. An outbreak of requests for feedback.

But why? The NY Times offered the following explanation for the survey epidemic in a piece on March 16:

One reason is that software companies like SurveyGizmo and QuestionPro have made it possible for small companies to create customer surveys at a fraction of the cost of traditional surveys done by established research companies. Businesses of all sizes, desperate to lock in customer loyalty, see surveys as a window into the emotional world of their customers and a database that will offer guidance on how to please them.

I could blame Eric Ries and the rise of the “lean startup” philosophy, one of the central tenets of which is eschewing so-called vanity metrics (like page views and followers) in favour of actionable metrics. I could also blame Dave McClure, who in 2007 unleashed pirate metrics onto the world.

Here’s more from one of McClure’s blog posts in 2007:

The basic concept is based on 5 types of measurements of user behavior:

A: Acquisition – where / what channels do users come from?
A: Activation – what % have a “happy” initial experience?
R: Retention – do they come back & re-visit over time?
R: Referral – do they like it enough to tell their friends?
R: Revenue – can you monetize any of this behavior?
(… otherwise known as “AARRR!”, and thus the “Pirate” designation 😉

But I shall save the blame for one Frederick F. Reicheld and his 2003 Harvard Business Review article, “One Number You Need to Grow“.

Cue dramatic irony:

Dramatic Irony

Reicheld argued that a single survey question – “would you recommend this company to a friend?” – was a company’s best predictor of revenue growth. This so-called “net promoter score” has been embraced by corporate behemoths (including Apple) and lean startups alike.

The third R of McClure’s model – “referral” – is just net promoter score with an eyepatch and a peg-leg.

Cue relevant anecdotal evidence:

Grub Hub hopes I like them (I do)

Companies love this kind of data. But the consumers who provide it? Not so much.

Here’s the NY Times again:

Consumer patience may be fraying under the onslaught. The constant nagging has led to a condition known as survey fatigue and declining response rates over the last decade.

The NY Times said companies are throwing money at the problem:

On their register receipts, stores like Walmart, Petco and Rite Aid include a Web address and an invitation to fill out a survey, with the chance to win a prize. At Staples, the prize is a $5,000 store card.At Staples, the prize [for filling out a survey] is a $5,000 store card.

From AARRR to aargh.

Would You Recommend Us? – Businessweek, 2006
What’s Wrong With the Net Promoter Score – ClickZ, 2009
What Do Companies Like Amazon, Virgin America, Apple and Trader Joe’s Have in Common? – Daily Disruption
The Order of AARRR – Market By Numbers

Elsewhere on Ent!
qualitative feedback is most effective when it’s overwhelmingly negative.
in-home research visits are key to Facebook’s continued success.