Jeffrey Beir: Metrics (Thu Nov 1, lecture 16)
On the importance of testing your assumption. Scientific method when applied to marketing?

Homework due for today

Legend: : Participation | : Early | : PDF | : Team | : Zipped

  1. Pirate Metrics and Growth Hacking: Growth Hacking is a ‘new profession’. The function is to develop analytical tools and techniques, and a methodology to connect spending on growth with actual results. It’s closely related to ‘pirate metrics’ which is a funny expression cointed to describe a very effective approach. The goal of this work is to introduce you to the fundamental ideas and proponents of it, and ask you to think critically about how these techniques might come into play for your projects:

    Deliverable: This is not obvious: it will require thinking and clear writing. Using the articles or any other source, please explain how Growth Hacking might apply to your team’s product

  2. Think and write one-two sentence answers to these three questions with respect to your product. I would like this to be the result of a real discussion among the team members who should be prepared to present them in class. Here are the questions:
    1. What “problem” are you solving? (in 15 secs or less)
    2. Whose “problem” are you solving? (describe your target user with fewer than 6 attributes)
    3. How are you going to prove whether you are solving this problem for those people? (what “single” thing should or would you do to demonnstrate this, say over the next 6-9 months?)

    Team Deliverable: Your answers to the three questions

  3. Weekly Team Update: Submit your Weekly Progress Updates according to the instructions. Team Deliverable: Weekly Progress Update
Continuing work
  • Continue to work on Stage 2: Will be due on November 13. Decide on and do more out of the building testing: survey, 1-1 interview, landing page, anything else you can think of. Make sure you know and indicate what the hypotheses were that you were testing as well as the methodology and results of the test. Record your work and the results. Remember we are looking for non-trivial amount of outside testing. You will need it for your term report.
Interesting, but not mandatory reading
  1. Pirate Metrics The seminal “Pirate Metrics” presentation by Dave McClure. Video of Pirate Metrics, and the corresponding slide deck: Startup Metrics for Pirates.

Speaker: Jeffrey Beir

We are lucky today to have Jeffrey Beir lead the discussion. His greatest claim to fame (ahem) is that he and I together founded eRoom Technology way back in the previous millenium. Jeffrey was my boss and the CEO. Before that he was a top executive at Lotus Development Corporation, where we first met. After eRoom he went on the being a successful Venture Capitalist. Jeffrey and I are still good friends.

Jeffrey Beir | LinkedIn

Pirate Metrics

  • AARRR!
  • Customer Lifecycle
  • Where did they come from?
  • Stages
    • Acquisition - users come to the site from some channel
    • Activation - users “enjoy” first visit, have a good experiecne
    • Retention - users come back, visit again
    • Referral - users like the products enough to refer others
    • Revenue - users conduct some monetization action
  • What are users?
  • What about two or three sided markets?
  • Conversion: Getting a specific user from one to the next stage
  • See: Example Converstion Metrics

Stages

Acquisition
  • SEO
  • Social networks
  • Organic search
  • Campaigns, contests, etc.
  • Goals: Look for largest volume, lowest cost, best performing (%)
  • Result: Users visit the web site
Activation
  • Landing page
  • Home page
  • Good user experience
  • Features
  • Jobs are done, pain is removed, gain is gained
  • Criteria: Vist 10-30 seconds, view 2-3pages, click 3-5 times, use 1 feature?
  • Result: Users decide later to come back
Retention
  • Emails and alerts
  • Updated content
  • Ongoing need and engagement
  • More jobs need to be done
  • Techniques: lifecycle emails, weekly/monthly status emails, event based emails
  • Result: Users want to invite others
Referral
  • Campaigns or contests encouraging referrals
  • Benefits to referring others
  • Jobs to be done need others
  • Techniques: Only encourage users after they have activation (happy experience)
  • Result: More users are acquired
Revenue
  • Pricing models
  • Visits lead to ad views lead to revenue
  • Subscriptions
  • Partnerships
  • Techniques: Business dependent, no general rules
  • Result: Revenue

Metrics

  • Qualititatve
    • Usability Testing, Session Monitoring
    • Watch what users do, figure out problems and solutions from small samples
  • Quantitative
    • Traffic Analysis, User actions
    • Track and report what users do, usage and conversion %’s for all or empirical sample of users (see cohort analysis)
  • Comparative
    • A/B and multivariate testing
    • Compare what users do in one scenario vs. other
    • Features, Visuals, Colors, Pricaes, etc.

Quantitative & Comparative Measurements

  • Conversion criteria
    • Best performing channels, campaigns, etc (%)
    • Largest volume channels, campaigns, etc (#)
    • Lowest cost channels, campaigns, etc ($)
  • Components
    • Audience segment (regional, age, status, etc)
    • Channel Source (social media, organic, SOE, etc.)
    • Landing Page and call-to-action
    • Copy and graphics
    • Pricing etc.
  • Taking action
    • Consider features that will improve conversion
    • Do a lot of A/B testing and cohort analysis
    • Measure measure measure

What is Cohort Analysis?

  • Just another tool
  • A metrics analysis technique
  • Key to avoiding vanity metrics and devloping actionable metrics
  • Concept: Look of metrics of separate groups of users or customers
  • Surgically measure the impact of changes, pivots: experiments

Picking the right stages is KEY

  • Look at the lifetime a customer and identify key stages
  • Make sure the stages can be measured and are associated with a specific user
  • Make sure that they are meaninful to the business and the engine of growth
  • Measure a baseline with current product
  • Changes can now be measured to see if they had desired impact
  • Identify key stages of relatioship with customer, e.g. AARRR
Step-by-step:
  • Decide what “experiment” you want to look at
  • Decide what “actions” you want to focus on
  • Cohort: all customers who signed up during a certain week (time period)
  • Each customer starts and stays forever in one cohort
  • Create a generalization of the traditional funnel report
  • Note that Cohorts can be non-time based too (e.g. employed vs. unemployed)
  • Note that Cohort analysis is usually for the experiment, over a particular time period

  • Cohort Analysis Example
  • Engine of Growth Example

Growth Hacking

  • It’s a new (cool) job description
  • Combination of marketing, business strategy, metrics, statistics, analysis and coding
  • Uses all the tools we’ve talked about in order drive growth of the startup
    • Measure growth
    • Identify and track actionable metrics
    • Interpret all the data
    • Analyze, form hypotheses
    • Do experiments (marketing, advertising, design, pricing)
    • Measure effects

Next Class