High Level Summary A startup needs to state what its measures of growth and success are measure where it is now; conduct experiments to attempt to move reality closer to its target measures.
- Standard accounting doesn’t work yet at the start because there is no model and the model is not working yet.
- Good question to ask: “Are you making your product better”? “How do you know”?
- How do you know that the changes/improvements I see over time in metrics are because of the changes/improvements I’ve made in the product?
- So you do need to decide what your ‘engine of growth is’ – in other words, what do you want to measure and improve upon? Initially it is as simple as I want people to want to buy my product. But then you have to go to how many, in what markets, at what price and so on.
- Basic pattern: use an MVP to get real data on where you stand now; next you make a series of micro changes, either in parallel or in series; and you measure whether the metrics you care about are better. After a while you might decide to pivot if you are not making progress. If you pivot, you measure may decide on a different metric and so you have to reestablish your baseline. Each MVP is a “learning milestone”.
- Always test the riskiest assumptions/hypotheses first! And always measure.
- Next learning milestone is a measurement after having made changes to something. You demonstrate validated learning by showing that the changes actually improved your key metrics.
- You need to know what your target (i.e. how you think your growth engine works) and measure the effectiveness of each experiment in that context. If you find you are not moving fast enough in that direction you need to pivot
- IMVU Experiences
- Don’t always assume that if metrics are not good, solution is adding or fixing features
- Learned to use Cohort analysis to be able to connect changes made to the effects on metrics
- When they realized that their metrics were not going to improve enough they pivoted and started a new baseline
- Metrics: vanity and actionable
- The fact that I have more users than I had yesterday isn’t enough
- You need to know what changes will impact which of your growth variables and which don’t
- You can make changes, but do they really matter to customers?
- Cohort analysis and a/b testing
- Kanban: backlog -> development -> completed -> validated. And limit capacity of each queue
- Metrics
- Actionable: there should be a clear cause and effect. Otherwise its just a vanity metric. You need to be able to tell what you need to do to get more of that result.
- Accessible: means transparent and understandable. No jargon.
- Auditable: Underlying data should be available
Discussion Guide
- What was Chapter 7 about
- What is a “split test”?
- Describe the Grokit story
- What about Lazy registration? How did they test it?
- They did a/b testing and found no difference in full registration
- What is the growth engine?
- Metaphor. Given where you are now and where you want to be, what metrics will have to change and how will they change? For example, is it number of courses registered (Piazza), or the number of students affected? Is it the amount of revenue? You then need to figure out what will make those metrics change that way. You need to think of the states that customers go through and how many need to go from one state to the other. It’s can also be known as the business model.
- Can you distinguish what a vanity metric is from an actionable metric with an example?
- What is Cohort Analysis
- A way to focus on the impact of changes you make to specific subsets of customers or users. Otherwise you can’t tell cause and effect.