Testing Habits

1. Access time-stamped data to see how people interact with the app

• Basing on one metric is not a great way to keep metrics

2. Calculate the percentage of habituated users

• You have to decide what a habituated user looks like
• Set a reasonable threshold and guesstimate
• A good metric is 20% or over that are habituated users

3. Find a cohort of users

• New product features will be released to one cohort

4. Do a cohort analysis

• Tells us if we should keep the feature or not a. Capture your baseline at least 16% of seattle b. Form a hypothesis Make a change to your product design and see if the level of habituated users went up or down c. Define the cohort or target d. Compare the data If new app users are higher than baseline data, you can keep the feature
• Evernote percentage of users upgrading to paid version was low at first, but then after year two and then it signfiiciantly increased
  • Reason for the smile graph? Users made an investment, left, missed the investment and came back

Key Learning

1. Test quickly early and often
2. Enagement before revenue

Testing Hooks: Amazon Echo

Continue to buy and test their hooks to make it easier to buy from Amazon.

Alexa helped you shop across the room. Started as an overqualified speaker.

1. Establish percentage of habitual users

2. Use cohort analysis

Used cohort testing by buying skets and prototypes to a long-standing pool of more than 100 customers. Persuade customers to place devices in prime locations in their home. THey collect a lot of data. It helped them design the echo.

They are so commited to cohort testing they actually have a whole website dedicated to it. 20 million own more than one echo

Key Leanrings

1. Even a successful product can be habit tested to increase customer use
2. Monetization comes through engagement.

Leads to increased engagement and revenue in ecosystem