Product Analytics Frameworks: Top-Down Analysis

Intro

This post presents a sample set of typical top-down product analytics exercises for a consumer app. This approach is useful for teams that want to use analytics to help align product development priorities or come up with feature recommendations.

The proposed analysis includes three areas: (1) broad overview of the state of the app, (2) description of the app usage ecosystem, and (3) deep dive on a specific product feature. The goal of analysis in each area should be to (a) establish shared understanding of usage patterns, and (b) guide discussion towards important focus areas and actionable improvements.

By the end of the project, you should have experience building basic growth metrics, performing descriptive feature analysis, and producing actionable recommendations for a product team.

Getting started

I’ll reference Poesie, a poetry-reading app I built and launched in early 2018, during some examples. You can download it from the iOS App Store here: https://itunes.apple.com/app/id1332841582.

As you begin using the app, think critically about each step in the user lifecycle: sign-up, first experiences, basic interactions, and repeated use.

Think from the perspective of users coming in with different hopes or expectations for the product: How well does the experience match those expectations? Where does it fail?

Think from the perspective of the development team: What is the mission of this app? Does the feature set (and user experience) match that mission?

Basic analytics

The broadest analytical overview of apps looks at core growth metrics. With any consumer app that hopes for repeated use, growth can be broken down into new user acquisition and ongoing user retention. A basic growth analysis organizes and describes the user base using this framework in order to begin aligning the product team around areas of opportunity.

1. Produce a basic plot of user growth. => How many WAU do we have? What is our growth rate? Any key changes over time or notable moments to investigate?
2. Produce a basic plot of user retention. => What % of new users are still WAU on Day X? At point do you see users dropping out of the app… Day 1? A few weeks? A few months?
3. Produce a growth accounting overview. => What is our w/w WAU retention? Where do you begin to see opportunities for growth? (Retained users, resurrected users, new users) [See linked medium post below.]

Example: https://medium.com/@benbregman/growth-accounting-walkthrough-cbeac63009a3

Ecosystem description

Once we have begun to understand the acquisition rates and user stickiness over time, we can shift this understanding into describing the product itself. What features are being used (or ignored) by users? Which features are contributing the most to retention? A successful ecosystem analysis should describe high-level usage patterns of the app and begin to uncover the most important (or lacking) features.

1. Produce a description of feature usage / user segments. => How many users use each feature? Or, framed differently, what different types of users (defined by feature usage) exist in our user base?
2. Begin investigating the relationship between feature use and retention. => Are there any features that seem highly correlated with retention? (You might want to break apart this analysis for first-time users and long-term users).
3. Begin developing an opinion on top opportunities in the ecosystem. => Can you begin to see any key opportunities? These might focus on improving under-leveraged features, or building new features for key segments of our user base.

Feature deep-dive

Based on your results from the previous analysis, you might begin to care deeply about improving a certain aspect of our app ecosystem — perhaps improving the experience for a certain type of user or improving the usage flow for a certain feature set. What do the current usage patterns for this feature look like, where are the key possibilities for improvement, and what suggestions can you propose?

1. Produce a usage funnel for a specific feature => What different steps need to happen for a successful experience with the feature? Where are the drop-off rates at each step?
2. Identify the greatest opportunities for improving the funnel. => Which of the steps has the most room for improvement? What are some suggestions you have for improving those steps?
3. Estimate the potential impact based on [any of those] improvements. => Suppose we make these changes. Based on your estimates of the usage funnel, and your estimates on the impact of this feature on the overall ecosystem, what do you think will happen when we make those changes?

Conclusion

Tying everything together, a useful output from this entire analysis might look like:

1. Basic analytics: We discovered that most (X%) user drop-off occurs after a few days of use. In other words, if people build a successful usage habit in their first week of use — they will probably stay with the app for a while (X months/years/forever). How can we help people build (and sustain) that successful usage habit?
2. Ecosystem description: We found that users who mark a favorite poem during a given session show X% chance of coming back (vs Y% if they don’t). Perhaps users who leave a session feeling happy with the content available in the app will be more likely to come back and use it in the future. How can we ensure that a user finds a poem that is great for them in any given session?
3. Feature deep-dive: There are few different ways to browse poems. Of those methods, we think that the X feature is extremely under-leveraged: people who use that feature have an X% of finding a favorite poem, but only X% of users find the feature in a given session! We want to think about X, Y, Z changes to make this great feature more prominent.

As you proceed, always remember to keep in mind that the goal of every analysis/recommendation should be to help the app achieve its mission and provide a better user experience. If we have product market fit, a better user experience should manifest in improvements to user retention. Since this is an observational analysis (i.e. no A/B testing), you should also keep in mind the difference between correlation and causation: successfully retained users may display certain usage patterns, but it is not guaranteed we can encourage such usage in other users (let alone expect that usage to lead to retention).

Good luck with your analysis!