N Day Metric Dramatically Underestimates App Retention Rates, According to Study

App-phone-illustration.jpgNew data collected from 500m mobile devices by analytics firm Amplitude suggests that one of the most commonly used retention metrics in the app industry is underestimating the percentage of users who return to apps over time by more than a factor of three.

A study based on the data calls into question one of the industrys most popular retention metrics and challenges how much importance companies should place on it as they plan business, product and marketing strategies.

N Day retention measures the percentage of users who return on a specific day post-install. With apps that have very high daily usage patterns, like casual games or social apps, this is still a useful metric, but many apps arent intended for everyday use, so Amplitude instead focused on how many users returned within a given bracket of days.

Previous studies using N Day results have suggested that most apps lose nearly all of their users within 30 days, but Amplitudes data suggests that measuring retention over periods of the users lifecycle paints a very different picture.

According to the data, 23 per cent of mobile app users will return to a new app between days 15 and 30 after first using an app, and 20 per cent between days 31 and 60. Using the N Day metric, however, only eight per cent returned on day 30, and only six per cent on day 60.

“The N Day retention metric might make sense for one per cent of app companies, like Facebook and Snapchat, who have exceptionally large numbers of users who are active daily,” said Spenser Skates, CEO of Amplitude. “But for the other 99 per cent, its a false indicator of how well users are retained over time.

“Companies need to make sure their retention metrics reflect the way people use their product, otherwise theyre making decisions with the wrong data. Only our analytics platforms provides custom bracket retention, allowing companies to customise retention metrics for their unique situation.”

The data also found that app retention rates among Android users are 15 per cent higher across the first seven days compared to iOS users, although for days 14 to 30, the retention rates levelled out to the same across both platforms.

“If your site or app is truly visited on a daily basis, measuring ‘N Day’ retention makes sense,” said John Egan, engineering manager for growth in Pinterests notifications team. “Because this isn’t the case for most, companies need to choose retention metrics that fit their product.

“For example, at Pinterest, our key retention metric is the day one to seven bracket. We also care about retention for the Day 28 to 35 bracket, to track what percentage of our users are still returning five weeks later. The time windows for retention that you care about will vary depending on your product.”