Android & iOS hotfix leaderboard

It turns out that by simply watching the App Store and Play Store, you get an interesting glimpse into what’s happening behind the scenes at all sorts of Android and iOS teams across the mobile world. It’s certainly not a full picture, but there are interesting aspects of a team’s mobile release process and maturity that can be picked up on.

For this leaderboard, we follow the top apps in both stores, across all categories, to try to understand their release cycles and how they handle hotfixes. Based on version history and versioning conventions for a given app, we detect and surface its latest hotfix version and the time to recovery (TTR) for that hotfix. We also calculate some averages that paint more of the picture on app level — things like normal release frequency, average number of hotfixes per release, and average TTR.

To make things fun, we rank apps against each other based on their latest hotfix’s TTR. It’s a good measure of a team’s ability to efficiently triage and fix critical bugs in production, and it makes for a more dynamic leaderboard day to day (versus average TTR, say, which changes less often).

We use two open source libraries by Facundo Olano which allow you to fetch publicly accessible data from the App Store and Play Store via certain endpoints they expose. At any given moment data is typically only available for the current live version of each app so, because we need full version histories to perform the kind of analysis we’re doing, we regularly fetch and then aggregate the data ourselves.

Because our analysis of each app’s releases and hotfixes is limited to just a few external signals, please take everything you see here with a grain of salt! The biggest contributors to accuracy for any given app are that app team’s versioning conventions and their consistency around them. To keep things fairly reliable, we require a team to be following semantic versioning (SemVer) — or a close enough approximation — in order to even include their app for consideration here. For the apps you do see on the leaderboard, things like skipped versions and inconsistent usage of patch versions will affect the overall accuracy of our analysis.

There are a couple of reasons why you might not be able to find an app you’d otherwise expect to see here. First, keep in mind that we collect and analyze data for a couple hundred top apps for each major category in the App Store and Play Store — we can’t cover the entire universe of millions of mobile apps! Also, as noted above, we only include apps for analysis if our algorithms have some baseline understanding of their releases, and that generally boils down to whether their versioning follows SemVer (loosely, at least). Finally, if the app in question is quite new or hasn’t had a hotfix in the past couple of weeks (nicely done!), it also won’t show up here.

Mobile app users are notoriously discerning, and any bugs that make their way to production can quickly have a big impact on app usage and revenue. As a result, making sure your team is able to get fixes out the door efficiently is super important. Quite simply, when your time to recovery is short, bugs will be in the hands of users for less time and the overall negative impact can be limited.

Of course, what’s better than a quick TTR for a hotfix? No hotfix in the first place. Bugs are inevitable, but frequent showstoppers that require a hotfix aren’t. Keeping tabs on the average number of hotfixes your team is having to ship per release is a good way to be sure there aren’t fundamental quality issues to address further “left” in your development process.