Engineering leadership · Android · Release quality

Shipping Android Apps Safely: A Release Validation Strategy for Testing, Rollout and Experiments

When a team asks me “should we A/B test this?”, the real question is usually bigger: how do we know this release is safe, and how do we know it is worth shipping? Those are different questions, and no single kind of test answers both. This article is how I think about it as an engineer, architect and manager: which risks a release carries, which test is the cheapest way to retire each one, who owns it, and what the exit criteria are. A/B testing is one tool in that set, and not the first one you reach for.

1. Start from risk, not from test types

Teams often copy a process: “we do QA, then UAT, then release.” That treats testing as a ritual. I prefer to ask what could go wrong with this change, how likely it is, and how bad it would be. A copy change in a settings screen and a rewrite of the payment flow do not deserve the same process.

2. The map: safe, correct, valuable

I group validation into five layers. Each layer has a bigger audience, more realism and less control than the one before.

3. Layer 1: automated tests the team owns

This is the foundation and it belongs to the engineers who write the code, not to a separate QA phase. I keep the rules simple:

• Many fast unit tests for business logic (use cases, view models, mappers) on the JVM. These run in seconds.
• Fewer integration tests around the boundaries: Room with a real in-memory database, Retrofit against a fake server, WorkManager with a test driver.
• A small set of UI tests for the critical journeys only: sign-in, the main purchase path, onboarding. Flaky end-to-end suites are a tax; keep them few and ruthlessly stable.
• Contract tests between the app and the backend. Most “mobile bugs” I have debugged were an API quietly changing shape.
• Regression test for every escaped bug. It is the best return on testing time you will get.

The practical detail on frameworks, test doubles and the pyramid is in Testing Android apps. The architecture article Testing and proof explains why a testable architecture makes this layer cheap.

4. Layer 2: the non-functional checks people forget

Features that pass every functional test still hurt users. These checks are where I find most of the “why did nobody notice?” incidents.

Assign each check a threshold, not just a task: “cold start p90 under 1.5 s” is a gate, “look at performance” is a wish.

5. Layer 3: humans before the market

Automation checks what we thought of. People find what we did not. This layer is a sequence, each step with a larger and less friendly audience.

Dogfooding

Employees use the in-progress app for real tasks, not scripts. It finds the slow screens, confusing wording and battery drain that only long use reveals. I make it work with three things: an automatic nightly or per-merge build to everyone in the company, a one-gesture feedback button that attaches logs and device info, and a rule that a blocking issue found by dogfooders stops the release train. The caveat: employees know the product too well, so dogfooding is not a substitute for outside users.

Internal testing

This is the gate right after the build exists. A named team runs a short, written smoke checklist on the exact release candidate: install, update from the previous version, sign in, three critical flows, push, deep link. The Play Console internal track suits this since builds reach testers within minutes. The output is one sentence: “build 4123 may continue” or “it may not, because…”

Closed and open beta

Invite real outsiders: partners, power users, one customer segment. You get device and network diversity you cannot buy. Ask them targeted questions, otherwise feedback is noise. Use open beta only when you need scale and can tolerate public exposure of unfinished work.

UAT

User acceptance testing is not about code defects; it is a business decision. The people who asked for the capability, or who represent the customer, run scenarios in their own words against a staging environment with realistic data, and either accept, accept with known issues, or reject. I insist on four things: written scenarios agreed before the build is ready, a named approver, the build number being tested, and a deadline. UAT that drifts for weeks is a sign that requirements were unclear at the start, not that more testing is needed.

6. Layer 4: production safety: staged rollout and observability

No pre-release test matches production. So the last pre-market gate is production itself, entered carefully. On Google Play this is a staged rollout: release to 1%, then 5%, 20%, 50%, 100%, and watch.

• Pre-agree the gates. For example: crash-free users must stay above 99.5%, ANR below the Play “bad behavior” threshold, checkout success not down by more than 1%. Whoever is on call can halt without asking.
• Watch the cohort, not the average. Compare the new version against the old one on the same days. Crashlytics and Android vitals both break results down by version.
• Make halting cheap. Halting a Play rollout stops new installs of that version, but people who already updated stay on it. That is why a remote kill switch (a flag that disables the new behavior) is part of the design, not an extra.
• Plan the rollback. You cannot truly roll back an Android release. You ship a fix forward, or flip a flag. Know which, and how long each takes.

7. Layer 5: proving value: A/B testing and other experiments

Everything so far asks whether the change is safe and correct. It does not tell you whether it is better. That is the job of experiments, and only for changes where the answer is genuinely uncertain and the metric is measurable.

What an A/B test is

An A/B test shows two or more versions to randomly chosen groups of real users at the same time, then compares a pre-chosen metric. Random assignment is the whole point: it removes payday, marketing campaigns, weather and every other confounder, so a gap between groups can be credited to the change.

Why teams do it

• Settle disagreements with data instead of seniority or taste.
• Limit blast radius. A bad idea shown to 5% of users is a cheap lesson.
• Improve metrics in small compounding steps: onboarding completion, conversion, retention.
• Decouple release from rollout. Ship code dark, turn it on remotely. On Android, where a store release is slow and irreversible, this matters more than on the web.

Ways to run experiments

When not to A/B test

• Too little traffic. A small app cannot detect a 1% change in reasonable time. Use usability sessions, interviews or before-and-after.
• Legal, security or correctness changes. You do not test whether to fix a bug.
• Network effects. A chat or marketplace feature affects users outside their group. Randomize by region or time instead.
• The decision is already made. If you will ship B whatever the result, you are performing an A/B test, not running one.

Doing it properly

1. Hypothesis: “If we cut sign-up to two screens, completion rises, because there is less effort.”
2. One primary metric plus guardrails (crash-free users, startup time, uninstall rate).
3. Sample size and duration set up front; at least one full weekly cycle.
4. Sticky assignment per user or installation, and log exposure (the user saw the variant), not only assignment.
5. Don’t peek. Early gaps are mostly noise.
6. Decide, record, clean up. Remove the losing branch and the flag.

8. The architecture that makes this cheap

I judge a codebase by how safely it can ship. These design choices are what turn the process above from heroics into routine.

• Promote one artifact. The AAB that passed internal testing is the one that reaches production. Rebuilding between stages invalidates every earlier gate.
• Flag with discipline. Every flag has an owner and an expiry date. Old flags are code you cannot reason about. Test with the flag on and off.
• Never break a live contract. Some users stay on old versions for months. Support a minimum version, then force-update below it, deliberately.
• Treat analytics as an API. An experiment is only as good as the events behind it. Version them, review them, and test them like code.
• Separate config from code. Anything you might want to change after release (thresholds, copy, endpoints) belongs in remote config with safe defaults.

9. Tooling on Android

A small Remote Config read for an experiment, with the safe default that every experiment needs:

class Flags(private val rc: FirebaseRemoteConfig = Firebase.remoteConfig) {
    init {
        rc.setDefaultsAsync(mapOf(
            "checkout_style" to "control",   // safe fallback if fetch fails
            "new_search_enabled" to false    // also the kill switch
        ))
    }
    fun refresh() = rc.fetchAndActivate()
    val checkoutStyle get() = rc.getString("checkout_style")
    val newSearch get() = rc.getBoolean("new_search_enabled")
}

// Exposure is logged where the user actually sees the variant
analytics.logEvent("checkout_viewed") { param("style", flags.checkoutStyle) }

10. Running it as a team: ownership, gates, metrics

Tools are the easy part. What decides whether this works is whether people know who decides what.

Scale the process to the risk

• Low risk (copy, small UI): CI, internal smoke, rollout at 20% steps.
• Medium (new screen, new API use): add closed beta and per-feature flag, finer rollout.
• High (payments, auth, data migration, regulated features): add UAT with sign-off, upgrade-path and security review, 1% canary with long observation, and a rehearsed kill switch.

Measure the process itself

I track a few numbers to see whether the system is improving: escaped defects per release (found in production), crash-free users per version, time from merge to production, change failure rate (releases needing a hotfix or halt), mean time to recover, and flaky test rate. If lead time grows while escaped defects stay flat, the gates are too heavy. If defects escape while gates are green, the gates test the wrong things.

11. Anti-patterns I keep seeing

12. Takeaways and glossary

• Release validation is a set of tools for different risks. A/B testing answers “which is better?”, not “is it safe?”
• Use five layers: automated tests, non-functional checks, human testing (dogfooding, internal, closed, UAT), staged rollout with observability, and experiments.
• Give every gate an owner, entry and exit criteria, and a time box. Scale the process to the risk of the change.
• Architecture is the lever: flags, a kill switch, compatible APIs, telemetry and one promoted artifact.
• Measure the release process itself and keep improving it.

Links and further reading

• Set up an open, closed or internal test · Staged rollouts · Firebase A/B Testing · Macrobenchmark
• On this site: Testing Android apps · Architecture: testing and proof · Remote Config · App Distribution · Crashlytics · Analytics