Mobile A/B Test Infrastructure: Setup, Stats, Pitfalls

Setting up A/B testing in a mobile app differs fundamentally from web experimentation: code shipping isn't instant, and sample-size math runs straight into store review time and version fragmentation. This article builds, step by step, the four core differences between mobile and web A/B testing, the choice between server-side flags and store-level experiments, sample-size calculation, and experiment-logging discipline — plus the real traps in mobile A/B test infrastructure.

💡 Pro Tip: Before you set up the experiment, write a one-sentence answer to "which metric is my primary decision metric?" and don't change it for the duration of the experiment — chasing a proxy metric is the single most common cause of distorted results in mobile experiments.

Table of Contents

• Four differences between mobile and web A/B testing
• Server-side flag vs. store-level experiment
• Sample size and duration calculation
• Version distribution and cohort contamination
• Early stopping and the multiple-comparisons trap
• Metric selection: how a proxy metric misleads you
• Experiment log and decision record
• FAQ
• How do you set up A/B testing in a mobile app?
• How many users do you need for a significant result?
• How does version distribution break the test result?
• Can a server-side flag and a store-level experiment be used at the same time?
• Why is early stopping risky?
• Update (September 2026)
• Conclusion
• Sources

Four differences between mobile and web A/B testing

On the web, you can flip an experiment variant and roll it out to all traffic within minutes. Mobile behaves differently, because four structural constraints get in the way:

• Rollout delay: a client-side code change has to pass store review, which means the experiment spreads "gradually" instead of "instantly."
• Version fragmentation: your user base is spread across multiple app versions at the same time; an experiment is only meaningful for users running at least a specific minimum version.
• Offline/sync delay: the mobile client can't always pull the variant assignment from the server in real time; a stale cached assignment can leave a user in the wrong cohort.
• Platform store rules: Apple and Google restrict store-page-level experiments (creative/metadata) to their own tooling; these fall under a different regime than in-app behavioral experiments.

These four constraints make it mandatory to clarify "which layer am I experimenting in?" when setting up mobile A/B test infrastructure: in-app behavior, or the store page?

A web assumption that doesn't hold on mobile: "the user opens the same tab every time." A mobile user can close the app and reopen it days later, with an OS update, a network change (wifi to mobile data), or even a device change in between. That makes assignment persistence a more critical engineering problem on mobile: keep the assignment only in memory, and the user can get reassigned when the app is backgrounded or the OS reclaims memory, breaking within-experiment consistency. So the variant assignment should be written to persistent on-device storage (Keychain/UserDefaults, or SharedPreferences/DataStore on Android), or computed with the deterministic hashing method described later — combining both is safest: hash, write the result to the device, and check the on-device record first on the next launch.

Server-side flag vs. store-level experiment

Confusing these two experiment types is the most common design mistake in mobile A/B testing.

Server-side flag experiments remotely toggle a behavior or UI difference while the app is already installed. Firebase Remote Config is a common example: the target metric can be an Analytics event or a conversion funnel; the target user group is defined by chaining multiple criteria with "AND" logic, and secondary metrics like crash-free rate, retention, and revenue track in parallel — while tracking the experiment, the platform flags the variant closest to the target as the "leader."

Store-level experiments use two separate tools on Apple's side, and it's important not to conflate them. Product Page Optimization (PPO) is a genuine A/B test: up to three alternative icon/screenshot/preview combinations (treatments) can be tested against the original page, allocated traffic splits evenly among treatments (e.g., 40% of traffic across two treatments gives each 20% of total traffic, with the original page keeping 60%), and the test runs for 90 days or until manually stopped.

Custom Product Pages (CPP), by contrast, are not an A/B test — they're alternative store pages with their own URL, used to direct traffic from outside the App Store (ads, social media, email); up to 35 active CPPs can be created per app. The Google Play equivalent is Store Listing Experiments. Either way, what's tested isn't the app itself but the store page's metadata and visuals (icon, screenshots, title, description) — part of the funnel before the user even downloads.

Mixing these two types in the same experiment dashboard leaves "which change affected which metric?" unanswered — you need a separate experiment log for each.

In practice there's a third, intermediate layer: build-time A/B testing. Before server-side flag infrastructure exists, some teams ship two separate builds (control/treatment) with a gradual, phased-rollout percentage. This works, but has two downsides: each variant must pass store review separately, compounding the start delay; and stopping the experiment also requires another review, so "kill the bad variant now" doesn't work instantly on mobile the way it does on the web.

Server-side flags eliminate that second problem entirely: stopping a bad experiment just means changing a server condition, not shipping a new build. So prioritize server-side flags for new mobile experiment infrastructure, and reserve store-level experiments for when you're actually testing store page metadata.

Sample size and duration calculation

Sample-size calculation on mobile rests on the same statistical foundation as on the web, but practical constraints (daily active user count, version-adoption speed) make the calculation tighter.

The standard sample-size formula for a two-proportion test:

1n = ( (Z_α/2 + Z_β)^2 × (p1×(1-p1) + p2×(1-p2)) ) / (p1 - p2)^2

Here Z_α/2 corresponds to the significance threshold (typically 1.96 for 95% confidence), Z_β to statistical power (typically 0.84 for 80% power), p1 is the control group's baseline conversion rate, p2 is the expected new rate, and (p1-p2) is the minimum detectable effect you want to catch.

A concrete example: control group conversion is 10% (p1=0.10), expected improvement is 2 points (p2=0.12):

1import math
2 
3z_alpha = 1.96  # 95% confidence
4z_beta = 0.84   # 80% power
5p1, p2 = 0.10, 0.12
6 
7numerator = (z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))
8denominator = (p1 - p2) ** 2
9n_per_group = math.ceil(numerator / denominator)
10print(n_per_group)  # 3834

This requires ~3,834 users per group (~7,668 total). On mobile, the real challenge isn't _reaching_ that number, but in which version mix: a new behavioral experiment is only meaningful for users running at least the minimum version carrying it, so your daily "eligible" pool isn't your entire DAU — just the subset past that version threshold. If adoption is slow, experiment duration is set by the rollout curve, not the sample size.

A simple division estimates duration: days_needed ≈ total_sample_needed / (daily_eligible_users × traffic_split_ratio). If 5,000 eligible-version users open the app daily and you allocate all traffic to the experiment, reaching ~7,668 users takes about 1.5 days; but if the eligible share is only 20% of DAU (the daily eligible pool drops to 1,000), the same math stretches to ~7.7 days. Always fold the adoption curve into the sample-size calculation; otherwise the day the experiment looks "statistically sufficient" it may still represent a narrow version slice, and generalizing to your whole user base is misleading.

Revisit the power analysis as the experiment progresses, not just at the start: if the realized conversion rate (p1) deviates noticeably from your assumption, the original calculation is no longer valid and the planned duration needs recalculating — this isn't "stopping early," since no significance test is performed, only the planning assumption is updated.

Version distribution and cohort contamination

The Android ecosystem strains the assumption of a single "current version" more than the web does: the user base is spread across many OS and app versions at once. This leaks into your experiment in two ways:

• Cohort contamination: if a user updates the app after the experiment has started and enters a different code path, it becomes unclear which variant that user's earlier measurements belonged to. Fix: "stick" the user to the variant they were first assigned (sticky assignment) and don't let a version update change their variant until the experiment ends.
• Stratification by version: if a new behavior can only run on minimum version N and above, apply the "eligible version" filter before randomly splitting into control and treatment groups — otherwise, users on older versions appear to have never entered the experiment pool at all, which skews the sample-size calculation.

1import CryptoKit
2import Foundation
3 
4// Version threshold + sticky assignment skeleton (simplified)
5struct ExperimentGate {
6    let minimumBuildNumber: Int
7    let currentBuildNumber: Int
8 
9    var isEligible: Bool {
10        currentBuildNumber >= minimumBuildNumber
11    }
12}
13 
14func stickyVariant(for userId: String, salt: String) -> String {
15    // The user ID and experiment salt are hashed together with SHA-256. Swift's
16    // built-in hashValue is NOT stable across process runs (Apple: "Hasher is
17    // usually randomly seeded ... will return different values on every new
18    // execution of your program"), so it can't be used for sticky assignment;
19    // SHA-256 output, by contrast, is deterministic and produces the same
20    // result even after the app is relaunched.
21    let hashInput = "\(userId)-\(salt)"
22    let digest = SHA256.hash(data: Data(hashInput.utf8))
23    let bytes = Array(digest)  // SHA256Digest is a Sequence, not a Collection
24    let value = (UInt32(bytes[0]) << 24) | (UInt32(bytes[1]) << 16) | (UInt32(bytes[2]) << 8) | UInt32(bytes[3])
25    let bucket = Int(value % 100)
26    return bucket < 50 ? "control" : "treatment"
27}

Early stopping and the multiple-comparisons trap

A live p-value on the dashboard triggers teams' urge to stop early "once it looks significant." The problem: a significance threshold (α=0.05) calculated for a fixed sample size pushes the false-positive rate much higher than intended when checked daily and stopped "as soon as it looks significant." This is known in the statistics literature as the "repeated significance testing" problem.

The second trap is multiple comparisons: testing several metrics at once (conversion, retention, session length, revenue) and reporting "whichever one comes out significant" makes you mistake a metric that's significant by chance for a real effect. Countermeasures:

• Write the primary metric in one sentence BEFORE the experiment and attach it to the experiment log; don't change it once the experiment is over.
• If you need early stopping, switch to a platform that uses sequential testing instead of the fixed-sample method — these methods are designed to allow continuous monitoring while keeping the false-positive rate under control.
• Label secondary metrics as "exploratory"; don't write their results into the decision log as "proven" without validating them with a new experiment.

Metric selection: how a proxy metric misleads you

A proxy metric is an earlier-observable indicator (e.g., first-session onboarding completion) used in place of the outcome you actually care about (e.g., 30-day retention) because the latter takes too long to measure. The problem: the correlation between proxy and real metric can be different across variants.

A concrete scenario: in an onboarding experiment, treatment might raise "first-session completion" versus control, but if that came from an aggressive notification-permission prompt, those same users might turn off notifications within days and use the app less. The proxy says "improved" while the real metric (retention) may have gotten worse.

• When choosing a proxy metric, validate the proxy-to-real correlation in advance using historical data (outside the experiment); this correlation can't be checked once the experiment has started.
• Always track the long-term metric in parallel, even if you decide and close the experiment based on the proxy metric.
• If the proxy metric "improved" but the mechanism can't be explained (you have no answer to why it improved), delay the decision.

It also helps to make the proxy-metric choice visible at the code level: leaving a note on the query or event definition that computes the metric — which real metric it's a proxy for, and the date it was validated — saves you from having to re-ask "why is this metric here?" months later.

1// Android: version threshold + experiment pool filter (simplified)
2data class ExperimentGate(
3    val minimumVersionCode: Int,
4    val currentVersionCode: Int,
5) {
6    val isEligible: Boolean
7        get() = currentVersionCode >= minimumVersionCode
8}
9 
10fun stickyVariant(userId: String, experimentSalt: String): String {
11    // userId and salt are hashed together; a version update
12    // or app relaunch won't change this result.
13    val input = "$userId-$experimentSalt"
14    val bucket = Math.floorMod(input.hashCode(), 100)
15    return if (bucket < 50) "control" else "treatment"
16}

Experiment log and decision record

As the number of experiments grows, "why did this metric change, and from which experiment?" needs to be answered by a written record, not team memory. A minimal experiment log should contain these fields:

• Experiment name and hypothesis: a one-sentence, measurable hypothesis.
• Primary metric and minimum detectable effect (MDE): fixed before the experiment starts.
• Target cohort and minimum version: which user subset is eligible.
• Start/end date and sample-size calculation: how long it will run, how many users are needed.
• Result and decision: was it significant, was it shipped, and if not, why.

1{
2  "experimentId": "onboarding-single-step-2024q4",
3  "hypothesis": "Reducing onboarding from 3 steps to 1 step increases the first-session completion rate",
4  "primaryMetric": "day1_activation_rate",
5  "minimumDetectableEffect": 0.02,
6  "targetCohort": { "minBuildNumber": 412, "platforms": ["ios", "android"] },
7  "startDate": "2024-11-04",
8  "plannedSampleSizePerGroup": 3834,
9  "status": "running"
10}

This record serves as a reference point against pressure to stop the experiment early (check the planned sample size — if you haven't reached it yet, wait), and it keeps future experiments from repeating the same mistake.

FAQ

How do you set up A/B testing in a mobile app?

First choose the layer you're testing: a server-side flag (like Firebase Remote Config) for in-app behavior, or Apple Product Page Optimization (PPO) / Google Play Store Listing Experiments for the store page. Then fix the primary metric, the target cohort (including minimum version), and the sample size before the experiment starts; assign users with a sticky hash and write the result to the experiment log.

How many users do you need for a significant result?

There's no fixed number; the required sample size depends on the control group's baseline conversion rate, the minimum detectable effect (MDE) you want to catch, and the confidence/power level you choose. Calculate your own scenario with the two-proportion test formula (see the Python example in this article); targeting a small MDE quickly grows your sample-size need.

How does version distribution break the test result?

If the experiment is only meaningful for users running at least a specific minimum version, including older-version users in the experiment pool skews the sample-size calculation; also, if users who update their version during the experiment aren't pinned to their variant (no sticky assignment), the same user can drift from group to group within the experiment and contaminate the measurement.

Can a server-side flag and a store-level experiment be used at the same time?

Yes, but with separate experiment logs: one measures the pre-download funnel (store page), the other measures post-download behavior (in-app). Mixing the two in the same experiment dashboard obscures which metric was affected by which change.

Why is early stopping risky?

A significance threshold calculated for a fixed sample size pushes the false-positive rate higher than intended when used with the habit of checking repeatedly and stopping once it looks significant. If you need continuous monitoring, switch to a method/platform that supports sequential testing.

Update (September 2026)

This article was written with the tools and versions current as of 2024-11-20; three developments have changed in mobile A/B test infrastructure since then:

• Firebase A/B Testing was integrated into Remote Config. The standalone Drafts flow has been deprecated and is being removed on October 31, 2026; existing drafts can now only be viewed/duplicated/deleted, not restarted. Experiments are now set up directly through Remote Config's condition-based targeting flow. (Source: firebase.google.com/docs/ab-testing/faq-and-troubleshooting)
• The scope of Apple Custom Product Pages expanded. As of July 30, 2025, keywords can be assigned to CPPs and they can appear in organic search results (previously reachable only via paid/Apple Ads links); on October 29, 2025, the maximum number of active CPPs per app was raised from 35 to 70 (reminder: a CPP is not an A/B test on its own — it's a targeting-specific store page tool, separate from PPO, which is the actual experimentation tool). (Source: adapty.io/blog/custom-product-pages-app-store)
• Android version fragmentation persists. Per Google's December 2025 distribution data, Android 16 has reached 7.5% of users and Android 15 has reached 19.3%; this confirms that stratifying by version (the "Version distribution and cohort contamination" section in this article) is still a critical control in 2026. (Source: androidheadlines.com, January 2026)

Related posts published later:

• App Store Optimization (ASO) — the ASO context of store-page experiments
• iOS Crash Reporting and Analytics — the analytics infrastructure feeding experiment metrics
• TipKit: iOS Onboarding and Feature Discovery Pattern Guide — patterns used in onboarding experiments
• I Made $1M on the iOS App Store: The Real Strategy and Numbers — how experiment decisions affected business outcomes

Conclusion

Setting up mobile A/B test infrastructure isn't just the web experiment ported to mobile — the rollout delay, version fragmentation, and store rules demand a separate discipline. Keep the server-side flag and the store-level experiment separate, calculate sample size before the experiment starts, make user assignment sticky, and log every experiment in a written decision record — these four habits prevent most of the cohort-contamination and early-stopping mistakes mobile teams fall into most often.

Related articles:

• iOS CI/CD Pipeline — the release pipeline for versions carrying experiments

Sources

• Firebase A/B Testing — FAQ and Troubleshooting — Remote Config integration and the Drafts flow removal timeline.
• Firebase A/B Testing — Overview — experiment setup and condition-based targeting.
• Apple Developer — App Store Product Page Optimization — the official scope and limits of store-page experiments.
• Apple Developer — Custom Product Pages — the official definition of CPPs for the App Store (see the Update section for the current limit).
• Adapty — Custom Product Pages Guide — the CPP count limit and keyword-assignment updates.
• The ASO Project — Google Play Store Listing Experiments Updates — changes to the Play Console experiment control panel.
• Android Headlines — Android Version Distribution 2025-2026 — the fragmentation data that makes cohort stratification necessary.
• Statsig — A/B Testing vs. Feature Flag Tools Comparison — sequential testing and variance-reduction methods.