A/B Test or Just Ship? A Framework for App Experiments
Not every change needs an A/B test. A framework for deciding when to test rigorously, when to ship and watch, and when to trust your gut.
A/B testing is a powerful tool that's also routinely misapplied. Indie devs either over-test (test every button color, never reach significance, ship slowly) or under-test (ship blindly and miss obvious improvements).
This is the framework for deciding when to A/B test, when to ship-and-watch, and when to trust your gut.
The three decision modes
Mode 1: A/B test rigorously
- Change has potentially big impact (10%+ on a critical metric).
- You have enough volume for significance (1k+ samples per variant).
- You can isolate the change (one variable).
- The cost of being wrong is high.
Mode 2: Ship and watch
- Change is plausible improvement based on data / first principles.
- Risk is low (easy to revert).
- Volume is too small for proper A/B testing.
- You can monitor post-ship metrics.
Mode 3: Just ship (no test)
- Change is obvious improvement (a typo fix, a clear UI bug).
- Cost of testing exceeds value.
- No interesting metric to test against.
Most indie devs default to Mode 2 / Mode 3, which is usually correct.
When to A/B test (Mode 1)
Conditions
- Volume: ≥1,000 conversions per variant for meaningful significance.
- Time: ≥14 days to capture weekly patterns.
- Effect size: hypothesized lift of 10%+ on the metric.
- Isolated variable: only one thing changes between variants.
- Measurable: clear before/after metric.
High-value A/B tests for mobile apps
- Paywall variant (hard vs soft).
- Trial length (3 vs 7 vs 14 days).
- Pricing tier ($4.99 vs $9.99).
- First screenshot (lifestyle vs UI vs outcome).
- Onboarding length (3 steps vs 7 steps).
- App icon variant (background color, character).
- Subtitle phrasing (outcome vs feature).
These are tests worth running rigorously.
When to ship and watch (Mode 2)
Conditions
- Plausible improvement based on user feedback or data.
- Low risk — easy to revert if it breaks.
- Volume too small for proper A/B testing.
- Time-sensitive — can't wait 14 days.
Examples
- Adding a new feature based on user requests.
- Improving error messaging.
- Refactoring onboarding flow based on drop-off data.
- Updating copy based on feedback patterns.
Methodology
- Ship the change.
- Monitor key metrics for 7-14 days.
- Compare to pre-ship baseline (same period last week/month).
- If metric improves: keep.
- If metric drops: revert or iterate.
This is most indie product work. A/B testing every change would slow you to a crawl.
When to just ship (Mode 3)
Conditions
- Pure quality improvement (bug fix, typo).
- Following platform best practices (compliance, accessibility).
- Internal-facing change (analytics events, code refactor).
- Obvious win with no plausible downside.
Examples
- Fix a crash.
- Add accessibility labels.
- Improve performance.
- Fix typos.
- Tag analytics events.
No need to test. Just ship.
The trap of over-testing
Over-testing burns time and slows progress:
- 2 weeks per test × 10 tests/year = 20 weeks/year on testing.
- Most tests show "no significant difference" — feels like wasted time.
- Even meaningful tests need samples; indie apps often lack samples.
The right framing: A/B testing is for high-stakes decisions where you can't afford to be wrong.
The trap of under-testing
Under-testing produces compounding mistakes:
- Ship a paywall variant that loses 30% conversion.
- Doesn't notice for 2 months.
- Cost: $30k+ in lost revenue.
For high-stakes decisions (pricing, paywall placement), test rigorously.
Sample size math
Quick estimator for A/B tests:
Required sample size per variant ≈ 16 × (1 - p) / p × (1/MDE)²
where:
p = baseline conversion rate
MDE = minimum detectable effect (e.g., 0.10 for 10%)
For 5% baseline conversion and 10% relative MDE:
≈ 16 × 0.95 / 0.05 × 100 = 30,400 per variant
For most indie apps, this means tests on conversion need months of runtime, which is impractical. Test bigger effects (20%+ MDE), accept lower confidence, or use Mode 2 instead.
Frequentist vs Bayesian
Some A/B testing tools use Bayesian methods (faster to conclude) vs frequentist (more conservative). Both are valid; both can be misinterpreted.
For indie purposes:
- Use whichever your A/B testing tool offers by default.
- Be skeptical of "winning" variants with <100 conversions per group.
- Trust 14+ day tests more than 3-day tests.
What to do at indie scale
Most indie apps have insufficient volume for proper A/B testing. Strategies:
Strategy 1: Test the biggest decisions only
Test paywall + pricing + first screenshot. Skip everything else.
Strategy 2: Pre/post comparison
Ship a change, compare 4 weeks before vs 4 weeks after. Not rigorous but better than nothing.
Strategy 3: Direction testing
Don't try to detect 5% lifts. Test changes you expect to move metrics 30%+. Smaller changes either ship-and-watch or skip testing.
Strategy 4: Combine signals
A/B test + qualitative feedback + analytics drill-down combined gives more signal than any single source.
A/B testing tools
Built into stores (free)
- App Store Connect: Product Page Optimization.
- Google Play Console: Store Listing Experiments.
Use these for icon / screenshot / video tests.
In-app testing
- Firebase A/B Testing (free, integrated with Remote Config).
- Statsig (free tier).
- PostHog (open source, included).
- RevenueCat / Adapty / Apphud paywall A/B testing (built-in).
Don't pay for testing tools at indie scale
Built-in tools are sufficient until you have substantial volume.
Common mistakes
- Testing too many variables at once. Can't isolate the cause.
- Stopping tests early. Below required sample = unreliable conclusion.
- Calling weak winners. A 3% lift with 200 samples isn't a real win.
- Testing trivial changes. Burns time; learns nothing.
- Never testing big decisions. Misses easy wins.
- Over-trusting frequentist p-values. Multiple-test problem.
- Ignoring qualitative data. Numbers + user feedback together.
A pragmatic indie cadence
For most indie devs, a working rhythm:
- Test rigorously: 4-6 times per year (the big decisions).
- Ship and watch: 20-40 times per year (the normal product work).
- Just ship: continuously (the obvious wins).
Run an audit
A/B testing compounds with good baseline data. Run free ASO audit regularly to see the absolute landscape — testing only matters if your baseline is healthy.
Related reading
- App Store Conversion Rate Optimization
- AB Testing App Store Listings
- App Store Connect AB Testing Guide 2026
- Mobile App Feature Flagging for Indie Developers
- Mobile Analytics Tools Comparison
- Freemium Conversion Rate Optimization
Try the tools
Ready to Optimize Your App Store Listing?
Try our free ASO tools — no signup required.