How to A/B Test App Store Screenshots

Write a one-change screenshot brief

Suppose a fictional meal-planning app already leads with a screenshot of its weekly plan, but the calendar is small beside a prominent recipe photo. The proposed change is not a new sequence or a new claim: make the weekly calendar the visual focus of that first image. Keep its caption, “Plan your week,” and every other screenshot and listing asset the same. The hypothesis is that visitors who can recognize the weekly plan sooner will be more likely to download the app. This is an illustrative test brief, not a predicted result.

Save both image sets and record which file belongs to the control and which belongs to the variant. Check that the caption still describes what the variant visibly shows. If you use Minimalist Screens while preparing alternate artwork, treat the images as test assets: the store’s experiment tool, not the image kit, assigns traffic and reports results. For asset specifications, consult your store’s requirements before uploading; this test brief is about isolating the creative change.

Choose the outcome before seeing the data. On Apple, compare the treatment’s estimated conversion rate and result against the original page. On Google Play, select the experiment’s target metric at setup; for a download-focused test, choose Unique user install clicks rather than an open or pre-registration metric. The two stores report results differently, so make a separate decision for each listing rather than pooling their numbers.

Run the variant in App Store Connect

First check eligibility. Apple says the app must be live on the App Store with Pre-Order Ready for Distribution or Ready for Distribution status, and the person creating the test needs an Account Holder, Admin, App Manager or Marketing role. Product Page Optimization does not test custom product pages or App Store pages for Apple Watch or iMessage. If your intended destination is one of those pages, this is not the test to run.

Select the app in App Store Connect and open Product Page Optimization. Create a test with one treatment, choose the treatment traffic proportion and select the localizations you intend to test; Apple selects all supported localizations by default. Check its estimated test duration against your plan: the estimate is guidance, while a test runs for at most 90 days unless you stop it earlier. More treatments can make a conclusive result take longer, which is another reason to start with one clear variant.

The treatment starts as a copy of the original page. In its device tabs, add the revised screenshot for the intended device size and leave the other imagery unchanged; sizes you do not edit use the original page’s screenshots or previews. Check the treatment for each localization included in the test before starting it. New treatment metadata must be approved before testing, and App Store Connect prompts you to submit it for review if needed. Then open the test and select Start Test.

To inspect results, open the app’s Analytics tab, then Acquisition → Product Pages and the test name. Apple shows a test there after at least five first-time downloads are attributed to it; five downloads make results visible, not conclusive. Stopping a test is final—you cannot restart that test—so do not stop just because an early estimate looks promising.

Run the variant in Play Console

For a published app, Google Play supports screenshot experiments on default and localized store listings. A default graphics experiment reaches viewers of the default listing language, but viewers of a language with localized graphics are excluded. Use a localized experiment when you specifically want to test that language. This matters for the meal-planning example: choose the listing whose visitors will actually see the revised first image.

In Play Console, go to Grow users → Store presence → Store listings and select Set up in the listing’s Experiment column. If setup is unavailable, Google directs you to fully roll out that listing first. Choose a graphics experiment on the default listing or an experiment on a localized listing, name it, and set the target metric to Unique user install clicks for this download-focused hypothesis.

Use one screenshot variant against the current listing. Play Console lets you adjust the audience percentage, minimum detectable effect and confidence level; its setup page estimates the time and data needed for a significant result. Check those settings before proceeding rather than choosing a tiny audience and expecting a quick answer. Select screenshots as the asset type, provide the variant, review it and save. The documented flow then offers the publishing overview to submit changes for review. Find the experiment afterward through Store listings and read its Results section against the target metric you selected.

Apply a winner, keep a draw, or gather more data

Return to the original hypothesis: did emphasizing the weekly plan improve the chosen outcome enough for the store to support a change? A higher displayed estimate alone is not the same as a supported winner. Interpret the store’s status first, then take the corresponding action.

• Supported winner: If Apple labels the treatment Performing Better, or Google Play reports the variant performing better and recommends applying it, inspect the actual artwork once more and apply it if it still accurately represents the app. Apple’s label requires at least 90% confidence. In App Store Connect, open the test, choose Apply Treatment to Current Product Page and select the versions to update; applying while the test is running stops it. On Google Play, use the action offered in that experiment’s results.
• Worse result or draw: If Apple labels the variant Performing Worse, retain the original. If Google Play says the current listing performed best, keep it. For a Google Play Draw, retaining the control is a reasonable default: this experiment has not supplied a performance reason to replace it. You can design a new, clearer visual hypothesis rather than treating a small apparent lift as a win.
• Insufficient data: If Apple shows Collecting Data, let the test run rather than calling a winner; Likely to be Inconclusive warns that current traffic may not establish a result within its 90-day limit. If Google Play says More data needed, return later. If the test finishes without a useful answer, keep the control and consider a more noticeable single change for a future test. Google Play automatically stops an experiment after six months without applying a variant and returns traffic to the current listing.