Every push is an experiment with a written hypothesis. verify is the part that checks it: same targets, same storefronts, before against after, and a verdict you can act on.
The baseline, saved at push
Right before it changes anything, push looks up the rank of every target in every storefront and writes it into the release file. That snapshot is the “before” every later check is measured against.
id: 2026-11-03-lead-with-fan-soundsnote: Lead with fan soundschanges:- en-US subtitle: White Noise & Sleep Aid → Rain, Fan & Baby Sleep Aidbaseline:us: { fan sounds: null, rain sounds: 21, white noise: 8 }
Ranks come from the Appeeky API when APPEEKY_API_KEY is set (both stores), or from Apple's public search for the App Store without it.
Day 7 and day 14
Run verify whenever you like: it only acts on releases that reached day 7 or day 14 and have not been measured for that day yet. Run it daily in CI and each release is checked twice, on time, with nothing to remember.
$ npx @appeeky/spec verify --playbook2026-11-03 day 7: won 3 up · 0 down · 0 top-20 lostup US "fan sounds" not ranked → 27up US "rain sounds" 21 → 18Added 1 line to appeeky/playbook.md
Won, lost, mixed, flat
- won
More targets rose by 3 or more places than fell.
Keep it. Write down why it worked.
- lost
A top-10 target, or two top-20 targets, fell out of the top 20 by 10 or more places.
Consider revert.
- mixed
Some rose, as many or more fell.
Look at which ones, and how popular they are.
- flat
Nothing moved 3 or more places.
Wait for day 14, then call it no effect.
When no target moved, or the release has no targets, the outcomes decide: organic installs and their revenue with attribution, otherwise store conversion, otherwise downloads. The check says which ones decided it.
Against the stores you did not change
Ranks also move on their own: the store's algorithm, the season, a competitor. So verify uses the storefronts your release did not change as a control group. A push to en-US changes the US store; Germany reads de-DE and did not change, but its ranks moved over the same days for every other reason.
$ npx @appeeky/spec verify2026-10-08T10-30-00Z-app-store day 7: woncontrol changed (us) +5.2 places · unchanged (tr) -0.5 · net +5.7up US "baby sleep" 42 → 192026-11-03T09-00-00Z-app-store day 7: mixedcontrol changed (us) +6.1 places · unchanged (de, tr) +5.4 · net +0.7the unchanged storefronts (de, tr) moved as much: the store moved, not the listing
When the changed storefronts gained less than 2 places per target beyond the unchanged ones, a “won” becomes “mixed”. It needs targets in at least two storefronts: add a few for a second country, even one you do not market in.
Beyond ranks
Ranks tell you whether more people find the app. With an Appeeky key, each check also compares the days before and after for whether more of them install and pay:
Conversion and downloads
App Store Connect, Google Play
Revenue and trials
RevenueCat
Cost per install
Apple Search Ads, Meta, TikTok
Ratings
the reviews you pulled
Organic installs and revenue
attribution, if you set it up
Each one is marked better, worse or flat. More ranks with lower conversion is not a win; flat ranks with better conversion can be.
Flat is measured, not guessed: verify reads the four windows before the push and learns how much each number moves on its own. A small app's downloads can swing 30% from one week to the next with nothing changed, and a +20% inside that range is flat:
$ npx @appeeky/spec verifybetter organic installs/day 31 → 38 (+23%) (moves ±11% on its own)flat downloads/day 100 → 120 (+20%) (moves ±41% on its own)
The playbook
With --playbook, every verdict becomes one line in appeeky/playbook.md. Add the why in your own words when it is clear. Over a few months it becomes the most useful file in the repo: what works for this app, and what does not.
# Playbook- 2026-10-12, app-store, en-US keywords: Swap relax for brown noise → day 14 won- 2026-10-20, app-store, en-US subtitle: Lead with meditation → day 7 lost · reverted- 2026-11-03, app-store, en-US subtitle: Lead with fan sounds → day 7 wonWhy: subtitle words rank faster than keyword-field words for us.
What is being measured
Releases and experiments are open loops until their last check. status lists them in one place: what changed, the metric that decides it, the next check and the result so far. It also writes appeeky/loops.md, which push, verify and experiment keep current, so the PR and your agent see it too.
$ npx @appeeky/spec statusdue 2026-11-05-screenshots-v2 experiment · next 2026-11-05decided by conversion · baseline 3.1%measuring 2026-11-03T10-12-00Z-app-store release · next 2026-11-10decided by organic installs/day · day 7 won by outcomes! measured over the same days as2026-10-30T09-00-00Z-app-store: the results mix
It flags three things to act on:
- Late checks. A check two days past its date with no result: the daily run is not running, or a key is missing.
- Overlapping releases. Two pushes to the same store within 14 days are measured over the same days, so neither result is clean. Wait for one to close before the next.
- Too few installs. Under 100 installs in a window, a big change can be noise. Treat it as a hint.
Closing the loop
The playbook is read before every new proposal. Your agent's skill tells it to start there and never repeat a change that lost unless something changed. That is the whole self-learning part: every push teaches the next one.
If a release lost, revert it and push. The revert is a release too, so it gets measured like any other.