appeeky spec
← Learn

Self-learning ASO · 7 min read

How verify decides if a change worked

The baseline saved at push, the checks on day 7 and 14, what won, lost, mixed and flat mean, and how the playbook fills itself.

Every push is an experiment with a written hypothesis. verify is the part that checks it: same targets, same storefronts, before against after, and a verdict you can act on.

The baseline, saved at push

Right before it changes anything, push looks up the rank of every target in every storefront and writes it into the release file. That snapshot is the “before” every later check is measured against.

appeeky/releases/2026-11-03.yaml
id: 2026-11-03-lead-with-fan-sounds
note: Lead with fan sounds
changes:
- en-US subtitle: White Noise & Sleep Aid → Rain, Fan & Baby Sleep Aid
baseline:
us: { fan sounds: null, rain sounds: 21, white noise: 8 }

Ranks come from the Appeeky API when APPEEKY_API_KEY is set (both stores), or from Apple's public search for the App Store without it.

Day 7 and day 14

#1#10#20#40#60day 7day 14pushfan soundsrain sounds
“fan sounds” was unranked at push and reached #27 by day 7. Ranks move for a week or two after a change, so day 14 is the steadier read.

Run verify whenever you like: it only acts on releases that reached day 7 or day 14 and have not been measured for that day yet. Run it daily in CI and each release is checked twice, on time, with nothing to remember.

zsh
$ npx @appeeky/spec verify --playbook
2026-11-03 day 7: won 3 up · 0 down · 0 top-20 lost
up US "fan sounds" not ranked → 27
up US "rain sounds" 21 → 18
Added 1 line to appeeky/playbook.md

Won, lost, mixed, flat

  • won

    More targets rose by 3 or more places than fell.

    Keep it. Write down why it worked.

  • lost

    A top-10 target, or two top-20 targets, fell out of the top 20 by 10 or more places.

    Consider revert.

  • mixed

    Some rose, as many or more fell.

    Look at which ones, and how popular they are.

  • flat

    Nothing moved 3 or more places.

    Wait for day 14, then call it no effect.

When no target moved, or the release has no targets, the outcomes decide: organic installs and their revenue with attribution, otherwise store conversion, otherwise downloads. The check says which ones decided it.

Against the stores you did not change

Ranks also move on their own: the store's algorithm, the season, a competitor. So verify uses the storefronts your release did not change as a control group. A push to en-US changes the US store; Germany reads de-DE and did not change, but its ranks moved over the same days for every other reason.

zsh
$ npx @appeeky/spec verify
2026-10-08T10-30-00Z-app-store day 7: won
control changed (us) +5.2 places · unchanged (tr) -0.5 · net +5.7
up US "baby sleep" 42 → 19
2026-11-03T09-00-00Z-app-store day 7: mixed
control changed (us) +6.1 places · unchanged (de, tr) +5.4 · net +0.7
the unchanged storefronts (de, tr) moved as much: the store moved, not the listing

When the changed storefronts gained less than 2 places per target beyond the unchanged ones, a “won” becomes “mixed”. It needs targets in at least two storefronts: add a few for a second country, even one you do not market in.

Beyond ranks

Ranks tell you whether more people find the app. With an Appeeky key, each check also compares the days before and after for whether more of them install and pay:

  • Conversion and downloads

    App Store Connect, Google Play

  • Revenue and trials

    RevenueCat

  • Cost per install

    Apple Search Ads, Meta, TikTok

  • Ratings

    the reviews you pulled

  • Organic installs and revenue

    attribution, if you set it up

Each one is marked better, worse or flat. More ranks with lower conversion is not a win; flat ranks with better conversion can be.

Flat is measured, not guessed: verify reads the four windows before the push and learns how much each number moves on its own. A small app's downloads can swing 30% from one week to the next with nothing changed, and a +20% inside that range is flat:

zsh
$ npx @appeeky/spec verify
better organic installs/day 31 → 38 (+23%) (moves ±11% on its own)
flat downloads/day 100 → 120 (+20%) (moves ±41% on its own)

The playbook

With --playbook, every verdict becomes one line in appeeky/playbook.md. Add the why in your own words when it is clear. Over a few months it becomes the most useful file in the repo: what works for this app, and what does not.

appeeky/playbook.md
# Playbook
- 2026-10-12, app-store, en-US keywords: Swap relax for brown noise → day 14 won
- 2026-10-20, app-store, en-US subtitle: Lead with meditation → day 7 lost · reverted
- 2026-11-03, app-store, en-US subtitle: Lead with fan sounds → day 7 won
Why: subtitle words rank faster than keyword-field words for us.

What is being measured

Releases and experiments are open loops until their last check. status lists them in one place: what changed, the metric that decides it, the next check and the result so far. It also writes appeeky/loops.md, which push, verify and experiment keep current, so the PR and your agent see it too.

zsh
$ npx @appeeky/spec status
due 2026-11-05-screenshots-v2 experiment · next 2026-11-05
decided by conversion · baseline 3.1%
measuring 2026-11-03T10-12-00Z-app-store release · next 2026-11-10
decided by organic installs/day · day 7 won by outcomes
! measured over the same days as
2026-10-30T09-00-00Z-app-store: the results mix

It flags three things to act on:

  • Late checks. A check two days past its date with no result: the daily run is not running, or a key is missing.
  • Overlapping releases. Two pushes to the same store within 14 days are measured over the same days, so neither result is clean. Wait for one to close before the next.
  • Too few installs. Under 100 installs in a window, a big change can be noise. Treat it as a hint.

Closing the loop

The playbook is read before every new proposal. Your agent's skill tells it to start there and never repeat a change that lost unless something changed. That is the whole self-learning part: every push teaches the next one.

If a release lost, revert it and push. The revert is a release too, so it gets measured like any other.