Bidemi Ajala 🌍

New York

← Thoughts

Smart Test Selection, Seven Products, One Job

Seven products in one Cypress repo, running the full suite on every pull request. I built a selection engine that reads the diff and runs only what it could have broken, and cut our Cypress bill by more than 70%.

4 min read

We have seven products under one Cypress repo, and for a long time every pull request ran the full suite for whichever product you touched. Change one line in a helper, wait forty minutes, watch a few hundred tests confirm what you already knew.

The waste was obvious to everyone. What made it worth fixing was the bill. Cypress Cloud charges by the test result, so a suite that mostly re-proves untouched code is money leaving the account every time anyone pushes a commit.

So in June I built a selection engine. It reads the diff and works out which specs could actually be affected, and CI runs those.

The rules

There are only four, and they're deliberately boring.

A changed spec runs itself. A changed helper runs every spec that imports it, directly or through three other helpers, resolved with dependency-cruiser over the real import graph. A changed global file like cypress.config.js adds the product's @smoke specs on top of whatever else got picked. And if your diff only touches docs or another product's specs, nothing runs at all.

That last one is the whole point, and it's also the one that needs a safety net. The full suite still runs on a schedule for every product, so anything the engine gets wrong shows up within a working day.

In CI

pull requestmerge to mainselectone small job, all productsgit diff origin/main...HEADchanged spec→run that specchanged helper→run every spec that imports it, at any depthchanged global→add the @smoke specs on topnothing matched→skip, no runner startscap 20 specs per product, cut from the tail · machines = clamp(ceil(n / 5), 1, 4)one PR commentwhat got picked for each ofthe seven products, and why,plus anything past the caprun-<product>×71 to 4 machines each,sized by how many specsthe diff actually reachedTHE SAFETY NETcron → <product>-tests.yml → the full suite, twice a dayso anything selection gets wrong surfaces within a working day
One selection job decides the work for all seven products.

The PR gets a single comment saying what was picked and why. If the engine picks nothing for your PR, no runner spins up at all.

The bits that took longest

Ordering matters more than the cap. There's a limit of 20 specs per product, because one unlucky helper change can fan out to the entire suite and you're back where you started. But which 20 you keep is the real decision. Directly changed specs go first, then smoke, then the helper fan-out, so the cap always cuts the tail. Your PR's own specs can never be the ones dropped. Anything past the cap gets listed in the comment so a human can decide whether to force a full run with a label.

A global change doesn't need a full run. My first version ran the entire product suite whenever cypress.config.js changed. That turned out to be a great way to throw a few hundred simultaneous logins at a shared auth endpoint. Now a global change adds the @smoke specs, which prove login and config still work for that tenant, and the rest of the selection carries on as normal.

grep -c . exits 1 when it counts zero. The step that sized the machine matrix ran under bash -e, so a PR that legitimately selected no specs failed the job instead of going green. It counted correctly and printed 0. The exit code killed it anyway.

Never git fetch --depth=1 into a full clone. This one cost me a real afternoon. The workflow checked out full history and then did a shallow fetch of main to refresh it, which writes main's tip into .git/shallow and grafts it as parentless. Git can no longer walk back through history from that tip, even though every object is sitting right there on disk. It worked fine until three PRs merged within three minutes of each other, at which point the grafted tip was no longer reachable from HEAD and git diff origin/main...HEAD died with "no merge base". The fix was deleting --depth=1.

Skip the diff that changes nothing. A comment reworded in a helper used to fan out to every spec importing it. The engine now tokenises both versions with acorn and compares token keys, so a change that only touches comments or whitespace is dropped before selection. It fails open, so if anything can't be parsed the change is treated as real.

What it saved

Cypress costs are down more than 70% since the pilot went in.

That comes from two places. The obvious one is volume, since a typical PR now runs a handful of specs instead of a few hundred. The less obvious one is that PR runs don't record to Cypress Cloud at all any more. Only merges to main do. PR runs split their specs round-robin across the machines instead, which means giving up Cloud's duration-based balancing on those runs. For the money it saves, that trade was easy.

The part I didn't expect was what it did to the feedback loop. When a PR run takes four minutes instead of forty, people stop context-switching away while they wait, and the failures actually get looked at.