A Red Test Is Not Enough: What Evidence Should Block a Merge?

The failure

A check is red. Nobody knows whether:

• the product broke
• the selector drifted
• the preview was not ready
• the runner timed out

So someone clicks merge anyway — or blocks the wrong change for a day.

A binary red/green without evidence is a coin flip wearing a CI badge.

Evidence that should accompany a blocking decision

Minimum useful set:

1. Failed assertion — what was expected vs observed
2. Screenshot — what the browser showed at failure
3. Step / trace — which step died
4. Console context — client errors if present
5. DOM / locator context — what was targeted
6. Merge recommendation — explicit safe to merge vs regression detected
7. Infra vs product classification — timeout/launch failure ≠ checkout math bug

Endia’s public security framing states results as safe to merge or regression detected from executed runs with evidence — verification failure must not silently claim success (/security).

No result must not become safe to merge

| Outcome | Merge interpretation | | --- | --- | | All critical steps passed | Candidate for merge (safe to merge) | | Assertion / product failure | Block (regression detected) | | Runner crash / timeout / missing check | Block or keep pending — never invent pass |

Fail closed means: if the browser journey did not actually complete, do not pretend it did. Treat incomplete verification as not mergeable under branch protection.

Why “just fail the job” is insufficient

CI systems can fail for reasons orthogonal to the customer journey. Without artifacts, on-call engineers cannot tell whether to revert product code or fix the harness.

Evidence turns a fight about feelings into a review of the failed step.

Checklist

• [ ] Require linked evidence on blocking checks.
• [ ] Train the team to read screenshot + assertion before arguing about flakes.
• [ ] Treat missing conclusions as not mergeable under branch protection.
• [ ] Separate harness outages from product regressions in incident language.