Your Test Did Not Fail Because the Feature Broke

The failure

CI is red. The feature demo still works when a human clicks through.

Two different problems share one symptom:

1. Functional regression — the product behavior changed.
2. Selector drift — the product still works; the locator no longer matches.

If your process treats every red E2E as “the app is broken,” you will either ignore real regressions or drown in maintenance noise.

Concrete cases

| What changed | Product OK? | Test OK? | Correct interpretation | | --- | --- | --- | --- | | Button label “Pay” → “Place order”; role/name still unique | Yes | Often no | Locator drift | | Checkout total wrong after tax change | No | Should fail on assertion | Functional regression | | DOM reordered; stable test id | Yes | Yes | Good locator strategy | | Auto-accept swaps to a similar-looking wrong control | Risky | False green | Unsafe healing |

Test maintenance vs product failure

Maintenance asks: Did our automation still point at the intended control?

Product failure asks: Did the user-visible outcome still occur?

Evidence should help separate them: screenshot of the page, the selector that failed, the assertion that failed, console errors, and whether a candidate locator was merely suggested.

Constrained healing (outcome level)

When a primary selector fails, Endia can propose alternative locators so a human (or a tightly scoped policy) can decide what to do next. Docs: selector healing.

What that means in practice:

• Default posture is suggest alternatives — not silently rewrite the test.
• Durable locator updates require an explicit decision path.
• URL assertions and non-locator steps are not rewritten by selector healing.
• The goal is recoverable maintenance signal, not “AI magically fixes tests.”

That is policy-gated locator repair, not generative test rewriting.

Unsafe healing looks like success

Auto-accepting a weak match can turn a broken flow into a green check against the wrong button. That is worse than a red check.

If healing goes beyond suggestions, require:

• an evidence trail of what changed
• human approval for durable locator updates
• review that the new target still matches the intended control

Checklist

• [ ] On red checks, ask: assertion failure or locator failure?
• [ ] Prefer stable semantic locators (role/name, test ids) before brittle structural CSS.
• [ ] Keep auto-accept off or tightly scoped until you trust the evidence trail.
• [ ] Never equate “healed” with “product verified” without reviewing the new target.