Challenge a check with deliberately flawed example outputs

A check can pass a correct example while also passing important errors. Challenge it with deliberately flawed outputs, not just another successful baseline. This manual exercise uses synthetic document records and a harmless variation. The resulting survivor register shows what the particular checks distinguish; it is not executed product testing or comprehensive proof of correctness.

State the narrow requirement and baseline

For this fictional output, identifiers must appear as A, B, C in that order. Heading whitespace is outside the requirement being checked. Keep the correct baseline list beside the rule so reviewers agree on what counts as meaningful failure.

Two candidate checks are available. The first asks whether the list contains three entries. The second compares the complete ordered identifier list with A, B, C. These are different assertions. A count can be useful without establishing identity or order; the exercise should expose its scope rather than call it useless merely because it is incomplete.

Build a controlled defect catalogue

Create three deliberately wrong outputs, changing one relevant aspect at a time. The omission example contains A, C. The substitution example contains A, A, C, replacing B with another A. The reorder example contains B, A, C.

Also retain a version with changed heading whitespace but the unchanged A, B, C list. That is a harmless variation under this specific requirement. Label the examples before comparing the checks, and preserve why each is harmful or equivalent. Do not classify a formatting change as a defect simply to reward a check that rejects more inputs.

Record what each assertion can detect

The three-entry check passes the baseline and rejects A, C because there are only two entries. It also passes A, A, C and B, A, C: both still contain three entries. Those two harmful variants survive this check. The harmless heading change passes too, as it should within the declared scope.

The ordered-identifier check passes the baseline, rejects all three deliberately wrong lists, and permits the harmless heading variation. Record these results against each example in a shared check-versus-defect register. Here they follow directly from the stipulated assertions; no application test runner was executed.

Strengthen the demonstrated gap and recheck

If the output must preserve identifiers and order, add that observation rather than reporting only that the count is correct. Repeat the entire catalogue after a proposed revision, including the baseline and the equivalent variation. A stronger check that incorrectly rejects the baseline creates a new problem, while rejecting harmless differences can make a check needlessly brittle.

Do not patch only the example that first exposed the gap. Keep the omission, substitution and reorder cases distinct so the new assertion's coverage remains inspectable. If an expected result is disputed, resolve the requirement before marking detection as a success. More rejection is not automatically better discrimination.

Keep surviving uncertainty visible

This catalogue supplies three meaningful error variants and one harmless change. It does not cover every incorrect output, missing record condition, encoding behavior, or document requirement. Detecting all three seeded defects therefore does not establish that the output is universally correct.

Ask another reviewer to explain why the count check misses substitution and reordering, and why whitespace should remain acceptable. If they cannot connect each result to the requirement, revise the register. The useful finish is a precise account of demonstrated detection and survivors, with the checks' limits preserved, rather than a reassuring claim that everything passed.