A test you cannot fail is a demo

Here is a blunt test for any experiment on your roadmap.

Write down, before it runs, the result that would make you not ship it.

If you cannot, it is not a test. It is a demo with a control group, and you are going to ship the feature either way. That is fine as a decision. It is expensive as a delusion, because you will file it as evidence.

What a demo looks like wearing a test's clothes

Five tells. Most teams have at least two of them running right now.

• Success criteria written after the data arrives. The number comes in, and then the room decides what would have counted as good.
• "Directionally positive." The most expensive two words in product. It means the result did not clear the bar and nobody wants to say so.
• No control. You shipped it to everyone and compared to last month, which also contains a pricing change, a holiday, and a marketing push.
• The metric was chosen because it moved. Three metrics were watched, one went up, that one is now the headline.
• "Let's see how it goes." A way of guaranteeing that any result can be read as encouraging.

None of these are dishonest, exactly. They are what a team does when the feature was already decided and the test is there to provide cover.

The test card, written first

The fix is small and slightly annoying, which is why it works. Four lines, written before anything ships:

• We believe that [change] will cause [effect] for [segment].
• To verify we will [what you actually build and expose].
• And measure [the one primary metric].
• We are right if [a number, and a threshold].

The first three lines are admin. The fourth is the entire discipline, and it is the one that gets left blank.

At elyps, the criterion on the onboarding test card was written as "overall onboarding churn decreases by at least 10%". At UX Pilot, the bar on the activation test was +10% before we started. Both cleared it comfortably, which is only a meaningful sentence because the bar existed first. Had the elyps result come in at 4%, we would have known it was a miss instead of finding a way to describe 4% as promising.

Why teams avoid the falsifiable version

Not because they do not know how. Because a documented no is politically more expensive than a vague yes.

Somebody promised the feature in a planning session. Somebody put it on a board slide. Three engineers spent a sprint on it. In that context, a threshold written down in advance is a device for making a team publicly wrong, and people are reasonable about avoiding those.

Which means this is only partly a method problem. Teams write falsifiable criteria when a no is treated as information rather than as a failure, and that is set by whoever runs the review, not by whoever writes the test card. If saying "we don't know that yet" is career-safe on your team, the test cards get sharper on their own. That is the same soil a culture of experimentation needs, and it is why the culture question comes before the tooling question.

Correlations are not results

The trap that catches the most rigorous teams is the strong correlation.

At UX Pilot we found that users who triggered one generation mode in their first week converted to paid at roughly nine times the rate of those who did not. 12.3% against 1.3%. That is a big enough gap that plenty of teams would have shipped the change on the strength of the chart alone and called the resulting uplift a win.

But the arrow could easily have pointed the other way. More motivated users might simply be more likely to find the feature, in which case turning it on by default would move nothing at all. The only way to know was to run it as a real A/B test, which we did: 120,000 users over 11 days. Free-to-paid conversion rose 44.67%, at a 96.2% win probability.

That win probability is worth reading properly. It is a statement about remaining uncertainty, not a victory lap. Reporting it honestly is part of the same discipline as scoring how strong your evidence actually is before you act on it.

The learning card, and the next Where

A test that ends in a slide is a dead end. A test that ends in a number is a loop.

So the last artefact is a learning card: the result, the insight, and the decision. The result is a number. The insight is what it means. The decision is what you did, including "nothing, and here is why".

Then that number becomes the next question of where it hurts. That is the fourth rule of the 3W Loop and the reason it is drawn as a loop rather than a checklist: Where, Why, Whether, and the result of the last one starts the next.

Teams that skip this end up rerunning the same argument twice a year, because nothing they learned was written anywhere a future team could find it. If your experiments produce decks instead of numbers, start there rather than with a new testing tool. The mechanics of the whole card, from hypothesis to threshold, are in how to design a product experiment.

The loop, the rules and the printable canvas are free under CC BY-SA 4.0. Getting a team to write "we are right if" honestly, on a decision that already has a champion, is harder than it sounds, and it is most of what we practise in the Product Discovery Workshop.