Interview first, or go straight to the test?

You know this meeting.

Half the room wants to talk to users first. The other half wants to put it behind a flag and see what happens. Both sides are right sometimes, the argument runs about twenty minutes, and it gets settled by whoever is most senior or most tired.

It does not have to. There is a decision rule, it takes about ninety seconds, and it is one of the four rules of the 3W Loop: you may skip the Why, but only deliberately and only under three conditions.

The three checks

Run them in order, out loud, with the people who disagree in the room.

1. Is the fix cheap and reversible? Cheap enough that being wrong costs days rather than a quarter, and reversible enough that you can turn it off without a migration. A default toggle qualifies. A pricing change qualifies. A schema change does not.

2. Is there one obvious mechanism, and does everyone name the same one? This is the check people fake. Do not ask "is the mechanism obvious", ask three people separately to state it in one sentence. If you get three sentences, the mechanism is not obvious, it is assumed.

3. Would you run the test anyway, whatever the interviews said? If yes, the interviews are not informing a decision, they are delaying one.

Three yeses and you go straight to the test. Fewer than three and you interview first. Either way you run the test, which is the part with no exception.

A case that failed check 2

At elyps, our analytics were unambiguous. Roughly eight in ten new sign-ups were abandoning onboarding, concentrated almost entirely on the screen right before identity verification.

Check 1 was arguably a pass. Check 3 was a pass. Check 2 was where it fell apart, and it fell apart quietly, because there was an obvious mechanism: the ID check is too much friction. Published market reports said the same thing. Everyone in the room would have signed that sentence.

Except when you pushed on it, people meant different things. Too many steps. Too slow. Too invasive. Distrust of a new bank holding a passport scan. Four mechanisms wearing one label, each implying a different fix.

So we interviewed churned users, and the answer was none of the four. People were opening the app on the subway and at work. They did not want to pull their ID out in public. The problem was the moment, not the step.

The fix was a "Do it later" option with an evening reminder, and drop-off on that screen went from roughly 80% to 60%. Had we gone straight to a test, we would have tested a shortened identity check, watched it do nothing, and concluded that onboarding friction was simply the cost of being a regulated bank. The full write-up is here.

A case that passed all three

At UX Pilot the analytics showed that users who triggered one particular generation mode in their first week converted to paid at roughly nine times the rate of users who did not.

Check 1. The fix was flipping a default on for first-time users. Days of work, one flag, reversible in an afternoon. Pass.

Check 2. One mechanism, and everyone stated it the same way: people who see the higher-quality output early believe the tool is worth paying for. Pass.

Check 3. Would we have run the A/B test regardless of what interviews told us? Obviously. A correlation that large still has to be tested, because the causal arrow could easily have pointed the other way, with more motivated users finding the feature rather than the feature creating motivated users. Pass.

So we skipped Why on purpose and went straight to Whether. 120,000 users over 11 days, success bar set at +10% before the run. Free-to-paid conversion rose 44.67%. That one is written up too.

Note that skipping the interviews here was not a shortcut. It was the correct instrument choice, and it is also a good example of how often the winning change is far smaller than the problem.

The two ways this goes wrong

Interviewing when you should have tested. Three weeks of scheduling, conversations, and synthesis to confirm what a two-day A/B test would have settled with a number. Worse, interviews cannot give you effect size. Ten people telling you they would use something does not tell you whether conversion moves 2% or 40%, and the difference between those two decides whether you build it.

Testing when you should have interviewed. You get a result and no mechanism. If it wins, you cannot generalise it or repeat it elsewhere in the product. If it loses, you cannot tell whether the idea was wrong or the execution was, so the idea comes back in six months with a new champion and no new evidence.

The second failure is more common in teams that are proud of their experimentation velocity, and it is why a standing interview slot earns its place even in a heavy testing culture. Five to eight conversations is a week, not a phase.

The check with no exception

Whichever route you take, the test happens. That is not a preference, it is the one rule in the loop with no conditions attached, and it is worth stating in its blunt form: a test you cannot fail is a demo.

The rest is judgement, and it gets better with reps. The three checks just make the judgement visible so the team can disagree about the right thing.

The whole loop, the four rules and the kit are free under CC BY-SA 4.0. Running the checks on your own live decisions, with your funnel on the wall, is what we do in the Product Discovery Workshop.