Find out: test your riskiest assumption
Why test at all
Because people are bad at predicting what they need and good at reacting to things that exist. Experiments put something that exists in front of them, cheaply, and watch. The best startups experiment before building. Ads to a landing page. A smoke test with a fake button. A single-feature MVP. A concierge version done by hand. Each one answers a question for a fraction of the cost of the real thing.
Interview first, or straight to the test?
Three checks. Is the fix cheap and reversible? Is there one obvious mechanism everyone can name? Would you run the test anyway, whatever the interviews said? Three yeses: go test it. Fewer: interview first. Either way, never skip the test. A test you cannot fail is a demo.
This is the decision rule from my 3W Loop: analytics shows where it hurts, interviews show why, an experiment shows whether the fix works. You may skip why. You may never skip whether.
The test card
Five fields, written before the test runs. We believe that. To verify, we will. And measure. We are right if. By. The fourth field needs a number. No number, no test, because without one you will read whatever result you get as a win.
Set the bar before you see data. UX Pilot set a success bar of plus 10% on free-to-paid conversion. The result was plus 44.67%. Had we set the bar after, we would have learned nothing about our own judgement.
The learning card
When the test ends: result, what happened, in numbers. Insight, what it means. Decision, what you will do. The decision line is what feeds the next test. A learning card with no decision is a report.
Pick the cheapest test that could prove you wrong
- Ads to a landing page: is there demand at all? Good click-to-signup is 5 to 10%.
- Smoke test or fake door: will they click the thing that does not exist yet?
- Concierge: deliver the value by hand for five customers before automating anything.
- Single-feature MVP: the one feature from your spec, nothing else.
- A/B test: only once you have traffic. Everything above works with none.
Two test cards, two routes
elyps. We believe that letting users defer identity verification to a convenient moment, with a reminder at home, will cut pre-verification churn. To verify, we will add a Do it later option and an 8pm push. And measure pre-verification churn, completion within 24 hours, push opt-in. We are right if overall onboarding churn drops at least 10%. By four weeks. Result: churn fell from 80% to 60%, a 25% relative decrease, and the flow became the default. Route: interview first, because none of the three checks were a yes.
UX Pilot. We believe that toggling Deep Design on by default will raise free-to-paid conversion by improving quality perception among first-time users. To verify, we will A/B test the default for first design generations. And measure sign-up to generation and free-to-paid conversion. We are right if free-to-paid rises at least 10%. Result over 120,000 users in 11 days: plus 44.67%, rolled out to everyone. Route: straight to the test. Cheap, reversible, one mechanism, we would have run it anyway.
Test card and learning card
One card for the riskiest assumption from Module 5. Fill the learning card when the result is in, not before.
Tick only what is true. Nobody is grading this. The next module assumes it.