Startup validation experiments: which test to run, and when
A validation experiment is a small real-world test of one claim, with the result decided in advance: a threshold that would confirm it and a threshold that would kill it. Pick the cheapest test that can falsify the claim you are least sure of. Interviews test whether a problem is real, landing pages and fake doors measure demand, and concierge and prototype tests show whether your answer works.
By Buteo · Updated
What is a validation experiment?
A validation experiment is a small real-world test of one claim your idea depends on, with the result decided in advance. It has three parts: the claim being tested, a threshold that would confirm it, and a threshold that would kill it.
The third part is what makes it an experiment. Without a kill threshold written down beforehand, any result can be read as encouraging. “If fewer than a set share of the people who see the pricing page start checkout, the price is wrong” can fail. “See how people react to the pricing page” cannot.
Experiments come after reading what is already published, not instead of it. They are for the claims research cannot settle. The steps before this one are in How to validate a startup idea before you build it.
Which experiment tests which claim?
Each method answers a different question. Choosing one that cannot falsify the claim you are worried about is the most common waste in validation.
| Experiment | What it tests | What it cannot tell you |
|---|---|---|
| Customer interview | Whether the problem is real, how often it bites, and what people do about it today. | Whether they will pay. People describe their past accurately and their future generously. |
| Survey | How widespread something is, once interviews have told you what to ask. | Why. A survey can only count the answers you thought of in advance. |
| Landing page test | Whether a described offer makes a stranger take a step: join a waitlist, book a call, leave an email. | Whether they will pay, or keep using it. A sign-up is interest, not a sale. |
| Fake door test | Whether people try to use or buy a specific thing before it exists. | Whether the finished product would satisfy them. |
| Concierge test | Whether your answer solves the problem, and whether people will pay for the outcome. | Whether it works at scale, or whether software can do what you did by hand. |
| Ad test | Which message and which audience respond, and roughly what it costs to get attention. | Demand on its own. A click is curiosity. |
| Prototype test | Whether people can use your solution and whether it does the job for them. | Whether they wanted it in the first place. Test the problem before you build even this. |
| Desk research | Anything already published: competitors, prices, market data, regulation, how people describe the problem. | Anything about your specific customer and your specific price that nobody has written down. |
As a rule: interviews and desk research test whether the problem is real; landing pages, fake doors and ad tests measure demand; concierge and prototype tests show whether your solution works.
How do you run each one?
Customer interview
Talk to people who have the problem. Ask about the last time it happened, what they did, and what that cost them. Do not describe your idea until the end, if at all.
The trap: Pitching. The moment you explain the product, you are collecting politeness.
Survey
Keep it short, ask about behaviour rather than opinion, and send it to people who actually have the problem rather than to whoever will answer.
The trap: Running it first. A survey written before any interview measures your own assumptions.
Landing page test
One page, one promise, one action. Send a known number of the right people to it and count how many take the action.
The trap: Counting visits or clicks. Only the committed action is the result.
Fake door test
Offer the feature or the plan as if it were available: a button, a pricing tier, a checkout step. Count who goes through the door, then tell them honestly that it is not ready and offer to let them know.
The trap: Leaving people at a dead end. Say what happened and why, straight away.
Concierge test
Deliver the result yourself, by hand, for a few real customers, and charge for it. You are the product.
The trap: Doing it free. An unpaid concierge test tells you people accept gifts.
Ad test
Run a few small ads that each state a different promise, to a defined audience, pointing at a landing page. Compare the committed actions, not the clicks.
The trap: Reading a cheap click as a cheap customer.
Prototype test
Put a rough working version, or a clickable mock-up, in front of people with the problem. Give them the task and watch without helping.
The trap: Explaining it. If you have to narrate, the prototype has not passed.
Desk research
Search for evidence on both sides of each claim, rank the sources, and open each page to check it says what you think it says.
The trap: Stopping when you find agreement.
How do you set success and kill criteria?
Write both before you start, in numbers, and make the kill criterion something that could really happen.
- Name the claim. One experiment, one hypothesis. A test of two claims at once cannot tell you which one failed.
- Choose the action that counts. A commitment, not a reaction: a deposit, a sign-up with a real address, a booked call, an hour of someone's time.
- Set the confirming threshold. How many of how many would convince you.
- Set the kill threshold. How few would make you stop. If you cannot imagine accepting it, the experiment is decoration.
- Fix the size and the deadline. Otherwise a weak result becomes “too early to say” forever.
The thresholds depend on your price and your costs, so there is no universal number to borrow. A product sold once for a small sum needs a far higher conversion than one sold on an annual contract.
What order should you run experiments in?
- The critical claim first. Test what the idea dies without before anything that is merely interesting.
- Problem before solution. Establish that people have the problem before testing your answer to it.
- Solution before price. A price test on something nobody wants measures nothing.
- Cheapest test that can fail. If an afternoon of interviews can kill the claim, do not build a prototype to find out.
What do you do with the results?
Record what happened, including the results you did not want. A test that found nothing is a result: it means the claim is still open, or that the test could not have answered it.
Weigh your own results above what you read. Something you observed real customers do is direct evidence about your idea; a published statistic is evidence about someone else's market. Then go back to the decision: continue, pivot or kill.
This is the loop Buteo, the AI idea validator, runs. After the research, it proposes experiments for the claims that are still untested, each with a success and a kill criterion. You choose which to run, the validation waits while you run them, and what you report is added to the evidence as a first-party finding, rated alongside the most credible published sources, before the idea is scored again. The details are on the methodology page.
Questions people ask about this
What is a fake door test?
A fake door test offers a product, feature or price as if it already existed, and counts how many people try to use or buy it. It measures real intent before anything is built. Once someone goes through the door, tell them honestly that it is not available yet and offer to notify them.
What is a concierge test?
A concierge test delivers the outcome of your product by hand, to a few real customers, usually for a fee. You do manually what the software would do. It shows whether the result is worth paying for before you automate anything, though not whether it will work at scale.
Is a landing page test enough to validate demand?
Not on its own. A landing page shows that a described offer earns a sign-up from people you sent to it. It does not show that they will pay or keep using the product. Follow it with something that asks for a real commitment: a pre-order, a deposit, a booked call or a paid concierge test.
Is a fake door test ethical?
It is when you are honest the moment someone acts. Do not take payment for something that does not exist, do not leave people at a dead end, and say plainly that you are gauging interest. Offering early access or an update when it ships is a fair exchange for their click.