The method
How we work
The same method for research and for client work. The only thing that changes is how expensive we let it get.
The answer usually matches whoever is being paid. A consultancy’s diagnosis lands on a project it can sell. A paper written to support a position finds support. Nobody has to be dishonest for that to happen; it takes an incentive and nothing in the process designed to push back. Below is what pushes back.
Seven steps
- 01
State the claim before gathering evidence.
Hypotheses are written as falsifiable statements with operational definitions, before anything is retrieved. What counts as “failure”, as “production”, as “ROI” is fixed in advance, so the definitions cannot drift toward the conclusion while the work is under way.
- 02
Write down what would prove you wrong.
Every hypothesis carries kill criteria and a disconfirmation path: which search, in which sources, would surface the strongest evidence against us. It is written before we look, because afterwards it is much harder to want to.
- 03
Gather the evidence blind.
The people and the models retrieving evidence receive the question and a neutral working title. They never receive the thesis, the preferred angle, or prior findings. A search that already knows what it is looking for will find it, which is why retrieval and argument are kept apart by design rather than by discipline.
- 04
Pay someone to attack it.
An adversarial pass tries to break every claim. Verdicts use six fixed terms rather than a judgment call: contradicted, unsupported, ambiguous, supported with qualification, not assessable, survived specified tests. A human adjudicates anything load-bearing.
- 05
Have an independent person check the load-bearing numbers.
Independent means having nothing to gain. An author cannot verify his own report. An employee cannot produce an error rate we then describe as independent. Where no external verifier is available for a piece of work, the check is labeled internal and non-independent in the published text. Honest labeling costs a little credibility. An internal check described as independent costs all of it.
- 06
Hold a gate where abandoning the work is a real option.
Evidence meets hypothesis, and the outcomes are proceed, reframe, or kill. A killed thesis is a published result, not a buried one.
- 07
Publish the error rate, and what would change the conclusion.
First-pass error rates with confidence intervals, material and minor reported separately. Two numbers, never conflated: the observed error rate among the claims we checked is a fact about what we checked, and the estimated prevalence across the whole report is an inference from a sample. Reporting the first as though it were the second is exactly the category error this method exists to prevent, so we report both.
Rigor is proportional. Honesty is not.
Running a three-day brief through journal-grade machinery is waste. Running a flagship report through brief-grade machinery and calling it a review is the one thing we will not do.
So the expensive parts scale with the work: systematic search, independent verification, external review, preregistration. Five things never scale, because they are cheap.
- Every claim traces to a source a reader can check.
- Nothing is asserted beyond what the evidence supports.
- Uncertainty is stated wherever it is material.
- Scope and limits are declared: what we searched, what we did not, and what would change the answer.
- Errors are corrected in place, dated, and visible. Silent edits are prohibited.
When there is not enough time, we downgrade the claim, never the honesty.
What we publish about the method, and what we do not
We publish what we do at each stage, why, what has to be true before the work proceeds, how a claim is graded, and what would stop us. That is a methods section, and it is what a reader needs to judge a finding.
We do not publish the implementation: the retrieval prompts, the model routing, the extraction schemas, the audit scripts. That is a lab notebook, and no journal asks for one.
An honest inventory
What we have actually run
- 01The AI Execution Gap, March 2026, was produced under an earlier version of the pipeline, with cross-model adversarial review and primary-source verification. It was not preregistered and carries no error rate.
- 02The current method, version 2.2, has been revised twice after external methods review. Its first full application is The Agentic Enterprise: protocol approved, registration on OSF next.
- 03The first published error rate will appear on our next Report, not the last one.