Research protocol
The model comes later.
Most of the work in systematic research happens before anything is fitted. The protocol exists to get the upstream decisions right, and to make every one of them checkable afterwards.
Where research goes wrong
The largest errors in systematic research usually happen upstream.
The wrong target. Information that wasn't available at the decision time. Correlated inputs mistaken for independent evidence. Thresholds chosen after seeing outcomes. A strategy specification that changes between research and production.
None of these are modeling problems, and no model fixes them. Twenty inputs that agree aren't twenty confirmations. They're one signal echoed twenty times.
The process
Specify the question
What exactly is being predicted or decided? What constitutes success or failure? The target has to represent the trade, not something adjacent to it, and the pass/fail criteria are written down before the data is touched.
Reconstruct the information set
Only data genuinely observable at the decision timestamp enters the experiment. Revised fields, inherited thresholds, and vendor histories are checked against what was actually knowable at the time.
Establish the simplest explanation
Always-hold, naive historical forecasts, structural rules, linear models. These are the benchmarks. Anything more sophisticated has to improve on them.
Add complexity only when it earns its place
Nonlinear models, gradient-boosted trees, hyperparameter optimization, evolutionary search, alternative data. Each addition is measured against the baseline it claims to beat.
Hold out the verdict
Development happens inside development history. Final data remains sealed - not used for signal selection, parameter tuning, or stopping rules. One pass. No reruns.
Preserve research/live parity
What goes live should be the same economic object that was tested: the same features, the same timestamps, the same specification, computed the same way.
Tooling
Search is a tool, not the thesis.
When a defined problem creates a genuinely combinatorial search space, we use evolutionary search and optimization methods to explore it inside the validation protocol. The search answers to the same baselines, the same sealed holdout, and the same pre-registered criteria as everything else.
Most people start with the model.
We start with the question.
Once the target is right, the information set is clean, and the baseline is honest, the choice of model matters far less than it appears. A linear regression, a threshold rule, a boosted tree. They tend to agree when the upstream decisions are right, and they all fail when those decisions are wrong.
If the evidence is there, you'll see it. If it isn't, we'll say so.
See how this runs as an engagement. For managers
Discuss a research problem