Most trading bots don’t fail because the strategy was bad. They fail because the engineering around the strategy was — a backtest that quietly saw the future, fee assumptions that flattered the numbers, an execution layer that double-fired orders on a reconnect. Building trading systems is a software-engineering discipline with money attached, and this is what doing it properly looks like: from architecture, through honest backtesting and paper trading, to the first careful live dollar.
Quick answer
A trading bot is trustworthy only after it survives four gates in order: (1) a clean architecture that separates data, signals, risk and execution; (2) a backtest with honest costs — fees, slippage, funding and latency — and no look-ahead; (3) an extended paper-trading run on live data where you measure tracking error against the backtest, not just profit; and (4) a micro-live stage with a hard capital cap and a kill switch, because real fills behave differently from simulated ones. Skipping a gate doesn’t speed the project up — it moves the discovery of the problem to the most expensive possible moment.
The architecture: four concerns that must never share code
Whether the strategy is a simple momentum rule or a portfolio of models, production trading systems converge on the same separation:
- Market data ingestion — normalised feeds (websocket and REST fallback), with gap detection and clock discipline. Bad ticks and silent feed drops generate more false signals than bad maths ever will. Persist raw data; you will need it to reconstruct incidents and to backtest honestly later.
- Signal engine — the strategy itself, computed from data as it was available at the time, emitting intents (“want to be long X at up to Y”) rather than orders. Keeping the strategy pure — no exchange calls, no side effects — is what makes it backtestable and paper-tradeable with identical code.
- Risk layer — an independent gate every intent passes through: position limits, per-order and daily loss caps, exposure and leverage ceilings, sanity checks against fat-finger sizes, and a global kill switch. This layer assumes the strategy is wrong; that’s its job. Hard caps belong here, enforced in code — not in the strategy’s good intentions.
- Execution — the only component that talks to venues. Idempotent order placement (client order IDs, so a retry can’t double-fire), explicit handling of partial fills, cancels and rejections, reconciliation of expected versus actual positions after every disconnect, and safe restart from persisted state. Execution code is where reconnect storms and half-filled orders turn a profitable strategy into a loss.
Around all four: a double-entry internal ledger reconciled against venue statements, structured logs of every decision, and dashboards with alerting. If the system trades while you sleep, observability is not optional — the 3am question is always “what is my position right now, and why?”
Backtesting: the discipline of not fooling yourself
A backtest is an argument that a strategy would have worked. Most backtests are wrong in the strategy’s favour, in the same handful of ways:
- Look-ahead bias — the classic. Signals computed with data that wasn’t available yet: indicators on incomplete candles, same-bar entries at prices you couldn’t have had. The defence is architectural: run the backtest through the same event-driven pipeline as live trading, feeding data point by point, so the strategy physically cannot see the future.
- Survivorship and selection bias — testing on today’s instrument list means testing only on the survivors. Delisted stocks, dead tokens and renamed pairs must be in the data, or your universe is flattering you.
- Cost realism — fees (maker vs taker, your actual tier), spread crossing, slippage scaled to order size and book depth, and — in perpetual futures — funding rates, which quietly dominate the economics of many strategies. A strategy that’s profitable at zero cost and dead at real cost isn’t a strategy; it’s a fee donation with extra steps. High-frequency, small-edge approaches are especially fee-dominated, and plenty of published “profitable” strategies evaporate the moment realistic maker/taker economics are applied.
- Overfitting — sweep enough parameters and something will look brilliant by chance. Defences: fewer parameters, walk-forward analysis (optimise on one window, validate on the next, roll), a genuinely untouched out-of-sample period you test once, and parameter-sensitivity checks — if performance collapses when a parameter moves 10%, you’ve fitted noise.
- Regime blindness — a backtest spanning one market regime proves the strategy suits that regime. Test across trending, ranging and stressed periods, and report them separately; the average hides the drawdown that kills you.
The output of a good backtest is not “the equity curve went up”. It’s a report: net-of-cost returns by regime, drawdown depth and duration, turnover, fee load, capacity estimate, and the sensitivity analysis — the evidence a trading firm’s risk committee (or your own scepticism) actually needs. This is the same statistical honesty that applies to any predictive ML — our guide to AI/ML integration done properly covers the general form of the leakage problem.
Paper trading: what it proves that backtests can’t
Paper trading — running the full system against live markets with simulated money — is not a victory lap for a good backtest. It tests different things:
- Live data is uglier than historical data — feed gaps, out-of-order messages, exchange maintenance windows, symbols behaving strangely at rollover. Paper trading is where the ingestion layer earns its keep.
- Operational reality — restarts, deploys mid-session, reconnects, daylight-saving boundaries, weekend gaps. You want the crash-on-reconnect bug to fire now, with zero dollars at stake.
- Simulated fills must be pessimistic to be useful — market orders should assume spread-crossing plus slippage; limit orders are the honesty test, because a naive simulator fills your maker order the moment price touches it, ignoring the queue of real orders ahead of you. Model queue position conservatively or your paper results will systematically exceed anything live can deliver. Exchange testnets help with API mechanics but their order books are fictional — treat them as integration tests, not economics.
- The metric that matters is tracking error, not profit — compare what paper trading did against what the backtest predicted for the same period. If they diverge, find out why before going live: the divergence is either a simulation flaw or a cost you haven’t modelled, and both are cheaper to discover on paper.
How long? Long enough to cover a meaningful sample of the strategy’s trades and at least a few operational incidents — for most intraday strategies that’s weeks, not days. A paper run that has never survived a bad week has proven very little.
Micro-live: the gate almost everyone skips
Between paper and production belongs a deliberately tiny live stage: real money, hard-capped at an amount you can lose without caring, with every risk limit set tight and the kill switch tested for real. Micro-live exists because real fills differ from simulated ones — queue position, partial fills, venue idiosyncrasies, the actual behaviour of your fee tier — and because nothing else tests the full loop of money moving. Graduate capital in steps, and only after the live tracking error against paper looks like the paper tracking error against backtest: small, explained, and stable.
Infrastructure that takes trading seriously
- Placement — a VPS in the right region is fine for minute-scale strategies; latency-sensitive work belongs in proximity or colocation environments near the venue’s matching engine. Match the spend to the strategy’s actual latency sensitivity — colocation for a daily-rebalance bot is theatre.
- Connectivity — REST and websockets for most venues; FIX for institutional venues and brokers, which brings session management, sequence-number recovery and conformance testing into scope.
- State and secrets — restart-safe persisted state (positions, open orders, strategy state), API keys scoped to trading-only (never withdrawal), stored properly, rotated on any suspicion.
- Monitoring — position, P&L, order-rate and feed-health dashboards with alerts to a human; an order-rate limiter as a circuit breaker against the runaway-loop failure mode that has embarrassed even major firms.
- Auditability — every order, fill, decision and config change logged immutably. When something goes wrong you reconstruct it from logs; when a counterparty or auditor asks, you answer from evidence. The same discipline covered in our fintech architecture guides applies here at higher speed.
Why trading firms bring Ken in
Founder Ken Armitt has spent 27 years building payments and trading-grade systems, and trading firms and platform owners internationally engage him for exactly the work described above: independent technical audits of trading platforms (architecture, risk controls, evidence of what a system actually did), building backtesting and paper-trading environments their own teams use self-serve, clean-room replication and validation of strategies against honest cost models, and execution infrastructure — from exchange APIs to FIX connectivity — across colocation environments in venues from London to Singapore and Tokyo. The consistent theme: separating what a strategy claims from what the engineering can prove, before capital finds out the difference. Engagement details are confidential as standard; the methods aren’t, which is why this article exists.
One thing we don’t do: promise returns. Nobody honest can. What good engineering promises is that the numbers you see are real, the risks are capped by code, and the system fails safe. That — not a secret indicator — is what separates trading operations that last from the ones that blow up.
Frequently asked questions
How long should a trading bot be paper traded before going live?
Until it has covered a meaningful sample of trades and survived operational incidents — typically weeks for intraday strategies — and until its tracking error against the backtest is small and explained. Time alone isn’t the criterion; evidence is.
Why do backtest results so often fail to survive live trading?
Four usual suspects: look-ahead bias in the backtest, optimistic fill and fee assumptions, overfitted parameters, and a live regime the test data didn’t cover. Each is detectable in advance with walk-forward testing, pessimistic fill models and honest cost accounting.
Can an existing dev team build a trading system, or do we need specialists?
A strong team can build it with specialist direction. The domain knowledge that prevents expensive mistakes — fill modelling, risk-layer design, venue behaviour, reconciliation — is exactly the kind of judgement a fractional specialist supplies without a full-time hire.
Is this financial advice?
No. This is engineering guidance about building and validating trading software. Nothing here is a recommendation to trade or an assurance of performance — and you should be suspicious of anyone in this field who offers one.