--test-split

🦉 AthenaAI is an AI-powered ensemble machine-learning trading bot for PocketOption, built on BinaryOptionsToolsV2 → Get it on GitLab
💬 Need help? Join the Chipa Discord · ⚡ Go no-code with ChipaEditor

The fraction of generated samples — the most recent ones, chronologically — held out from training and used for the walk-forward evaluation.

What it does

Trainer.run() splits the sample arrays at a single chronological cut point — no shuffling:

split = int(len(X) * (1.0 - cfg.test_split))
train_X, train_y = X[:split], y[:split]
test_X, test_y = X[split:], y[split:]

Everything before the cut trains the online models, batch models, and neural model; everything after is never trained on. Because the holdout is strictly newer than all training data, the evaluation is walk-forward: it estimates how the brain would have performed on genuinely unseen future data, preventing look-ahead leakage.

The holdout is used twice:

1. MLP vote weight — the neural model’s accuracy on the holdout becomes its accuracy_ema (its vote weight) in the ensemble; if the holdout is empty it defaults to 0.5.
2. Walk-forward report — _walk_forward() runs the full ensemble over every holdout sample and reports wins, losses, honest win rate, and the edge versus the payout-derived breakeven (1 / (1 + payout), 52.1% at the default 0.92 payout). If the holdout has fewer than 100 samples, the test is skipped with Holdout too small (N) — skipping walk-forward test. and the metrics dict is {"samples": 0} — training still completes and the brain is saved.

Choosing a value

• 0.30 is a sound default: enough holdout for a statistically meaningful win-rate estimate while leaving 70% for training.
• Smaller splits (e.g. 0.1) give the models more data but a noisier win-rate estimate; make sure the holdout still clears 100 samples or you lose the evaluation entirely.
• Larger splits give a more reliable estimate on less-trained models.
• 0 disables both the walk-forward test and a meaningful MLP weight (it falls back to 0.5) — not recommended.

Risk note: the walk-forward win rate is your honest pre-deployment estimate. Deploying a brain whose holdout WR is below breakeven means trading a known negative edge; binary options trading can lose money and past performance doesn’t guarantee future results.

Examples

python train.py --test-split 0.30     # default
python train.py --test-split 0.15    # more training data, noisier estimate

Interactions

• --max-candles — determines total samples, hence holdout size.
• TrainerConfig.payout (0.92, not CLI-exposed) — sets the breakeven line in the report.
• --no-nn — with the NN skipped, the holdout only feeds the walk-forward report.

See also

• Training guide (reading the walk-forward report)
• Ensemble concept

💬 Unsure how this interacts with the rest of the configuration? Ask in the Chipa Discord — or prototype strategies no-code with ChipaEditor.