Candle data formats

🦉 AthenaAI is an AI-powered ensemble machine-learning trading bot for PocketOption, built on BinaryOptionsToolsV2 → Get it on GitLab
💬 Need help? Join the Chipa Discord · ⚡ Go no-code with ChipaEditor

Both the standalone trainer and the bot’s in-process pre-training read historical candles from a CSV file. The same auto-detecting parser logic lives in two places:

• load_candles() in src/training/dataset.py — used by train.py (see Training)
• _load_dataset() in src/bot.py — used at bot startup when no saved brain exists and PO_DATASET is set

Files are opened with utf-8-sig encoding, so a UTF-8 BOM (common in Windows exports) is handled.

Format detection

The parser reads the first line and decides:

• If it contains ; and none of the header words time, open, high, date (case-insensitive) → HistData semicolon format (Detected HistData semicolon format (no headers) is logged).
• Otherwise → headered CSV, delimited by ; if the first line contains one, else ,. The log prints the detected columns: CSV columns found: [...] (delimiter=',').

Format 1: HistData semicolon (no header row)

20240102 000000;1.10432;1.10441;1.10428;1.10439;0

Fields, in order: datetime;open;high;low;close[;volume]. Rules:

• Lines with fewer than 5 fields are skipped.
• The datetime is parsed as YYYYMMDD HHMMSS (when the field is ≥15 characters) or just YYYYMMDD (≥8 characters — date-only rows land at midnight). Anything shorter is skipped.
• Volume is optional; missing volume becomes 0.0.

This is the format of the free M1 exports from histdata.com (“Generic ASCII” M1 bars).

Format 2: Headered CSV (comma or semicolon delimited)

Any CSV with a header row. Accepted column names (first match wins, in this order):

This covers MetaTrader 5 exports (which use time, open, high, low, close, tick_volume, real_volume) and most broker/data-vendor CSVs out of the box.

Note: timestamps in other layouts — e.g. DD/MM/YYYY, ISO with a T separator, or with milliseconds — are not recognized. 2024-01-02T00:00:00 contains - and : so it is attempted as %Y-%m-%d %H:%M:%S and fails, and the row is skipped.

Silent row-skipping

Both parsers wrap each row in a try/except and silently skip anything that fails to parse (bad numbers, unrecognized timestamps, short rows). This means:

• A partially dirty file still loads — good.
• A file whose timestamp format is wholly unrecognized yields zero candles with no per-row errors — the only clue is the final count: Parsed 0 candles from file. Always compare that count against your file’s row count.

Downstream minimums: the trainer aborts under 50 candles (Dataset too small) and under 200 valid samples; the bot logs Dataset too small (N candles), skipping pre-training. under 50 candles and Only N valid samples from dataset, skipping. under 20 samples. Both keep only the most recent 500,000 candles (--max-candles is configurable in the trainer).

Where to get data

• histdata.com — free 1-minute FX history going back years. Download the “Generic ASCII” M1 files; they are exactly Format 1. Concatenate monthly files into one CSV for longer histories.
• MetaTrader 5 export — open the symbol’s chart data (View → Symbols → Bars, or use a script), export to CSV; the default MT5 column names match Format 2.
• Any other source works if you can reshape it to Format 2’s column names and one of the two timestamp forms.

Make sure the candle period of the data matches --timeframe (trainer) / PO_TIMEFRAME (bot); the label horizon is computed as expiry / timeframe.

Quick validation

head -3 EURUSD_M1.csv          # bash

Get-Content EURUSD_M1.csv -TotalCount 3   # PowerShell

Then run the trainer and confirm the log’s parsed count is close to the file’s line count.

See also

• Training
• –data and PO_DATASET
• Troubleshooting

💬 Questions about running AthenaAI? Ask in the #trading-bots channel on the Chipa Discord — and if you’d rather design strategies without touching Python, try ChipaEditor.