How it was measured
The baseline is each row’s own previous close, so the model and the baseline are always scored on identical rows. Skill score is 1 − RMSEmodel / RMSEpersistence: positive means the model beat persistence, zero means it matched it, negative means the trivial forecast was better.
Experiment A — LSTM against persistence
Two target parameterisations. Level predicts the next close directly. Return predicts the day-over-day change and converts back to a price.
Why the level model fails
It is not undertraining. A min-max scaled level target cannot express a price above the training maximum, and this universe roughly doubled over the held-out period. The model tops out well short of where the prices actually went.
Is the return model predicting anything?
Mostly it predicts “no change”. The moves it does predict are far smaller than real ones — though their direction is faintly, consistently correlated with reality. That correlation is real and it is not enough to beat doing nothing.
Experiment B — does news sentiment help?
Both arms share dates, splits and seed. The only difference is two extra input columns: a daily sentiment score and a missing-indicator. The window stops before Experiment A’s held-out period so no architecture is chosen on protected data.
DELL is a control: the sentiment source never covered it, so its two columns are entirely missing.
Experiment C — what a pre-split scaler buys you
One line changes: whether the feature scaler is fitted on the training rows or on the whole series. The leaky arm never sees a test row during training — it only inherits the test period’s minimum and maximum through the normalisation constants.
What this is not
- Not a trading system. No costs, no slippage, no position sizing, no execution. Nothing here was backtested as a strategy.
- Not investment advice, and not a price forecast.
- Not a claim that LSTMs are useless. It is a claim about this task, this universe, this period, and this architecture, measured against the right baseline.
- A negative result. That is the finding, not a failure to tune — the diagnostics above identify the mechanism in each arm.
Reproducing this
git clone https://github.com/Ssavan99/fortune-teller cd fortune-teller python -m venv .venv && .venv/Scripts/activate # source .venv/bin/activate on Unix pip install -r requirements.txt python -m scripts.run_baselines python -m scripts.run_experiment_a
Data snapshots are committed, so nothing above needs a network connection or an API key. Every figure on this page is read from the JSON those scripts write.