The Fortune Teller

Live monthly forecast scoreboard

Predicting the price is hopeless. Predicting the uncertainty isn't.

Each month, three arms — persistence, a price-only LSTM, and an optional LLM reading recent headlines — commit a price range 21 trading days out for 15 tickers, before the outcome is knowable. The price itself turns out to be close to unforecastable, and this page proves that rigorously, as a negative control. What is genuinely predictable is how much each price is likely to move — and the ranges below are calibrated to be honest about that uncertainty, not to look confident.

Live track record so far

Loading…

What's actually predictable

Three separate questions, three separate answers — never blended into one score. This is the short version; each has its own section below with the full evidence.

Open predictions

Forecasts already committed to git, waiting on the calendar — grouped by ticker so all three arms' ranges can be compared at a glance. The commit history is the proof they were made before the outcome was knowable — see it here.

Volatility the real positive result

Same discipline as the price arms above: a model forecast scored in dollars-equivalent units against a trivial baseline, on identical rows. Here the target isn't next month's price, it's the realized volatility over the next 21 trading days (annualized standard deviation of daily returns) — and the baseline is persistence of volatility: "the next 21 days will be as calm or wild as the last 21." The model is a simple EWMA estimator, not a neural net — the point is to show a real edge exists with an honest, well-understood method, not to show off a bigger one.

Skill by ticker

All 15 tickers are positive — a consistency check, not just a headline average that could be one lucky ticker away from evaporating.

Predicted vs. realized volatility over time

Sampled roughly weekly for readability; skill and RMSE above are computed from the full daily series, not this downsampled view.

Why volatility is predictable when direction isn't

Daily stock returns are close to a random walk — the sign of tomorrow's move carries almost no information about the sign of the next move, which is the whole reason persistence is such a tough baseline for direction. But the size of moves behaves completely differently: calm periods tend to stay calm, and turbulent periods tend to stay turbulent, a well-documented empirical regularity called volatility clustering — it's the entire reason GARCH-family models exist in econometrics. An exponentially-weighted moving average of recent squared returns exploits exactly this: recent turbulence is weighted more heavily as a forecast of near-future turbulence, and it works, consistently, across every ticker tested here.

Live track record committed before the outcome

Scored only once 21 trading days have actually passed. Coverage and interval score are always shown together — coverage alone is trivially gamed by a wide-enough range.

Backtest track record replayed history

Produced now by replaying history the code already knows the outcome of — weaker evidence than the live table above, shown for context and volume only. Never averaged together with it.

Calibration fixed, and shown honestly

An "80% interval" is a promise: the real outcome should land inside it about 80% of the time, no more and no less. This scoreboard shipped with a real defect — its intervals only covered 67-68% of the time, both arms, a promise it wasn't keeping. Coverage is never shown alone below, because it's trivially gamed: predict a wide-enough range and you can "cover" everything at the cost of the range being useless. Width and interval score (a score that penalizes both misses and needless width) are shown alongside every coverage figure, before and after.

before the fix after the fix vertical line = 80% nominal target shaded band = pre-registered 76–84% success range

The cause turned out not to be the initially-suspected volatility-regime shift (coverage by volatility tercile came back nearly flat) but overlapping-window residual autocorrelation shrinking the effective calibration sample size. The fix — split-conformal prediction with the nonconformity score normalized by each row's own trailing EWMA volatility — brought both arms inside the pre-registered success band, with width up only 1.15–1.22x and interval score improving for both arms, which rules out "coverage fixed only by making the ranges enormous." Full per-ticker, per-year and per-volatility-regime breakdown, plus every rejected alternative, is in results/improvement.json.

Per-ticker: actual price vs. predicted band

Solid line: actual closing price. Bands: predicted range at the time of forecast — muted/hatched for backtest, solid accent for live. A band that misses the line is a miss; width is the cost of getting it right.

Price direction no skill — the negative control

Every credible forecasting study needs a result that's supposed to fail, to prove the evaluation itself isn't rigged to find edges that aren't there. This is that result, scored with the same rigor as the positive findings above — as a probability, not bare accuracy, and against a proper baseline, not a coin flip.

A model with real directional skill would beat simply predicting the historical base rate every time. This one doesn't — its Brier score and log loss are both slightly worse than the base-rate baseline, and its confidence doesn't even rank-order correctly (full calibration curve in results/direction_evaluation.json). That's not a tuning failure to go fix. Markets pricing in almost all easily-available information is the textbook expectation here, and getting exactly that result is what makes the volatility and calibration findings above credible rather than cherry-picked out of a pile of things that were tried.

What this is not

  • Not a trading system. No costs, no slippage, no execution modelled.
  • Not investment advice. Nothing here is a recommendation to buy or sell.
  • A volatility forecast is not a trading signal. It's a statement about how wide the uncertainty is, not about which direction to bet — direction is the one thing this page shows no skill at predicting.
  • The LLM arm is expected to lose to persistence. It reads headlines with no deep price history behind it; losing is a legitimate, publishable result, not a bug to fix by tuning the prompt.
  • Backtest rows are weaker evidence than live rows. They are produced by code that already knows what happened; live rows are committed to git before the outcome exists. They are never combined into one number on this page.

How this relates to the main study

The main results page asks whether an LSTM beats persistence at next-day close prediction on a fixed 2.5-year held-out period — the answer there is no, and it's the same negative-control pattern this page finds again at a 21-day horizon. This page goes further: alongside the same "does it beat persistence on price" question, it separately asks whether the range around that price prediction is forecastable and honestly calibrated — and there, unlike price direction, the answer is yes. Same tickers, same persistence baseline, same "no lookahead" discipline — and the live evidence keeps accumulating monthly.

How this actually works

Written for a reader who is comfortable with the idea of a stock price but hasn't necessarily trained a model. No step is skipped for being "obvious."

1. The data pipeline

Every month, on the 1st, a scheduled job pulls fresh daily open/high/low/close/volume bars for 15 large-cap tech tickers (prices are split- and dividend-adjusted, so a 4:1 stock split doesn't look like a 75% crash to the model). It scores last month's predictions against what actually happened, trains a fresh model on everything known up to today, predicts a price range 21 trading days out, and commits both the scoring and the new prediction to git — before anyone, including the code itself, knows what will actually happen.

2. What persistence is, and why it's a serious baseline

Persistence means "tomorrow's price is today's price" — or here, "the price 21 days from now is the price today." It sounds too dumb to beat, which is exactly why it's dangerous to ignore: daily stock returns are close to a random walk, meaning the best single-point guess for a future price is very nearly today's price. Decades of published finance and forecasting research keep re-finding this. A model has to clear this bar, not a coin flip — and on this page, most of the time, it doesn't.

3. What an LSTM is, and why it's used here

An LSTM (Long Short-Term Memory network) is a type of neural network built to find patterns in sequences — a natural fit for a window of the last 20 trading days' prices and volumes. This project uses a bidirectional LSTM: it reads the window forwards and backwards internally, which helps it use the whole window rather than just the most recent day, but it still only ever sees days strictly before the day it's predicting — the model architecture doesn't create lookahead, and a dedicated test (test_as_of_boundary_holds) checks that no leak exists in the surrounding code either.

4. Prediction intervals, and what "coverage" means

Instead of a single number ("$142.30"), each arm predicts a range ("$135–$150") at a stated confidence level, here 80%. Coverage is the fraction of the time the real outcome actually landed inside the predicted range — it should be close to 80% for an 80% interval that's telling the truth about its own uncertainty. Coverage alone can be gamed trivially (predict $0–$100,000 and you'll "cover" everything), which is why this page always reports the interval score alongside it — a score that penalizes width as well as misses, so a wide, lazy interval scores worse than a tight, honest one even at equal coverage.

5. Conformal calibration — how the ranges are actually built

The ranges aren't guessed. Each month, the model's own prediction errors on recent, known data are measured, and a split-conformal procedure turns that error distribution into a range with a real statistical coverage guarantee — not just "the model's confidence," which neural networks are notoriously bad at reporting honestly. The range width is further scaled by each ticker's own recent volatility (how much it's been moving lately), so a historically calm stock gets a tighter range than a historically turbulent one, for the same nominal confidence level.

6. The LLM arm

An optional third arm gives a large language model the last 60 closing prices and up to 10 recent headlines, and asks it for a price range in strict JSON. If the reply is malformed, refuses to answer, or gives a range where the low is above the high, that counts as an abstention — recorded honestly as "no prediction," never smoothed into a fabricated number. This arm only runs live: the free news source used here has no deep archive, so there's no honest way to backtest it against history it could never have actually read at the time.

7. Why backtest and live never mix

A backtest row is produced today, by code that already knows what happened — it's a replay, useful for volume and context, but it can't prove the model wasn't shaped (even unconsciously) by knowing the answer. A live row is committed to git before the outcome exists — the commit timestamp itself is tamper-evident proof it was made in advance. Averaging the two into one headline number would quietly launder the weaker evidence into the stronger claim, so this project keeps them in separate tables, separate totals, everywhere, enforced by an automated test that fails the build if they're ever combined.

8. The honest finding

On the fixed next-day study, the LSTM does not beat persistence. On this live scoreboard, the same pattern holds at the 21-day horizon, scored properly as a probability, not bare accuracy — Brier score and log loss both come out slightly worse than a base-rate baseline. That is not a failure of this project; it is the finding, and it is what makes the two results below credible rather than cherry-picked.

9. What's actually predictable

Two things came out of the same evaluation discipline applied to price direction, but landed on the opposite side of zero. Volatility — how much a price moves over the next 21 days — beats a persistence-of-volatility baseline by 8.4% on every one of 15 tickers, using nothing fancier than an exponentially-weighted moving average of recent squared returns; see the "Volatility" section above for the per-ticker breakdown and why volatility clustering makes this possible even though direction is unpredictable. And the prediction intervals themselves were miscalibrated (67-68% actual coverage against an 80% target) until a split-conformal fix, scaled by each ticker's own recent volatility, brought both arms to 77.7-79.8% — see "Calibration" above. Neither result claims the price is forecastable. Both are honest, checkable claims about the shape of the uncertainty around it. Full detail on every variant tried, kept or rejected, is in the main study's README and results/improvement.json.