Google's newest forecasting model lost to a formula from 1994

In late August, Google released TimesFM 3.0. The pitch was blunt: first place across three mainstream time-series benchmarks, on both point-forecast and probabilistic metrics.

I happened to have something on hand to test it against. In training material I'd built earlier for a manufacturing client, I'd included a small tool that actually runs: it takes international futures quotes for three industrial metals and outputs "over the next N days, here's the range the price is most likely to fall within." The method it uses is EWMA, published by J.P. Morgan in 1994.

So the question was simple:Should a 32-year-old formula be replaced by an AI model released last week?

Half a day later I had an answer. The answer is no. But this piece isn't about "AI is useless" — it's abouthow to ask the question so that you actually get an answer. Because before I ran the comparison, I assumed the new model would win.

As is customary in this series, the method first: this was an AI collaboration experiment. Claude wrote the code, the experimental design was hashed out back and forth, and every "is this a fair comparison?" judgment call was mine.

1. The two contenders

Contender A: EWMA (1994)

Exponentially weighted moving average. In plain terms it does one thing:it measures how violently prices have been moving recently, weighting recent days more heavily.Yesterday's volatility matters more than last month's, last month's more than last year's, with weights decaying backward at a fixed ratio.

It has one crucial property —it never guesses direction. It only answers "how far will it swing," then assumes prices start from today with equal odds of moving up or down. So what it produces isn't a forecast line, it's afan: one day out it's roughly this wide, five days out wider, twenty days out wider still.

The whole formula fits on one line and runs in under a second.

Contender B: TimesFM 2.5 (2025)

Google's time-series foundation model. In plain terms, it'sGPT for time series: it's been pre-trained on over a trillion time points, and you feed it a stretch of historical numbers and it hands back the future directly — no retraining on your data required. This is called "zero-shot."

It outputs nine quantiles at once: the 10th, 20th, ... through the 90th percentile. So it gives you intervals by design, not just a single number. That matters, because it means both contenders can bescored on the same sheet of paper。

Something every freelancer needs to know first

I tested 2.5, not 3.0. The reason has nothing to do with technology:

• the TimesFM 3.0 weights are under a non-commercial licenseand are prohibited in production environments.
• 2.5 is Apache-2.0, which permits commercial use.

If you plan to put the model inside something you deliver to a client, 3.0 is currently a closed door (unless you go through Google's own paid hosted service). Nobody puts this on a launch slide, but it determines whether your project can ship at all.

Check the license first, then test performance — do it the other way around and the half day you spent testing is wasted the moment you find out you can't use it.

2. The rules: what makes a comparison fair

This section is the most valuable part of the article.Most "we A/B tested it" conclusions are untrustworthy, and this is where they die.

Rule 1: Score the old method first

It sounds obvious, but I've seen it too many times: the person pushing for a new tool can tell you how good the new tool is, but can't tell youwhat the current setup scores. Without a baseline, the word "improvement" means nothing.

So the first thing I did was re-run the current method end to end. Using four-plus years of cleaned data (1,072 trading days), I recomputed the volatility figures and got numbers matching what the tool's own page displaysexactly, to three decimal places. This step confirmed I was testing the same thing, not some approximation I'd written myself.

That reconciliation took ten minutes, but skipping it would have left every downstream conclusion standing on nothing.

Rule 2: Score the intervals, not the "accuracy"

Most people compare forecasts by asking "did it get it right?" But both contenders output intervals, not single numbers, so you need a score built for intervals, called pinball loss。

In plain terms, here's what it penalizes:you say there's a 90% chance the price won't exceed a certain level and it does — heavy penalty; you play it safe and open the interval as wide as the ocean — also penalized.It punishes both over-optimism and saying nothing useful. Lower is better.

There's also a more intuitive metric:coverage. If you claim this is an 80% interval, then roughly 80% of days should actually land inside it. Landing at 70% means you're understating risk; landing at 95% means your interval is so wide it tells you nothing.

Rule 3: Run a significance test, or you're comparing luck

This is the step most often skipped, and the most fatal to skip.

I used 300 rolling test points — taking the later portion of the four years of data, treating each successive day as "today," and forecasting 1, 5, 10, and 20 days ahead. Here's the problem:those 300 test points overlap.Today's 20-day forecast and tomorrow's 20-day forecast share 19 of the same days.

Overlap means the errors are highly correlated. One stretch of market that happens to be easy to predict will make twenty-odd consecutive test points look good at once. Compare the two average scores directly at that point andyou're quite possibly just comparing who got luckier。

The fix is a Diebold-Mariano test with Newey-West correction. The name is intimidating; what it does is easy to grasp:it first estimates how correlated the test points are with each other, discounts the credibility of the gap accordingly, and then answers one question — what are the odds this gap is pure coincidence?

By convention, a p-value below 0.05 counts as a real difference.

The experiment demonstrated why this step matters partway through. On one set of numbers, the new model won by 2.9% — it looked like a victory. The test came back p = 0.44. In plain terms:there's a 44% chance that "victory" was pure luck.Without that step, I'd have taken the 2.9% and changed the product.

Rule 4: Don't test just one parameter setting

Announce a result from a single configuration and the easiest rebuttal is "you just didn't tune it properly."

So I swept the "how many days of history does the model see" setting from 64 to 720, five values × three metals × four horizons = 60 head-to-head comparisons, with the test points held completely fixed and only this one variable changing.

Rule 5: Break down which half it won on

I also built a hybrid version:the new model's median as the center, paired with the interval width from the old method.

This answers a diagnostic question — if the new model wins, does it win at reading direction, or at estimating width? The two answers lead to completely different next steps.

3. Results

The headline first:across 60 comparisons, the new model won only 21. Worse than a coin flip.

Broken out by metal (referred to below as Metals A / B / C):

That 8.6% at p=0.055 on Metal B looks tempting — just shy of the threshold. But it's the best result I picked outafter sweeping five different configurations. Sweep enough settings and you'll always find one that comes close to significance — that's multiple comparisons, and it doesn't count. The honest phrasing is:even under the most favorable conditions, the new model only ties.

A detail that matters more than who won

The new model's intervals aresystematically too narrow。

at longer horizons. On Metal A's 20-day forecast, it claims an 80% interval but actually covers only 69.7%; the old method reached 77.7%.

For anyone running a business, an error in this direction is far worse than losing by a few percentage points.A tool that understates risk will leave you without margin exactly when you need it.An interval that honestly says "I'm not sure" is more useful than one that looks precise but misses.

Breakdown: which half it lost on

The hybrid version gave the answer: Metal A's losses came almost entirely fromthe center being off— that is, the model tried to predict direction and predicted it wrong.

I tested this separately: comparing the new model's median against the completely brainless guess of "tomorrow equals today." On Metal A,the new model was significantly worse than "tomorrow equals today"(20-day: −10.5%, p=0.003).

In other words, on this problem, the part where it "thinks" is a net negative.

4. Why this happened

The model isn't bad. TimesFM beats every comparable model on the benchmarks, and those results are real.

The issue isthe nature of the problem。

What foundation models are good at is learningrecurring patternsfrom a trillion time points — cycles, seasonality, trends, holiday effects. Restaurant footfall, power grid load, website traffic: these carry a lot of learnable structure, and the model performs well on them.

But international metal prices are close to arandom walk: tomorrow's price ≈ today's price + a shock nobody can anticipate. Series like this haveno pattern to learn. The model walks in with skills honed elsewhere, finds nothing to apply them to, and those skills become noise — it "sees" patterns that aren't there and pushes the median in the wrong direction.

And this is exactly why the old formula survives:it gave up on guessing direction from the start.It does one thing — estimate how far things will swing. And volatility is one of the few parts of a financial series that genuinely has structure and genuinely can be predicted (violent moves today usually mean violent moves tomorrow; this is called volatility clustering).

A narrow method that does one thing, where that one thing happens to be doable, beat a general method that does everything, where none of it applies.

I'm not the only one to find this. A 2026 study ran the same family of foundation models on US equity returns and reached the same conclusion: these models rank beautifully against each other, buttheir margin over "tomorrow equals today" is small enough to be statistically insignificant. Another paper is more direct: some models show negative out-of-sample explanatory power and directional accuracy below 50%.

I just confirmed it once more on my own data.

5. The real takeaway had nothing to do with AI

The first run of the experiment blew up outright.

Digging in, I found two problems in the client's source file — neither of them an "error," both of themtraps that look perfectly normal：

First:0 those aren't missing values, those are days the market was closed. The source file has a row for every calendar day, with the price column set to 0 on weekends and holidays. Over four-plus years that's more than 400 rows. Any program that computes returns straight off that column will silently produce garbage — or, as in my case, take the log of 0 and detonate on the spot.

The explosion was actually a gift.Had those zeros happened to be filled in with some other number, the program would have run smoothly and produced a very professional-looking report that was wrong from end to end.

Second, two entries had their thousands separator typed as a decimal point. A five-digit price became a two-digit one, off by a factor of a thousand. On a chart that's two bars drilling through to the earth's core; statistically it's two days of ±690% moves.

The second problem has a follow-up worth recording: those two errors hadno effecton the numbers the tool displays today — because that old formula only remembers roughly the last 50 days and had long since forgotten them. But another machine learning model on the same data, one that consumesthe entire history, learned those two insane days in full.

The same dirty data does completely different amounts of damage to different methods.There's no general rule here; you have to check case by case.

The lesson is simple, but worth carving into a wall:a new model won't fix your old data.Feed dirty data to the strongest model of 2026 and it will generate the wrong answer in the most advanced way possible.

6. For the people making the decision

The whole experiment took half a day. The actual computation — three metals × 300 test points × five configurations — finished in under two minutes on an M1 Mac.

This kind of validation is extremely cheap now. Cheap enough that "we didn't do it" is no longer an excuse.

A few things you can use directly:

1. "First on the benchmark" is not "first on your data." Benchmarks measure general capability. Whether your problem has learnable structure determines whether that first-place finish means anything.
2. Before switching, score what you have now. If you can't state the baseline, you have no standing to talk about improvement. This step is usually the fastest and the most often skipped.
3. An A/B comparison without a significance test is mostly a comparison of luck. Especially when the test windows overlap. That "won by 2.9%, p=0.44" is a textbook case.
4. Check whether intervals cover what they claim to cover; don't look only at average error. A tool that systematically understates risk is more dangerous than one that honestly says it isn't sure.
5. Check the license before testing performance. The strongest version may not be licensed for commercial use at all.
6. Cleaning your data usually pays better than switching models. The only definitively useful output of this experiment was catching two mistyped numbers and 400-plus fake zeros.

One last thing.

This was afailed experiment— if "success" means ending up with a better tool. But what half a day bought was: a firm conclusion that no switch is needed, a comparison framework that can be re-run, two data errors caught, and a process I can apply directly the next time someone asks "should we switch to the new X?"

Negative results are worth keeping.What they save isn't this half day; it's every future occasion when you hear "there's a powerful new thing out" and no longer have to guess from scratch.