Round 7 · FNO's structural limitation
FNO's architecture pipeline is: FFT → learnable frequency weights → inverse FFT. This closes the operation over the input domain — everything maps from positions {0..N_in} back to {0..N_in}. What happens when you ask it to predict values at new positions {N_in..N_in+N_out}, outside the domain it was trained on? This would be required for any real extrapolation task.
FNO is like a very skilled interpolator: given measurements at equally spaced points from 0 to 10, it can reconstruct the values at any point between 0 and 10 with high accuracy. But if you ask "what's the value at 11?" — a step outside the grid — it has no answer. Its entire architecture is defined over the input grid. It can't even represent positions outside it.
A Transformer, by contrast, can in principle extrapolate: it learns the pattern and can apply it at new positions, though whether it does so accurately depends on what it learned.
A periodic signal extrapolation task: train on signal values at positions {0..100}, predict values at positions {101..200}. Effectively asking both models to extend a periodic pattern beyond what they've seen.
Performance measured as relative L2 error on the out-of-domain positions.
| Model | Relative L2 error (extrapolation) |
|---|---|
| Transformer | 0.0076 |
| FNO | 1.000 |
FNO achieves exactly 1.0 relative L2 error — equivalent to predicting zero everywhere. It learns nothing about the extrapolation task despite 10K training steps.
FNO structurally cannot extrapolate. This is not a training failure or a hyperparameter issue — it's an architectural impossibility. The rfft → frequency weights → irfft pipeline maps input positions to output positions via learned frequency coefficients. The inverse FFT is defined over the input grid only. There is no mechanism for producing values at new positions.
This is the clean, definitive characterisation of FNO's niche: it is a closed-domain operator. It excels when input and output share the same spatial grid (solving PDEs, operator approximation). It has zero capability for any task requiring extrapolation, sequence generation, or language.
The Transformer succeeds at 0.0076 — a good result — because it learns the periodicity pattern and can apply it at new positions. Transformers are general-purpose sequence models; FNO is a physics simulation tool.
One sentence: FNO predicts zero outside its training domain, always, by construction.
- Could a cross-attention decoder added on top of FNO's internal representation allow it to produce values at new positions — essentially grafting an extrapolation head onto a powerful interpolator?