Financial markets generate an enormous amount of data.

But once you adjust for correlation, regime changes, and noise, the effective amount of information is much smaller than it first appears.

A First Pass: How Big Is Financial Data Really?

In the era of large language models, it’s easy to be dazzled by scale. Trillions of tokens feed today’s frontier models, enabling generalization across a vast landscape of tasks. By comparison, financial data—especially high-frequency data—can feel similarly enormous, with markets generating observations every second across thousands of assets.

But raw data size is not the same as usable data size. In finance, the structure of the data aggressively compresses the effective information content. Correlation, non-stationarity, and noise all shrink the signal contained in even the largest datasets, and the longer the time horizon we look at, the more pronounced the effect becomes.

As a rough benchmark, suppose we look at liquid U.S. equities:

• About 4,000 reasonably liquid stocks
• Observations every second
• ≈ 24 billion observations per year
• ≈ 500 billion observations over 20 years

On paper, that looks comparable to large-model training corpora. In practice, it isn’t.

Correlation: Thousands of Assets, But Far Fewer Independent Samples

Although equity markets trade thousands of instruments, these assets are highly correlated. Much of the cross-sectional structure can often be explained by a limited number of latent factors.

If a factor model has ~200 effective dimensions:

• Raw samples per year: ~24 billion
• After adjustment: ~1.2 billion

We’ve already lost an order of magnitude.

Regime Change

Markets evolve — microstructure, regulation, participants, and volatility all shift.

If only the most recent 5 years are relevant:

• Effective sample size drops from ~500 billion → ~5 billion
  
  

Noise

Noise is the dominant issue in nonparametric estimation.

A local averaging estimator gives

 Where:

•  = noise level
• = dimension
• = sample size
• is the distance over which averaging is performed
• are constants 

This shows a classic bias–variance tradeoff, i.e., choosing a smaller ball reduces bias (not sampling from points where function values have diverged further), but it reduces the sample size over which averaging occurs, increasing variance.

Optimizing yields:

A short heuristic derivation of this particular instance of the bias-variance tradeoff is included in the appendix.

The key lesson is simple but important:

Noise acts multiplicatively on sample complexity. If the noise level increases by a factor of 10, the number of samples required to achieve the same target error grows accordingly. In other words, noise further shrinks the effective size of a dataset—potentially by another order of magnitude or more.

Returning to our earlier estimates: even a mild noise penalty can reduce an already-compressed dataset’s effective size from billions of observations to something closer to a few hundred million usable samples.

The Impact on Modelling

Pooling Across Assets (Carefully)

One natural response to limited time-series data for any single asset is to pool information across assets: fit models jointly across a cross-section of stocks, share parameters hierarchically, or use factor structures that tie together related names. This effectively increases the amount of data available to estimate common structure, at the cost of assuming that assets share similar dynamics.

Done well, pooling can dramatically improve effective sample size—for example, learning a common mapping from features to returns across a large universe, while allowing for asset-specific intercepts or random effects. Done poorly, it ignores genuine heterogeneity or regime differences and simply averages away important structure. The same correlation and noise issues that shrink your data also limit how aggressive you can be when pooling.

Denoising Is a Core Skill

Whether via shrinkage, factor modeling, or filtering, denoising  noise is a central modelling challenge. Much of the art in quantitative finance consists of extracting a small, stable signal from a large, unstable, and noisy flow of observations.

Structure Helps Beat Dimensionality

Models that incorporate structure—economic constraints, invariances, dynamics, monotonicity, or symmetries— can beat naive sample complexity limits by effectively lowering the problem’s intrinsic dimension.

Choosing the Best Approach for the Job at Hand

The modelling strategies mentioned here are all strategies that we employ routinely, but even in finance, we work across a variety of different regimes depending on the asset class, the time horizon, and the nature of the trading. You can expect to see more discussions of these modelling issues in the blog.

Conclusion

High-frequency markets generate immense quantities of data, but once we adjust for correlation, regime changes, and noise, the effective amount of information is a small fraction of what the raw counts suggest.

• This is not a disadvantage so much as a fact of the domain. Models that succeed in finance are those that account for structure, impose regularization, exploit invariances, and extract signal from noisy, evolving environments.
• Finance may produce billions of data points, but statistically speaking, it lives in a world where usable data is precious.

Appendix

Let be i.i.d. samples from density.

We observe:

Where

Assume

Define:

• Ball:  
• Count:

Estimator: 

This defines an estimator for at  using local averaging. The argument below walks through the classic bias-variance trade-off for this simple estimator in a very direct way. Choosing a smaller ball reduces bias (not sampling from points where function values have diverged further), but it reduces the sample size over which averaging occurs, increasing variance.

Error Decomposition

Let:

Then:

Noise term has variance:

So:

]

Density Scaling (Heuristic)

For small(heuristically using that concentrates around its mean for small)

, where is the volume of the unit ball in

So: 

Where is a constant that depends on  

Optimal Radius

Optimal  

Resulting error:
 

References

• Vapnik — Statistical Learning Theory
• Steinwart & Christmann — Support Vector Machines
• Cucker & Zhou — Learning Theory
• Tsybakov — Nonparametric Estimation