Statistics, Probability, and Noise

Describing a mess numerically

Almost every problem in this field is “there is something I want, mixed with something I don’t.” You cannot make progress until you can say how much of the unwanted thing there is. “A bit noisy” is not actionable; “signal-to-noise ratio of fifty” is.

The mean locates a signal; the standard deviation says how much it moves. Averaging the raw deviations always gives exactly zero — positives and negatives cancel by construction — so you square first. That also weights large excursions more heavily, and it corresponds to power, which connects the statistics to the physics.

Divide by N − 1, not N. You estimated the mean from the same data, so your samples sit slightly closer to it than to the true mean, and the deviations come out too small. One degree of freedom was spent on the estimate.

Why noise is Gaussian

The normal distribution is not a convenient assumption imposed for tidiness — it is what actually happens when many small independent contributions add up, which is what physical noise nearly always is.

Roughly 68% of samples fall within one standard deviation, 95% within two, 99.7% within three. Worth memorising: it converts a standard deviation into an intuition instantly.

The square root of N, and where it fails

Averaging N samples reduces uncertainty by the square root of N. To halve the noise you need four times the data; for a factor of ten, a hundred times. Past a point the answer is never “average for longer” — it is “find a better measurement.”

But that law assumes the samples are independent. Distributions alone don’t capture that: sort a noise signal and its histogram, mean and standard deviation are all unchanged, yet it is now a smooth ramp.

Precision and accuracy are also different things. Precision is repeatability and comes from random error, so averaging helps. Accuracy is correctness and comes from systematic error, so averaging does nothing — you simply become more confident about a wrong number.

Key points

1. A discrete signal is written x of n with square brackets; a continuous signal is x of t with round brackets. For N samples, the sample numbers run from zero to N minus one, never one to N.
2. The mean is the sum of the samples divided by N. It locates the signal and says nothing about how much it varies. In electrical terms it is the DC component.
3. Averaging raw deviations from the mean always gives exactly zero, because positive and negative deviations cancel by construction. Squaring first removes the signs, weights large excursions more heavily, and corresponds to power.
4. Variance is the average squared deviation and represents the power of the fluctuation. Standard deviation is its square root, which restores the original units so it can be compared against the mean.
5. Divide by N minus one, not N, because the mean was estimated from the same data. The estimate moved toward your samples, so the deviations came out too small, and N minus one is exactly the right correction. With large N the difference is negligible; with small N it matters.
6. RMS squares without subtracting the mean first. If the signal has no DC component, RMS and standard deviation are the same number.
7. Signal to noise ratio is the mean divided by the standard deviation. Coefficient of variation is the reciprocal as a percentage — a CV of two percent is an SNR of fifty.
8. Running statistics need only three accumulators: the count, the sum, and the sum of squares. But when the mean is much larger than the standard deviation, this method subtracts two nearly equal large numbers and loses precision badly.
9. The signal is not the process. Your computed statistics describe your samples exactly, but they only estimate the parameters of the underlying process, and those estimates are themselves random.
10. The uncertainty in an averaged measurement falls as sigma over the square root of N. Halving the noise requires four times the data; a factor of ten requires a hundred times. Past a point, the answer is a better measurement, not a longer one.
11. A histogram counts how many samples took each value, and its bars sum to N. It makes the standard deviation visible as width, and it lets you compute statistics by looping over a few hundred bins instead of millions of samples.
12. The probability mass function is what the histogram converges to as N goes to infinity, normalized to sum to one. For continuous signals the equivalent is the probability density function, where probability is area under the curve rather than height of a bar.
13. The normal distribution is fixed entirely by its mean and standard deviation. Roughly sixty eight percent of samples fall within one standard deviation, ninety five percent within two, and ninety nine point seven percent within three. Its tails approach zero but never reach it.
14. The central limit theorem says that summing many independent random contributions gives a normal distribution regardless of what the individual contributions looked like. Physical noise is usually exactly such a sum, which is why Gaussian noise is everywhere.
15. To generate Gaussian noise, add twelve uniform random numbers between zero and one and subtract six. Twelve is chosen because uniform noise has variance one twelfth, and variances add, so twelve of them give variance one exactly. The result is bounded at plus or minus six sigma, unlike a true Gaussian.
16. The distribution of values says nothing about their ordering. Sorting a noise signal into ascending order leaves the histogram, mean, and standard deviation completely unchanged while turning the signal into a smooth ramp.
17. White noise is uncorrelated from sample to sample, which makes its energy flat across all frequencies. Thermal noise and shot noise are both very nearly white.
18. Pink, or one over f, noise carries more energy at low frequencies and appears in the time domain as slow drift. Because its samples are not independent, the square root of N averaging law does not hold — which is why long measurements so often fail to improve as expected. Lock-in detection sidesteps it by modulating the measurement up to a frequency where the noise is white again.
19. Precision is repeatability and comes from random error; accuracy is correctness and comes from systematic error. Averaging fixes precision only. An averaged measurement with a calibration bias becomes a very precise estimate of the wrong answer.