Numbers, Precision, and Speed

A number format is a choice about what you can express

Sixteen bits gives 65,536 patterns and that is all. Whatever scheme you invent, you are choosing which 65,536 numbers you can represent; every other number in the universe must be approximated by one of yours.

Floating point’s danger is never its range, which is astronomically generous — ten to the 308 is vastly more than the atoms in the observable universe. It is precision, which stays constant in relative terms while the absolute gap grows with magnitude.

Where it goes wrong

Rounding errors are roughly random in direction, so they accumulate as the square root of the operation count rather than in proportion to it. A million operations costs about a thousandfold, not a millionfold.

Catastrophic cancellation is the exception that breaks that rule. Subtract two nearly equal numbers and the leading digits cancel while the absolute error survives at full size — so the relative error of the result can be worse by orders of magnitude. The running-variance formula from episode two is a textbook case, and it can return a negative variance, which is impossible.

Zero point one has no exact binary representation, for the same reason one third has no exact decimal one. Never test floats for equality. And build a time axis by multiplying the sample number by the interval rather than accumulating additions, so representation error cannot compound.

What actually makes code fast now

The book was written when arithmetic was the bottleneck, so counting multiplications was a good model of speed. That relationship has inverted. A main-memory access now costs roughly a hundred times a multiply, so the question is no longer “how many operations” but “how does this touch memory.”

Sequential access beats scattered access, because caches fetch contiguous blocks — an algorithm with worse arithmetic complexity but better locality frequently wins. SIMD lets one instruction operate on many values at once, and signal processing is exactly the workload it was built for, which is a strong reason to write simple regular loops.

And the timeless advice: measure before optimising. Intuition about where time goes is reliably wrong in the presence of caches, pipelines and optimising compilers.

Key points

1. A number format is a choice about which finite set of values you can represent. Sixteen bits gives sixty five thousand five hundred and thirty six patterns, and every other number must be approximated by one of them.
2. Fixed point spaces representable values evenly, giving constant absolute precision. Floating point spaces them proportionally, giving constant relative precision. Nearly every practical difference follows from this.
3. Two's complement is universal for signed integers because addition and subtraction work with identical hardware regardless of sign. Its range is asymmetric — the most negative value has no positive counterpart, and negating it overflows.
4. Integer overflow usually wraps, turning a peak that slightly exceeds full scale into a violent inversion. Saturating arithmetic clips instead, which is far safer for signals. Know which one you have.
5. Q notation is a convention, held only in the programmer's head, that reinterprets an integer as a fraction. Q fifteen makes a sixteen bit integer span minus one to just under plus one. Multiplying two Q fifteen values yields a Q thirty result, so every multiplication changes the format and must be shifted back — which is the source of most fixed point bugs. Block floating point shares one exponent across a block of samples, buying much of floating point's range at fixed point's cost.
6. IEEE single precision is thirty two bits: one sign, eight exponent with an offset of one hundred and twenty seven, and twenty three mantissa bits that give twenty four bits of precision thanks to an implied leading one. Its range is about ten to the minus thirty eight up to ten to the plus thirty eight.
7. Double precision is sixty four bits: one sign, eleven exponent with an offset of one thousand and twenty three, and fifty two mantissa bits giving fifty three bits of precision, spanning about ten to the minus three hundred and eight to ten to the plus three hundred and eight.
8. Floating point's danger is never its range, which is astronomically generous. It is precision, which stays constant in relative terms while the absolute gap between representable numbers grows with magnitude.
9. Zero point one has no exact binary representation, for the same reason one third has no exact decimal one. Never test floating point values for exact equality — compare against a tolerance. And build a time axis by multiplying the sample number by the interval rather than accumulating additions, so representation error cannot compound.
10. The standard defines signed zeros, infinities that propagate sensibly instead of wrapping, and NaN for undefined operations. NaN is contagious, which makes it an excellent debugging signal, and it is famously not equal to itself.
11. Denormals represent magnitudes below the normal range at reduced precision, and on many processors they are drastically slower. A filter whose state decays into the denormal range while processing silence can slow down by a factor of a hundred. Add tiny noise, or enable flush-to-zero.
12. Rounding errors are roughly random in direction, so they accumulate as the square root of the number of operations rather than in proportion to it. A million operations costs about a thousandfold, not a millionfold.
13. Catastrophic cancellation is the exception that breaks that rule. Subtracting two nearly equal numbers cancels the leading digits while preserving the absolute error, so the relative error of the result can be worse by many orders of magnitude. The running variance formula from episode two is a textbook case.
14. Adding a small number to a much larger one can change nothing at all, so summation order matters — smallest to largest is more accurate. Compensated summation recovers most of the loss for about double the cost.
15. Use floating point unless you have a specific reason not to. The historical speed penalty is gone on general purpose processors, and fixed point costs you manual scaling and overflow management throughout. Fixed point still wins on processors without a floating point unit, in FPGA designs, at extreme unit cost, and where bit exact reproducibility is required.
16. Single precision is usually sufficient for signal processing and is faster mainly because it halves memory traffic and cache footprint. Reach for double when accumulating over very many samples, when cancellation is unavoidable, or in recursive filters where errors feed back.
17. The bottleneck has moved from arithmetic to memory. A main memory access costs roughly a hundred times a multiplication, so counting multiplications is no longer a good model of speed.
18. Sequential memory access is much faster than scattered access because caches fetch contiguous blocks. An algorithm with worse arithmetic complexity but better locality frequently wins in practice.
19. SIMD lets one instruction operate on many values at once, and signal processing is the workload it was built for. Simple regular loops without unpredictable branching are what allow the compiler to use it.
20. Look-up tables, hoisting transcendental functions out of inner loops, and precomputing loop-invariant values all remain worthwhile. And measure before optimizing — intuition about where time is spent is reliably wrong.