Website Search

Support Center - Help topic

Home > Support Center > Search > Help Topic
Contact Us
Can't find what you're looking for in our Tech Support Library? Send us a message.
Contact Us
Floating Point Rounding Errors
Tags: FAQ Data

Generally, “floating point error” refers to when a number cannot be stored in the IEEE floating point representation.

Integers are stored with the right-most bit being 1, and each bit to the left being double that (2,4,8,...). It's easy to see that this can store any integer up to 2^n, where n is the number of bits.

The mantissa (decimal part) of a floating point number is stored in a similar way, but moving left to right, and each successive bit being half of the value of the previous one.

Thus, numbers like 0.5 (1/2) are easy to store, but not every number <1 can be created by adding a fixed number of fractions of the form 1/2, 1/4, 1/8, ...

A really simple example is 0.1, or 1/10. This can be done with an infinite series, but whenever a computer stores 0.1, it's not exactly this number that is stored.

Content Created by David Franzwa