Computing desk
< November 3	<< Oct | November | Dec >>	November 5 >

Welcome to the Wikipedia Computing Reference Desk Archives
The page you are currently viewing is a transcluded archive page. While you can leave answers for any questions shown below, please ask new questions on one of the current reference desk pages.

It's frequently stated that computer floating point is "inherently inaccurate", because of things like the way the C code

float f = 1. / 10.;
printf("%.15f\n", f);

tends to print 0.100000001490116.

Now, I know full well why it prints 0.100000001490116, since the decimal fraction 0.1 isn't representable in binary. That's not the question.

My question concerns those words "inherently inaccurate", which I've come to believe are, well, inaccurate. I believe that computer floating point is as accurate as the numbers you feed into it and the algorithms you use on them, and that it is also extremely precise (120 parts per billion for single precision, 220 parts per quintillion for double precision.) So I would say that floating point is not inaccurate, although it is indeed "inherently imprecise", although that's obviously no surprise, since its precision is inherently finite (24 bits for single precision, 53 bits for double, both assuming IEEE 754).

The other thing about binary floating point is that since it's done in base 2, its imprecisions show up differently than they would in base 10, which is what leads to the 0.100000001490116 anomaly I started this question with. (Me, I would say that those extra nonzero digits …1490116 are neither "inaccurate" nor "imprecise"; they're basically just false precision, since they're beyond the precision limit of float32.)

But I can see from our page on accuracy and precision that there are a number of subtly different definitions of these terms, so perhaps saying that floating point is "inherently inaccurate" isn't as wrong as I've been thinking.

So my question is just, what do other people think? Am I missing something? Is saying "floating point is inherently inaccurate" an informal but inaccurate approximation, or is it meaningful? —scs (talk) 13:50, 4 November 2024 (UTC)[reply]

Wiktionary: accurate says "accurate ... Telling the truth or giving a true result; exact", but float is not exact so it is not accurate in that sense. 213.126.69.28 (talk) 14:10, 4 November 2024 (UTC)[reply]

See also Floating-point arithmetic § Accuracy problems. What is not mentioned, is the problem that little inaccuracies can accumulate. For example, consider this code:

x = 0.1
for i in range(62):
  x = 4*x*(1-x)

The true mathematical value computed, rounded to 15 decimals, is 0.256412535470218, but the value computed using IEEE floating point arithmetic will come out as 0.988021660873313.  --Lambiam 16:06, 4 November 2024 (UTC)[reply]

@Lambiam: Cute example. (Does it have a name?) Lately I've been impressed at how often cascading precision loss isn't a problem, although it's certainly one of the things you always have to watch out for, as here. (It's why I said "as accurate as... the algorithms you use on them".) —scs (talk) 14:28, 5 November 2024 (UTC)[reply]

See Logistic map § Solution when r = 4.  --Lambiam 20:01, 5 November 2024 (UTC)[reply]

@Lambiam: Aha: An "archetypal example of complex, chaotic behaviour". So we shouldn't be too surprised it's particularly sensitive to slight computational differences along the way... :-) —scs (talk) 22:48, 5 November 2024 (UTC)[reply]

The basic floating point operations are as accurate as it possible for them to be and the specification normally talks about precision which measures the unavoidable deviation from being completely accurate. But no-one is going to carp about calling it inaccurate! NadVolum (talk) 16:57, 4 November 2024 (UTC)[reply]

@NadVolum: I am here to prove you wrong, because that is exactly what I am carping about! :-) —scs (talk) 17:31, 4 November 2024 (UTC)[reply]

There are two issues with floating point numbers that are both wrapped up in "they are not accurate." I personally never say that. What I say is that the same number can be stored in memory different ways. For example, a human can tell you that 10e4 and 100e3 are the same number. But, to a computer, they are different. It doesn't parse out the value and computer 100000 and 100000. It compares exactly what it sees. 10e4 and 100e3 are not the same. Of course, computers use binary, not decimal, but that isn't the point. The point is that you have the same value being stored in different ways. You, as the human, don't control it. As you do operations on a value in memory, the value updates and the exact way it is stored can change. Separately, floating point numbers do tend to drift at the very end. So, 3.000000... can become 2.99999999... or 3.00000000....0001. That is not "wildy" inaccurate. But, 2.99999... is not the same value as 3.00000. In the end, why do we care? It comes down to programming. If you have two floating point variables x and y and you want to know if they are the same value, you can't simply compare x==y and hope to get the right answer. What if x is 3.00000.... and y is 2.99999...? Instead, you do something like abs(x-y)<0.000000001. Then, if there is a little drift or if the two numbers are the same value but stored slightly different, you get the correct answer. This came to a head way back in the 80s when there was a flame war about making the early c++ compiler automatically convert x==y to abs(x-y)<0.0000000000000000001. But, what I believe you are arguing is that memory storage should be fixed instead of the programming so the numbers are always stored in the exact same format and there is never ever any drift of any kind. That would be more difficult in my opinion. 17:24, 5 November 2024 (UTC)

That's not what I was saying, but thanks for your reply. [P.S. 10e4 and 100e3 are the same number, in any normalized floating-point format; they're both stored as 1.52587890625 × 2¹⁶, or more to the point, in binary as 1.10000110101₂ × 2¹⁶.] [P.P.S. Testing fabs(x - y) < some_small_number is not a very good way of doing it, but that wasn't the question, either.] —scs (talk) 20:12, 5 November 2024 (UTC)[reply]

A fundamental problem is that numbers represented in numerical form, whether on paper or in a computer, are rational numbers. In print, we typically have numbers of the form ${\displaystyle m\cdot 10^{n},}$ where ${\displaystyle m}$ and ${\displaystyle n}$ are whole numbers. In computers, ${\displaystyle m\cdot 2^{n}}$ is more common. However, most numbers are not rational. There is no way to compute the exact value of, for example, ${\displaystyle {\frac {\ln 24591257752}{\pi {\sqrt {58}}}}.}$ There is no known algorithm that will decide in general whether the mathematical value of such an expression with transcendental functions is itself transcendental, so at a branch asking whether this value is equal to ${\displaystyle 1}$ we have no better recourse than computing this with limited accuracy and making a decision that may be incorrect.

BTW, "comparison tolerance" was a feature of APL.^[1] The "fuzz", as it was colloquially called, was not a fixed constant but was a system variable with the strange name ⎕ct to which a user could assign a value. The comparison was more complicated than just the absolute difference; it was relative to the larger absolute value of the two comparands (if not exactly equal as rational numbers).  --Lambiam 21:37, 5 November 2024 (UTC)[reply]

I actually don't agree with the claim that numbers represented in numerical form...are rational numbers. If you're talking about the main use for them, namely representing physical quantities, they aren't rational numbers, not conceptually anyway. Conceptually they're "fuzzy real numbers". They don't represent any exact value, rational or otherwise, but rather a position along the real line known with some uncertainty. --Trovatore (talk) 22:36, 5 November 2024 (UTC)[reply]

I am not talking here about numbers representing values, but numbers being represented by (finite) numerical forms, such as decimal expansions or floating-point representations. A bit pattern can be used to represent a number. Humans can interpret such numbers as representing the cube root of 2 or the size of the observanle universe, but the numbers representing these irrational or fuzzy values are themselves rational.  --Lambiam 19:33, 11 November 2024 (UTC)[reply]

I don't agree. Floats are a different thing; they aren't rationals or irrationals. They aren't really real numbers at all. They're a different structure, which approximates the structure of the reals. Sure, there's a standard interpretation of them as rationals, but that interpretation is not really faithful, and the objects that the floats are intended to represent are not those rationals, particularly, but rather an approximation to a real value. --Trovatore (talk) 20:00, 11 November 2024 (UTC)[reply]

Right. If you say that "floating-point numbers are rational numbers", that's true in that, for example, the single-precision floating-point number you get for 0.1 is exactly 13421773/134217728. But it's pretty badly false if it leads you to conclude that you should be able to represent numbers like 1/3 and 2/7 exactly. The best description really is that floats are an imperfect, finite-precision approximation of real numbers. —scs (talk) 21:35, 11 November 2024 (UTC)[reply]

(Taking the above comments as read:) Most of the significant issues with floating point numbers are programming errors, often slightly subtle ones. It is possible in the majority of cases to use rational numbers as an alternative, only producing a floating point representation when display or output is needed in that form. Again for the majority of cases this would be a good solution, but for a very few cases the numerator and denominator could be very large (the logistic map example above would require ~ 2^62 digits (cancelling helps but a little)), and for compute intensive cases the general slowdown could be important. All the best: Rich Farmbrough 12:00, 6 November 2024 (UTC).[reply]

It's conceptually wrong, though, in most cases. Floating-point numbers usually represent physical quantities, and physical quantities aren't conceptually rational numbers. What we want is something that approximates our state of knowledge about a real-valued quantity, and floating point is the closest thing we have to that in wide use. (Interval arithmetic would be a *little* closer but it's a pain.)

That doesn't actually prove that you couldn't get good solutions with rationals, but it's kind of an article of software-engineering faith that things work best when your data structures align with your concepts. I don't know if that's ever been put to a controlled test. --Trovatore (talk) 18:25, 6 November 2024 (UTC)[reply]

Sure you are absolutely right for representing physical quantities in most cases - in chaotic scenarios whatever accuracy you measure with might not be enough regardless of the way you calculate. However computing is used for many purposes including mathematics. It's also used in ways where careful application of floating point will bring an acceptable answer, but naive application won't. All the best: Rich Farmbrough 23:03, 6 November 2024 (UTC).[reply]

Incidentally here's a perl program that gets a more accurate answer to the logistic map problem above, using floating point:

use Math::BigFloat;
my $x = Math::BigFloat->new('0.1');
$x->accuracy(50);  # Set desired precision
for (my $i = 0; $i < 62; $i++) {
    $x = 4 * $x * (1 - $x);
}
print "$x\n";

All the best: Rich Farmbrough 23:07, 6 November 2024 (UTC).[reply]

I guess you meant "...using arbitrary precision floating point" (i.e. perl's "BigFloat" package).

But this ends up being a nice illustration of another principle of numerical programming, namely the importance of using excess precision for intermediate results. Evidently that call to accuracy(50) sets not only the final printout precision, but also the maximum precision carried through the calculations. So although it prints 50 digits, only 32 of them are correct, with the rest lost due to accumulated roundoff error along the way. (The perl program prints 0.25641253547021802388934423187010674334774769183115, but I believe the correct answer to 50 digits — at least, according to my own, homebrew multiprecision calculator — is 0.25641253547021802388934423187010494728798714960746.) —scs (talk) 04:02, 7 November 2024 (UTC), edited 13:45, 8 November 2024 (UTC)[reply]

My original question was about how to describe the imperfections, not what the imperfections are or where they come from. But since someone brought up rational numbers, my own take is that there are several models you might imagine using for how computer floating point works:

• rational numbers
• decimal fractions
• scientific notation
• real numbers

Now, although real numbers are arguably what floating-point numbers are the farthest from — there are an uncountably infinite number of real numbers, but "only" 18,437,736,874,454,810,626 floating-point ones — it's the real numbers that floating-point at least tries to approximate. The approximation is supremely imperfect — both the range and the precision are strictly limited — but if you imagine that floating-point numbers are approximate, limited-precision renditions of certain real numbers, you won't go too far wrong. (As for the rationals, it's not strictly wrong to say that "floating point numbers are rational numbers", because the floating point numbers are indeed a proper subset of the rational numbers — but I don't think it's a useful model.) —scs (talk) 13:35, 8 November 2024 (UTC)[reply]

Actually, it is "strictly wrong to say that 'floating point numbers are rational numbers'". At least there is no injective ring homomorphism from the floats into the rationals, because the arithmetic is different. Of course the floats aren't literally a ring in the first place, but you can work out what I mean. --Trovatore (talk) 19:35, 8 November 2024 (UTC)[reply]

@Trovatore: I'm just a simple computer programmer who doesn't even know how to spell — in jek tiv ring homeopathy (?) — but yes, now that you mention it, even I can work out that between the fact that they're barely closed, don't always have inverses, and aren't always associative, floating point numbers aren't rationals in the group-theoretical sense. :-) (I was agreeing with Lambiam insofar that every normal and subnormal floating-point number is trivially representable as p/q, albeit with q always 2ⁿ.) —scs (talk) 15:08, 11 November 2024 (UTC)[reply]