Computers and numbers

If you are new to programming, then this lesson may contain some surprising information. But even if you are used to writing programs, computers can do unexpected things with numbers. The purpose of this lesson is to shed some light on some of the mysteries and quirks you can encounter.

These mysteries exist independently of the programming language, though one programming language may be better at isolating you from them than another. The problem is that computers can not deal with the numbers we are used and in the way we are used to.

For instance (*):

# Compute 1 million times 1 million
% puts [expr {1000000*1000000}]
-727379968
To most people's surprise, the result is negative! Instead of 1000000000000, a negative number is returned.

Important note: I used Tcl 8.4.1 for all examples. In Tcl 8.5 the results will hopefully be more intuitive, as a result of adding so-called big integers. Nevertheless, the general theme remains the same.

Now consider the following example, it is almost the same, with the exception of a decimal dot:

# Compute 1 million times 1 million
% puts [expr {1000000*1000000.}]
1e+012
The reason is simple, well if you know more about the background of computer arithmetic:

Tcl's strategy

Tcl uses a simple but efficient strategy to decide what kind of numbers to use for the computations:

What are those mysteries and quirks?

Now some of the mysteries you can find yourself involved in. Run the following scripts:

#
# Division
#
puts "1/2 is [expr {1/2}]"
puts "-1/2 is [expr {-1/2}]"
puts "1/2 is [expr {1./2}]"
puts "1/3 is [expr {1./3}]"
puts "1/3 is [expr {double(1)/3}]"

The first two computations have the surprising result: 0 and -1. That is because the result is an integer number and the mathematically exact results 1/2 and -1/2 are not.

If you interested in the details of how Tcl works, the outcome q is determined as follows:

a = q * b + r
0 <= |r| < |b|
r has the same sign as q

Here are some examples with floating-point numbers:

set tcl_precision 17  ;# One of Tcl's few magic variables:
                      ;# Show all decimals needed to exactly
                      ;# reproduce a particular number
puts "1/2 is [expr {1./2}]"
puts "1/3 is [expr {1./3}]"

set a [expr {1.0/3.0}]
puts "3*(1/3) is [expr {3.0*$a}]"

set b [expr {10.0/3.0}]
puts "3*(10/3) is [expr {3.0*$b}]"

set c [expr {10.0/3.0}]
set d [expr {2.0/3.0}]
puts "(10.0/3.0) / (2.0/3.0) is [expr {$c/$d}]"

set e [expr {1.0/10.0}]
puts "1.2 / 1.0 is [expr {1.2/$e}]"

While many of the above computations give the result you would expect, note however the last decimals, the last two do not give exactly 5 and 12! This is because computers can only deal with numbers with a limited precision: floating-point numbers are not our mathematical real numbers.

Somewhat unexpectedly, 1/10 also gives problems. 1.2/0.1 results in 11.999999999999998, not 12. That is an example of a very nasty aspect of most computers and programming languages today: they do not work with ordinary decimal fractions, but with binary fractions. So, 0.5 can be represented exactly, but 0.1 can not.

Some practical consequences

The fact that floating-point numbers are not ordinary decimal or real numbers and the actual way computers deal with floating-point numbers, has a number of consequences: