how to fix this simple addition 9.6-0.2

edited January 2018 in Programming Questions

here is the code

float x = 9.6;

x -= 0.2;

println(x); // the answer shown 9.400001

Tagged:

Answers

  • how would your write 1/3 as a decimal number?

    that's the same problem computers have (but they use binary)

  • // Forum.Processing.org/two/discussion/26195/
    // how-to-fix-this-simple-addition-9-6-0-2#Item_2
    
    // GoToLoop (2018/Jan/31)
    
    int x = 96;
    
    void setup() {
      frameRate(1);
    }
    
    void draw() {
      background((color) random(#000000));
      print(nf(.1 * (x -= 2), 0, 1) + " - ");
    }
    
  • Approaches:

    1. filter your output, e.g. by rounding or dropping precision
    2. use higher precision, e.g. double, which may work to an extent in some cases (although it has the same problem)
    3. use a special data type, like java.math.BigDecimal
    4. use integers, multiplied to the appropriate level of precision. This gives precise answers, but there can be headaches converting for storage or output.

    This last approach is what @GoToLoop is doing. So, if you wanted to use values with two digits of decimal precision, then:

    float x = 9.6;
    x -= 0.2;
    

    ...would become:

    int x = 960;
    x -= 20; // 940, aka 9.4
    
Sign In or Register to comment.