We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
‹ Previous Topic | Next Topic ›
Page Index Toggle Pages: 1
Math Question (Read 1056 times)
majg
Math Question
May 8th, 2010, 5:48am  
I have been struggling with this all day. I can't understand why the variables "test" and "test2" give two different answers in this simple program:

----------------------------------------

float test,test2;


void setup(){

 test=(6317180.0-6316832.8)*2.0;
 test2=(347.2)*2.0;  
 println (test+" "+test2);  

}


----------------------------------

When I run this program i get
test=694.0
test2=694.4



Can anyone tell me why these two variables would be set to different values in this program? Am I doing something wrong with the order of operations? I am at a loss.
Back to top
 
 
majg
Re: Math Question
Reply #1 - May 8th, 2010, 5:56am  
Here is a simpler program that gets to the root of the issue I am describing:

-------------------------

float temp;

void setup(){
 temp=6317180.0-6316832.8;
 println (temp);  
}


------------------------------

This outputs the value 347.0 but the calculation should yield 347.2
Back to top
 
 
PhiLho 
Re: Math Question
Reply #2 - May 8th, 2010, 6:31am  
Looks like you have hit the limit of the resolution of the float type...
I made various tests, I found out that 3317180.2 is printed as such while 4317180.2 is displayed as 4317180.0...
float has a limited number of significant digits.
You can use double if you need more, but its usage in Processing, particularly as literal numbers, is a bit hard: there is no 'd' symbol like we have 'f' and Processing forces all numbers with a decimal point to be float, adding a f at compile time.
The best workaround here is to have a .java file where the constants are defined...

Note also the limits of representing float numbers in digital notation: when I do the subtraction in my calculator, I get 347.199999999999818 while Java with doubles gives 347.20000000018626...
Back to top
 
WWW  
majg
Re: Math Question
Reply #3 - May 8th, 2010, 6:39am  
Thank you for your response. I am creating a mapping application and the large numbers are co-ordinates in the map projection we are using.

I think I will try and translate the information to get it closer to the origin before I work with it in Processing. This will be a major effort. It took me over a day to track down what was causing the issue in my program in the first place.

I had no idea there was a limit to the resolution of a float type.

Thanks again.
Back to top
 
 
Lie Ryan
Re: Math Question
Reply #4 - May 8th, 2010, 2:33pm  
if you need to compare two float/double values, you must use something like:
if (abs(A - B) < eps) ...

where eps is the "tolerance".

Otherwise if something like this is unsuitable for you, you would need to use either a fixed-point BigDecimal (which you can control exactly how to round) or a Rational (fractions, exact value, but cannot represent irrational number, AFAICT doesn't come with Java standard library).
Back to top
 
 
Page Index Toggle Pages: 1
‹ Previous Topic | Next Topic ›