Started By
Message

re: True/False Excel Statement Issue

Posted on 11/11/16 at 5:36 pm to
Posted by mdomingue
Lafayette, LA
Member since Nov 2010
30820 posts
Posted on 11/11/16 at 5:36 pm to
quote:

quote:


Excel uses floating point which is inexact. For example, 0.1 has no exact representation in binary notation, and it introduces errors into calculations.

This makes total sense now that I think about it. When my budgets are done for a project, I try to get every subtotal of Divisions 1-16 to be a whole dollar, but even when all of the subtotals read "$X.00", the bottom grand total will read $1,500,000.01. It annoys the shite out of me


That explanation is inaccurate. Machine and round off error can and do occur but it will not be at the 100ths level. The things you're seeing at x.00 and getting a x.01 in the sum is a result of showing 2 decimal places in the individual cells but you're formula is giving you numbers in the 3rd decimal places that add up to give you the .01.

You can fix this by adding the Round() function in you formula. Assuming you want two decimal points, the new formula should be should be Round(original formula, 2). That should give you exactly what you see when those cells are summed.
Posted by Spock's Eyebrow
Member since May 2012
12300 posts
Posted on 11/11/16 at 9:14 pm to
quote:

That explanation is inaccurate. Machine and round off error can and do occur but it will not be at the 100ths level.


I believe my explanation was correct WRT to the message I replied to.

However, it's not really correct to talk about "the 100ths level". As I said, Excel's FP numbers have 15 decimal digits of precision, and the errors I was talking about are limited typically to the least significant digit or two. It might be hundredths, millionths, or whatever, depending on where the decimal point is in a given number. These small errors can and do cause comparisons to fail for numbers computed in different ways.

quote:

quote:

This makes total sense now that I think about it. When my budgets are done for a project, I try to get every subtotal of Divisions 1-16 to be a whole dollar, but even when all of the subtotals read "$X.00", the bottom grand total will read $1,500,000.01. It annoys the shite out of me


The things you're seeing at x.00 and getting a x.01 in the sum is a result of showing 2 decimal places in the individual cells but you're formula is giving you numbers in the 3rd decimal places that add up to give you the .01.


That seems to be a different issue than I was responding to, and I agree that cumulative small errors aren't likely to make 1,500,000 equal to 1,500,000.01. That's due to errors in the 10th significant digit, not the 15th or so.
first pageprev pagePage 1 of 1Next pagelast page
refresh

Back to top
logoFollow TigerDroppings for LSU Football News
Follow us on Twitter, Facebook and Instagram to get the latest updates on LSU Football and Recruiting.

FacebookTwitterInstagram