- My Forums
- Tiger Rant
- LSU Recruiting
- SEC Rant
- Saints Talk
- Pelicans Talk
- More Sports Board
- Fantasy Sports
- Golf Board
- Soccer Board
- O-T Lounge
- Tech Board
- Home/Garden Board
- Outdoor Board
- Health/Fitness Board
- Movie/TV Board
- Book Board
- Music Board
- Political Talk
- Money Talk
- Fark Board
- Gaming Board
- Travel Board
- Food/Drink Board
- Ticket Exchange
- TD Help Board
Customize My Forums- View All Forums
- Show Left Links
- Topic Sort Options
- Trending Topics
- Recent Topics
- Active Topics
Started By
Message
re: True/False Excel Statement Issue
Posted on 11/11/16 at 9:14 pm to mdomingue
Posted on 11/11/16 at 9:14 pm to mdomingue
quote:
That explanation is inaccurate. Machine and round off error can and do occur but it will not be at the 100ths level.
I believe my explanation was correct WRT to the message I replied to.
However, it's not really correct to talk about "the 100ths level". As I said, Excel's FP numbers have 15 decimal digits of precision, and the errors I was talking about are limited typically to the least significant digit or two. It might be hundredths, millionths, or whatever, depending on where the decimal point is in a given number. These small errors can and do cause comparisons to fail for numbers computed in different ways.
quote:quote:
This makes total sense now that I think about it. When my budgets are done for a project, I try to get every subtotal of Divisions 1-16 to be a whole dollar, but even when all of the subtotals read "$X.00", the bottom grand total will read $1,500,000.01. It annoys the shite out of me
The things you're seeing at x.00 and getting a x.01 in the sum is a result of showing 2 decimal places in the individual cells but you're formula is giving you numbers in the 3rd decimal places that add up to give you the .01.
That seems to be a different issue than I was responding to, and I agree that cumulative small errors aren't likely to make 1,500,000 equal to 1,500,000.01. That's due to errors in the 10th significant digit, not the 15th or so.
Posted on 11/12/16 at 8:50 pm to Spock's Eyebrow
quote:
I believe my explanation was correct WRT to the message I replied to.
However, it's not really correct to talk about "the 100ths level". As I said, Excel's FP numbers have 15 decimal digits of precision, and the errors I was talking about are limited typically to the least significant digit or two. It might be hundredths, millionths, or whatever, depending on where the decimal point is in a given number. These small errors can and do cause comparisons to fail for numbers computed in different ways.
You are indeed correct, I misread something not to mention I am getting old and senile.
Using the round function should correct this issue, I think.
Popular
Back to top
Follow TigerDroppings for LSU Football News