- My Forums
- Tiger Rant
- LSU Recruiting
- SEC Rant
- Saints Talk
- Pelicans Talk
- More Sports Board
- Fantasy Sports
- Golf Board
- Soccer Board
- O-T Lounge
- Tech Board
- Home/Garden Board
- Outdoor Board
- Health/Fitness Board
- Movie/TV Board
- Book Board
- Music Board
- Political Talk
- Money Talk
- Fark Board
- Gaming Board
- Travel Board
- Food/Drink Board
- Ticket Exchange
- TD Help Board
Customize My Forums- View All Forums
- Show Left Links
- Topic Sort Options
- Trending Topics
- Recent Topics
- Active Topics
Started By
Message
re: True/False Excel Statement Issue
Posted on 11/11/16 at 8:41 am to TexasTiger90
Posted on 11/11/16 at 8:41 am to TexasTiger90
Excel uses floating point which is inexact. For example, 0.1 has no exact representation in binary notation, and it introduces errors into calculations. Another example would be calculations that should have the same result that were performed in different ways. I believe Excel tries to deal with this in comparisons by ignoring the least signficant bit or two, but at some point, it doesn't work.
That's what I'm talking about. The workaround is to use a tolerance value in your comparisons, but since you're talking about budgets and hence currency, you may be able to use the ROUND function. More generally, you'd subtract the two numbers and compare the absolute value of the difference to your tolerance factor. (From memory, Excel's binary notation is IEEE 754 double precision, a 64 bit number with range 1E+/-308 with 15 decimal digits of precision.)
quote:
UNLESS I manually input the value rather than let the formula do it
That's what I'm talking about. The workaround is to use a tolerance value in your comparisons, but since you're talking about budgets and hence currency, you may be able to use the ROUND function. More generally, you'd subtract the two numbers and compare the absolute value of the difference to your tolerance factor. (From memory, Excel's binary notation is IEEE 754 double precision, a 64 bit number with range 1E+/-308 with 15 decimal digits of precision.)
This post was edited on 11/11/16 at 8:50 am
1
Posted on 11/11/16 at 10:06 am to Spock's Eyebrow
quote:This makes total sense now that I think about it. When my budgets are done for a project, I try to get every subtotal of Divisions 1-16 to be a whole dollar, but even when all of the subtotals read "$X.00", the bottom grand total will read $1,500,000.01. It annoys the shite out of me
Excel uses floating point which is inexact. For example, 0.1 has no exact representation in binary notation, and it introduces errors into calculations.
Page 1 of 1
Popular
Back to top
Follow TigerDroppings for LSU Football News