Incorrect rounding of number before math calculation

Started by hamishr, November 08, 2022, 10:44:04 PM

Previous topic - Next topic

hamishr

I am new to using the Metadata Mechanic and variables. I have decrees and minutes as two separate attribute values and want to turn them into a single decimal degree value to be stored in the Latitude field. The variable specification I came up with is:

{File.AT.Image.longdeg|cast:real;math:add,{File.AT.Image.longmin|cast:real;math:div,60,6},6}

The values of -16 degrees and -52.988 minutes return the decimal degree value of -16.883167, whereas it should be -16.883133, which is because it is rounding -52.988 to -52.99 before performing the calculation. I thought cast:real and 6 decimal places would have avoiding rounding problems but clearly I am doing something wrong - what is it?

sinus

As far as the variable is concerned, I don't know.
I can only say that I once had a problem with rounding money.

And the solution was in the local settings of Windows, where you also had to enter the roundings and point or comma.
Maybe this has nothing to do with that, I don't know.
Best wishes from Switzerland! :-)
Markus

Mario

Show us what the individual variables return in the VarToy.
Show us what you have input into your Attributes. What the variables return without casing and with casting.
Trying to guess things would just waste time.

You are using data you have filled into Attributes.
There will be rounding and number formatting based on your Windows numeric settings as sinus noted.
Attributes are formatted using the user-specific number settings, so if you set the Windows number settings to two digits, real Attributes will be returned as nn.dd or nn,dd depending on your local numeric formatting rules.
I also think I recall that Attributes of type Real have four digits precision (.nnnn).
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

hamishr

Windows was indeed the culprit - one has to dig deep to find the right place to make the change to the number of decimal places:
1. In the Windows settings menu, click on Time & Language
2. Click on Date, Time and Regional Settings (top right)
3. Click on Additional Date, Time and Regional Settings (top right)
4. Click on Region - date, time and number formats
5. Click on Additional Settings (bottom right)
6. You have reached your destination.

Thanks sinus and Mario for your help!