Saving/loading a table with floats

Report any Hollywood bugs here
Post Reply
User avatar
mrupp
Posts: 147
Joined: Sun Jan 31, 2021 7:44 pm
Location: Switzerland
Contact:

Saving/loading a table with floats

Post by mrupp »

Hi there

I might have stumbled over some issue concerning floats when used in a table which is saved as file and loaded again. For some values, HW doesn't regognize these values as equal anymore, on some platforms.

Please check out this example which does the following:
  • Creating a table as an array containing the values 1.0, 1.1, 1.2, ..., 2.0
  • Outputting these values to the "Saved values" column
  • Saving this table to a file using the WriteTable() function
  • Loading the table from the file using the ReadTable() function (in a real life scenario, this part would probably be on the next use of the app)
  • Outputting the values from the loaded table to the the "Config file" column
  • Comparing the original values with the loaded values on equality and print "equal" or "not equal" in between the 2 columns
  • Loading the file once more using the FileToString() function
  • Outputting the contents of the written file as additional information next to the columns on the right

Code: Select all

@APPTITLE "Float Test"
@APPAUTHOR "Michael Rupp"

m_configFilename = "Test.config"
@IF #HW_AMIGAOS3
	m_configFilename = "Test_OS3.config"
@ENDIF
@IF #HW_AMIGAOS4
	m_configFilename = "Test_OS4.config"
@ENDIF
@IF #HW_MORPHOS
	m_configFilename = "Test_MOS.config"
@ENDIF
@IF #HW_WINDOWS
	m_configFilename = "Test_WIN64.config"
@ENDIF
@IF #HW_MACOS
	m_configFilename = "Test_MACPPC.config"
@ENDIF
@IF #HW_MACOS And #HW_LITTLE_ENDIAN
	m_configFilename = "Test_MAC64.config"
@ENDIF

SetFontColor(#LIME)
SetFontStyle(#ANTIALIAS)
SetFont(#SANS, 36)

m_marginLeft = 20
m_top = 10
TextOut(m_marginLeft, m_top, "Saved values")
m_top = m_top + 60
SetFont(#SANS, 24)

configToSave = CreateList()
For Local i = 0 To 10
	Local float = 1 + (i / 10)
	InsertItem(configToSave, float)
Next

For Local i = 0 To ListItems(configToSave) - 1
	TextOut(m_marginLeft, m_top, configToSave[i])
	m_top = m_top + 24
Next

file = OpenFile(Nil, m_configFilename, #MODE_WRITE)
WriteTable(file, configToSave, { adapter = "Default" })
CloseFile(file)

file = OpenFile(Nil, m_configFilename)
configLoaded = ReadTable(file, { adapter = "Default" })
CloseFile(file)

m_top = 10
m_marginLeft = 250
SetFont(#SANS, 36)
TextOut(m_marginLeft, m_top, "Config file")
m_top = m_top + 60
SetFont(#SANS, 24)

For Local i = 0 To ListItems(configLoaded) - 1
	TextOut(m_marginLeft, m_top, configLoaded[i])
	TextOut(m_marginLeft - 120, m_top, IIf(configToSave[i] = configLoaded[i], "equal", "not equal"))
	m_top = m_top + 24
Next

configAsString = FileToString(m_configFilename)
configAsString = ReplaceStr(configAsString, "[", "[[")
configAsString = ReplaceStr(configAsString, "]", "]]")

m_top = 45
m_marginLeft = 400
TextOut(m_marginLeft, m_top, configAsString)

Repeat
	WaitEvent
Forever
 
Originally, I stumbled over this because some values where saved to my config file with lots of values after the decimal. This does make sense, though, because some decimal values in general can't be stored properly using floating numbers. So I thought: This is OK as long as HW takes care of this when loading the values from the file. On first sight, this actually seems to be the case as all values where output without the additional decimal places that were stored in the file.
But unfortunately, when checking these values on equality to the original values, the test result is quite different on the various platforms:

Windows x64:
Image

macOS x64:
Image

macOS PPC:
Image

MorphOS:
Image

OS4:
Image

OS3 non-fpu:
Image

OS3 fpu:
Image

Result:
  • OK on Windows, macOS x64, macOS PPC and OS3 (compiled for FPU)
  • NOT OK on MorphOS, OS4 and OS3 without FPU
Sometimes the equality test turns out to be quite surprising, a value 1.9 saved in the file as "1.9" turns out to be equal on OS3 FPU but not on OS4 or MorphOS.

I think this is something that needs to be looked into...

Cheers, Michael
User avatar
airsoftsoftwair
Posts: 5425
Joined: Fri Feb 12, 2010 2:33 pm
Location: Germany
Contact:

Re: Saving/loading a table with floats

Post by airsoftsoftwair »

Welcome to the wonderful world of floating point arithmetics ;)

What you're seeing here is pretty much a limitation/feature of the C runtime library floating point (de)serialization precision with certain compilers. This will get a little better with the next Hollywood update on MorphOS because I was able to convince the MorphOS team to improve the precision in libnix but in general the way to deal with these precision issues after deserializing is to never check floating point values for exact equality but for near equality, by doing something like this:

Code: Select all

If Abs(v - v2) < 1e-14 Then Print("Equal!")
Even on platforms that don't show any issues with your demo script (Linux, macOS, Windows) there is no guarantee that you'll get the exact floating point value back because a conversion is involved. Thus, if you want to be on the safe side, you must check for near equality like shown above, not for total equality.

Or you could serialize the data as binary using Hollywood's legacy serializer but this won't be in a human-readable format then.
User avatar
mrupp
Posts: 147
Joined: Sun Jan 31, 2021 7:44 pm
Location: Switzerland
Contact:

Re: Saving/loading a table with floats

Post by mrupp »

Thanks for clarifying this and for the formula on how to test for near-equality. Maybe you could include this solution as an internal function for the next Hollywood release, maybe call it something like AreFloatsEqual(float1, float2) or similar. For people that are not all that familiar with floats, it's probably not that easy to come up with this kind of formula.

What do you think?
User avatar
airsoftsoftwair
Posts: 5425
Joined: Fri Feb 12, 2010 2:33 pm
Location: Germany
Contact:

Re: Saving/loading a table with floats

Post by airsoftsoftwair »

Sure, that could easily be added.
User avatar
airsoftsoftwair
Posts: 5425
Joined: Fri Feb 12, 2010 2:33 pm
Location: Germany
Contact:

Re: Saving/loading a table with floats

Post by airsoftsoftwair »

Code: Select all

- New: Added NearlyEqual() function for comparing floating point numbers for near equality; comparing
  floating point numbers using the equality operator can lead to problems in case there are very minimal
  differences on the bit level, e.g. caused by (de)serialization; that's why it's recommended to compare
  floating point numbers against near equality instead of absolute equality for more reliability
Post Reply