Re: [sqlite] sqlite3AtoF giving incorrect results

Basically a variant of the final idea you had is what we went with. Thanks for the suggestions! Jim Borden Privacy Policy Marketing Preferences ___

Re: [sqlite] sqlite3AtoF giving incorrect results

On 2/27/19, Jim Borden wrote: > For reference here > was the query that I was working with: > > SELECT fl_result(prediction('EchoModel', dict_of('max_double', > 1.7976931348623153e+308))) FROM kv_default AS _doc WHERE (_doc.flags & 1 = > 0); > As a work-around, you could code up a UDF max_real()

Re: [sqlite] sqlite3AtoF giving incorrect results

The platforms exhibiting the behavior have so far been MSVC Windows 10 64-bit (both 32 and 64 bit processes), and Android x86. iOS 64-bit was not affected and iOS 32-bit was not testable due to outside factors but my guess is that it is not affected. macOS 64-bit was not affected. I tried to

Re: [sqlite] sqlite3AtoF giving incorrect results

On 2/27/19, Jim Borden wrote: > > My theory as to why this varies by platform is because of the use of long > double during this calculation, which can vary in size. On the failing > platforms they are the same size as regular double. This results in a > rounding error of the "scale" portion of

Re: [sqlite] sqlite3AtoF giving incorrect results

On 2/27/19, Jim Borden wrote: > Hello, > > I’ve been tracking down an odd problem where some of our results from an > application defined SQLite function are coming back as infinity on some > platforms (notably 32-bit Android, and all Windows). I’ve narrowed it down > to the following: > > Given

[sqlite] sqlite3AtoF giving incorrect results

Hello, I’ve been tracking down an odd problem where some of our results from an application defined SQLite function are coming back as infinity on some platforms (notably 32-bit Android, and all Windows). I’ve narrowed it down to the following: Given the example input "1.7976931348623153e+308