Personally, I do not see this as a problem for a float doing this calculation. Since it is transcendental function, this will happen. Plus, float is not 64 bits of precision, it is 52 bits. I suggest use INT or LONG This type of round-off error goes back to the beginning of time -Fernando
---------- Original Message ----------- From: gam...@googlecode.com To: gambas-user@lists.sourceforge.net Sent: Wed, 21 Dec 2011 19:12:21 +0000 Subject: [Gambas-user] Issue 165 in gambas: Arithmetic problem with float > Status: New > Owner: ---- > Labels: Version Type-Bug Priority-Medium OpSys-Any Dist-Any Arch-Any > Desktop-Any GUI-Any > > New issue 165 by jussi.la...@gmail.com: Arithmetic problem with float > http://code.google.com/p/gambas/issues/detail?id=165 > > 1) Describe the problem. > > Dim ff As Float = 2 ^ 23 > > Print ff > > Gambas gives result: 8388607.99999999 while expected is 8388608. > Floating point (64bit) precision should be enough for accurate result. > > 2) GIVE THE FOLLOWING INFORMATIONS (if they are appropriate): > > Version: TRUNK > Revision: r4297 > Operating system: Linux > Distribution: Xubuntu > Architecture: x86_64 > GUI component: QT4 and GTK+ > Desktop used: XFCE > > 3) Provide a little project that reproduces the bug or the crash. > > Attached. > > Attachments: > ArithmeticError-0.0.1.tar.gz 4.6 KB > > ------------------------------------------------------------------------------ > Write once. Port to many. > Get the SDK and tools to simplify cross-platform app development. Create > new or port existing apps to sell to consumers worldwide. Explore the > Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join > http://p.sf.net/sfu/intel-appdev > _______________________________________________ > Gambas-user mailing list > Gambas-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/gambas-user ------- End of Original Message ------- ------------------------------------------------------------------------------ Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev _______________________________________________ Gambas-user mailing list Gambas-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/gambas-user
