[Mono-dev] Weird performance problems possibly due to floats and precision issues?
Hi,
So I've come across this very odd situation, which I believe I may know the
cause, but don't understand the result. I'm hoping someone can provide some
insight on the matter.
It would appear when float values get very small in a simple single line
arithmetic equation that performance suddenly drops of a cliff. Now I know
how that sounds, but i'm struggling to find any other cause and so wonder if
there is a logical explanation for it?
I'd converted Jos Stam C demo for 'Real-Time Fluid Dynamics for Games' into
C# (actually Mono 2.6 in Unity), but noticed when using certain values for
viscosity one function would suddenly take an order of magnitude many times
longer to complete than normal!
Profiling pinpointed the problem to a specific function, the kicker being
that there is no reason I can see why this function shouldn't always
complete in constant time (excluding differences due to system interruptions
etc). In fact as long as the grid size remains constant (its a fluid solver
based on a 2D grid) I don't think there is any function in the code that
shouldn't complete in constant time between frames.
Essentially the function has three nested loops, all of which iterate the
same number of times every call. The inner loop is a single line of simple
arithmetic instructions akin to x[i] = y[i] + (a * ( y[s] + y[t] + y[u] +
y[v]) )/c. It normally completes in a few milliseconds, but when passing in
the troublesome viscosity value it slows down quickly to taking half a
second!
Interesting the profiler showed that the execution time followed a bell
curve when the slow down occurred, suggesting that perhaps an error in the
inner loop was propagating out (through the grid) and then dissipating away,
returning to normal. However no exceptions or errors are raised on the code,
I even tried wrapping the 'checked' directive around the inner loop to no
affect.
My gut-feeling then based on observation (debugging is possible but awkward
due to the sheer number of operations happening per frame), is that it comes
down to some type of floating point precision or perhaps underflow issue (
most grid cell values have become similar to 2.802597E-44 or 1.401298E-45
during the slow down).
This is somewhat backed up by the fact that when switching to doubles the
issue goes away for my test value, but eventually returns when I raise it
higher. However since no exceptions are raised and I don't notice any
difference in the results I can't fathom why it slow downs. It feels like
its something deep within within Mono or perhaps the hardware itself which
is trying to be helpful and not produce an error but results in unacceptable
slowdown?
So I'd welcome any input into why a single line of code, that just adds and
multiplies a few floats can suddenly take 100's of times longer to execute
than normal?
The function is
N is the dimension of the grid/array (i.e. N*N = total cells in grid)
b is a flag for use in the set_bnd() call
x is a an array of the current float values
x0 is an array of the previous float values
a = dt*viscosity *N*N (where dt is a constant 0.15)
c = 1+4*a
int IX(int i, int j)
{
return ( (i)+(N+2)*(j) );
}
void lin_solve ( int N, int b, ref float[] x, ref float[] x0, float a, float
c )
{
int i, j, k;
for ( k=0 ; k20 ; k++ )
{
for ( i=1 ; i=N ; i++ )
{
for ( j=1 ; j=N ; j++ )
{
x[IX(i,j)] = (x0[IX(i,j)] + a*(x[IX(i-1,j)] +
x[IX(i+1,j)] + x[IX(i,j-1)] + x[IX(i,j+1)]))/c;
}
}
set_bnd ( N, b, ref x );
}
}
Some additional notes.
Specific values i'm testing with
N = 128
viscosity = 0.09
dt = 0.15
In profiling whilst set_Bnd() did take slightly longer than normal the vast
amount of time was in lin_solve().
Calling IX() for converting the grid(i,j) into a single index is obviously
non-optimal and i've optimised that in my actual code. However the original
code shown here still exhibits the slowdown issue and is easier to read.
--
View this message in context:
http://mono.1490590.n4.nabble.com/Weird-performance-problems-possibly-due-to-floats-and-precision-issues-tp3829087p3829087.html
Sent from the Mono - Dev mailing list archive at Nabble.com.
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] question: huge amounts of RAM
Hi, as some might already know we at sones are working on a C# based graph database. And since we are about to test our in-memory version of our database with huge amounts of memory the question came up if we will be able to address those huge amounts of memory. When speaking of huge I mean 1 to 2 terabytes of RAM. Since we do not own machines with such extreme amounts of memory we have been a limited time-frame assigned on such a machine to test our software on. So we want to make sure upfront that we actually will be able to use that amount of RAM . any ideas, problems and suggestions will be appreciated. Best regards, Daniel Kirstenpfad Founder / CTO sones GmbH Schillerstraße 5 D-04109 Leipzig Mail: daniel.kirstenp...@sones.com twitter: @bietiekay Tel.: +49 341 392968-0 Fax: +49 361 2445 008 Web: www.sones.com ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] question: huge amounts of RAM
On Wed, Sep 21, 2011 at 11:42 AM, Daniel Kirstenpfad dan...@sones.de wrote: When speaking of huge I mean 1 to 2 terabytes of RAM. Since we do not own machines with such extreme amounts of memory we have been a limited time-frame assigned on such a machine to test our software on. So we want to make sure upfront that we actually will be able to use that amount of RAM…. Do you care about the length of GC pause times? I mean the absolute length, not the relative amount of time the GC consumes. (If your programs do anything but batch processing then you probably do). Mark -- Mark Probst http://www.complang.tuwien.ac.at/schani/ http://www.flickr.com/photos/schani/ http://schani.wordpress.com/ ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Weird performance problems possibly due to floats and precision issues?
On Wed, Sep 21, 2011 at 7:04 AM, Ivo Smits i...@ufo-net.nl wrote: Op 21-9-2011 8:31, noisecrime schreef: ... Essentially the function has three nested loops, all of which iterate the same number of times every call. The inner loop is a single line of simple arithmetic instructions akin to x[i] = y[i] + (a * ( y[s] + y[t] + y[u] + y[v]) )/c. It normally completes in a few milliseconds, but when passing in the troublesome viscosity value it slows down quickly to taking half a second! I have seen similar behaviour on MS .Net in an audio processing application. The slowdown there was caused by arithmetic operations on a non-value (NaN/Infinity/negative infinity). Such an operation will again result in a non-value, which might explain why the slowdown propagates. You could easily check for these values in your code. I don't know exactly why these things cause such a slowdown. Maybe there is some (hardware) exception processing getting in the way, but that's just guessing. In my case it helped a lot to check the result for these 3 non-values, and set it to some valid value instead. It's not a proper solution, but in my case helped to keep the application useable even for incorrect input values. NaN and denormals are many times slower than regular numbers. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] question: huge amounts of RAM
Well actually we're about to introduce some changes into our import functionality which will be as cautious as possible to not create intermediate objects but only objects that stay. So mainly it will be a growing operation regarding the number of object instances. We already looked into the data structures we use and that those don't impose a limit of Int32 to us when growing. Which of the two garbage collectors would be best in your opinion? Would someone of the Mono-dev team want to come and test with us? -Original Message- From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of Mark Probst Sent: Mittwoch, 21. September 2011 11:52 To: Daniel Kirstenpfad Cc: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] question: huge amounts of RAM On Wed, Sep 21, 2011 at 11:42 AM, Daniel Kirstenpfad dan...@sones.de wrote: When speaking of huge I mean 1 to 2 terabytes of RAM. Since we do not own machines with such extreme amounts of memory we have been a limited time-frame assigned on such a machine to test our software on. So we want to make sure upfront that we actually will be able to use that amount of RAM . Do you care about the length of GC pause times? I mean the absolute length, not the relative amount of time the GC consumes. (If your programs do anything but batch processing then you probably do). Mark -- Mark Probst http://www.complang.tuwien.ac.at/schani/ http://www.flickr.com/photos/schani/ http://schani.wordpress.com/ ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Weird performance problems possibly due to floats and precision issues?
Thx for the replies. @ Ivo Smits Initially I did think of Nan or Infinity, but then wouldn't these give a runtime exception? I couldn't find any trace when stepping through a debugger, though I didn't check all 16000 entries in the array. I've since tried testing directly for Nan (float.IsNaN) but nothing was found. @ Rodrigo Kumpera Denormals, that sounds much more likely, especially as the values I got from the debugger were towards 1e-44. That would explain the performance drop. Trouble is I've since tried checking for these and simply zero-ing them (e.g. if(Math.Abs( x[i] ) 1e-38 ) x[i] = 0;), but that appeared to have no effect? Indeed I found it was zero-ing values when the performance was running fine, so clearly the fluid sim is generating some very small values normally. However its the best explanation i've heard, it makes a lot of sense in the context of the code. -- View this message in context: http://mono.1490590.n4.nabble.com/Weird-performance-problems-possibly-due-to-floats-and-precision-issues-tp3829087p3829700.html Sent from the Mono - Dev mailing list archive at Nabble.com. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] question: huge amounts of RAM
On Wed, Sep 21, 2011 at 1:20 PM, Daniel Kirstenpfad dan...@sones.de wrote: Well actually we're about to introduce some changes into our import functionality which will be as cautious as possible to not create intermediate objects but only objects that stay. So mainly it will be a growing operation regarding the number of object instances. We already looked into the data structures we use and that those don't impose a limit of Int32 to us when growing. Which of the two garbage collectors would be best in your opinion? In that case a generational garbage collector is actually of little use - the working premise of a generational collector is that most objects die young. I suggest giving Boehm a try. Would someone of the Mono-dev team want to come and test with us? Sure! Send me an email when you're ready to test. Mark ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Weird performance problems possibly due to floats and precision issues?
Good news, looks like I may have found a solution to the problem. Though I probably need to do some further testing, it does indeed appear to be denormals causing the issue. After reading up on it online its been a very common problem for DSP/audio developers and thankfully there appears to be a simple solution. This page describes the problem and various solutions http://phonophunk.com/articles/pentium4-denormalization.php?pg=3 I went with the simplest “On every location in the code where denormalization might occur, just add 1.0e-24 to the float value, then subtract it again, that should fix it.” Doing that in just the lin_solve function such as x[i+N2J] = denormal + (( x0[i+N2J] + a * (x[i-1+N2J] + x[i+1+N2J] + x[i+N2JN] + x[i+N2JP]) ) * oneOverC); x[i+N2J] -= denormal; gives constant performance, though I suspect i'll need to add it to a few other places in the code as well. -- View this message in context: http://mono.1490590.n4.nabble.com/Weird-performance-problems-possibly-due-to-floats-and-precision-issues-tp3829087p3829895.html Sent from the Mono - Dev mailing list archive at Nabble.com. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Mono-devel-list Digest, Vol 77, Issue 6
Thanks Robert, I expected something like that. Is there an easy way to determine if the runtime has already been initted? I would like to use Mono for practically all logic for my unmanaged SO needs, yet allow for the possibility that the application is managed. I would like to avoid requiring the app to pass some Mono handle into the SO to do all its work, though if that's the way it is, then that's the way it is. Then comes the issue of versioning the runtimes, 2 corlibs being loaded, etc. Should I just stay away from this whole thing? Are there any complications that I should be aware of when the SO+managed code is ready to be unload from the process? Thanks, Joe Date: Tue, 13 Sep 2011 11:53:07 +0200 From: Robert Jordan robe...@gmx.net Subject: Re: [Mono-dev] Multiple Monos in a single process To: mono-devel-list@lists.ximian.com Message-ID: j4n96i$6is$1...@dough.gmane.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 12.09.2011 22:23, Joe Dluzen wrote: Hi all, let's say I have a C# app A which PInvokes to a native SO/DLL B, which embeds Mono for basically 100% of its logic. Do bad things happen on Linux vs Windows? Yes. Under Linux, libmono is statically linked by default. If you p/invoke a library that it turn is linked against libmono, you'll end up with 2 runtime instances. Even if you link libmono dynamically (there is a `configure' switch for this), you must take care to initialize the runtime only once. This means that you can't call mono_jit_init/cleanup from the SO. Robert ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Weird performance problems possibly due to floats and precision issues?
Hi, Are you on 32 or 64 bits? Our FP code is quite different on those two. Can you have a run on .NET without the added denormal hack? We strike to be as compatible as possible, specially when it comes to numerical stability and output. On Wed, Sep 21, 2011 at 10:00 AM, noisecrime no...@noisecrime.com wrote: Good news, looks like I may have found a solution to the problem. Though I probably need to do some further testing, it does indeed appear to be denormals causing the issue. After reading up on it online its been a very common problem for DSP/audio developers and thankfully there appears to be a simple solution. This page describes the problem and various solutions http://phonophunk.com/articles/pentium4-denormalization.php?pg=3 I went with the simplest “On every location in the code where denormalization might occur, just add 1.0e-24 to the float value, then subtract it again, that should fix it.” Doing that in just the lin_solve function such as x[i+N2J] = denormal + (( x0[i+N2J] + a * (x[i-1+N2J] + x[i+1+N2J] + x[i+N2JN] + x[i+N2JP]) ) * oneOverC); x[i+N2J] -= denormal; gives constant performance, though I suspect i'll need to add it to a few other places in the code as well. -- View this message in context: http://mono.1490590.n4.nabble.com/Weird-performance-problems-possibly-due-to-floats-and-precision-issues-tp3829087p3829895.html Sent from the Mono - Dev mailing list archive at Nabble.com. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Multiple Monos in a single process
Whoops, forgot to edit subject. This one should make the above email more apparent. On Wed, Sep 21, 2011 at 2:13 PM, Joe Dluzen jdlu...@gmail.com wrote: Thanks Robert, I expected something like that. Is there an easy way to determine if the runtime has already been initted? I would like to use Mono for practically all logic for my unmanaged SO needs, yet allow for the possibility that the application is managed. I would like to avoid requiring the app to pass some Mono handle into the SO to do all its work, though if that's the way it is, then that's the way it is. Then comes the issue of versioning the runtimes, 2 corlibs being loaded, etc. Should I just stay away from this whole thing? Are there any complications that I should be aware of when the SO+managed code is ready to be unload from the process? Thanks, Joe Date: Tue, 13 Sep 2011 11:53:07 +0200 From: Robert Jordan robe...@gmx.net Subject: Re: [Mono-dev] Multiple Monos in a single process To: mono-devel-list@lists.ximian.com Message-ID: j4n96i$6is$1...@dough.gmane.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 12.09.2011 22:23, Joe Dluzen wrote: Hi all, let's say I have a C# app A which PInvokes to a native SO/DLL B, which embeds Mono for basically 100% of its logic. Do bad things happen on Linux vs Windows? Yes. Under Linux, libmono is statically linked by default. If you p/invoke a library that it turn is linked against libmono, you'll end up with 2 runtime instances. Even if you link libmono dynamically (there is a `configure' switch for this), you must take care to initialize the runtime only once. This means that you can't call mono_jit_init/cleanup from the SO. Robert ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Multiple Monos in a single process
On 21.09.2011 20:30, Joe Dluzen wrote: Whoops, forgot to edit subject. This one should make the above email more apparent. On Wed, Sep 21, 2011 at 2:13 PM, Joe Dluzenjdlu...@gmail.com wrote: Thanks Robert, I expected something like that. Is there an easy way to determine if the runtime has already been initted? I would like to use Mono for practically all logic for my AFIK, there is no such function, but you could use this test: if (mono_get_root_domain ()) // runtime is already initialized unmanaged SO needs, yet allow for the possibility that the application is managed. I would like to avoid requiring the app to pass some Mono handle into the SO to do all its work, though if that's the way it is, then that's the way it is. Then comes the issue of versioning the runtimes, 2 corlibs being loaded, etc. Should I just stay away from this whole thing? You should avoid this if you don't control Mono on the target machine, because it won't work with a statically linked Mono. If you still believe you need this, then provide a stub for the managed application: a simple native app that embeds Mono and launches the managed application. This way you can be sure that a dynamically linked libmono will be used. Are there any complications that I should be aware of when the SO+managed code is ready to be unload from the process? Don't try to unload the runtime. While API-wise supported, there are still issues with the unloading. Robert Thanks, Joe Date: Tue, 13 Sep 2011 11:53:07 +0200 From: Robert Jordanrobe...@gmx.net Subject: Re: [Mono-dev] Multiple Monos in a single process To: mono-devel-list@lists.ximian.com Message-ID:j4n96i$6is$1...@dough.gmane.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 12.09.2011 22:23, Joe Dluzen wrote: Hi all, let's say I have a C# app A which PInvokes to a native SO/DLL B, which embeds Mono for basically 100% of its logic. Do bad things happen on Linux vs Windows? Yes. Under Linux, libmono is statically linked by default. If you p/invoke a library that it turn is linked against libmono, you'll end up with 2 runtime instances. Even if you link libmono dynamically (there is a `configure' switch for this), you must take care to initialize the runtime only once. This means that you can't call mono_jit_init/cleanup from the SO. Robert ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
