Hi Przemek,
Well, you're right, I've tried this same example
with MingW and it happily goes up to 100000 iterations,
with a top memory consumption of 32MB.
Then I tested with BCC 5.8 and it went mad a bit
later @ 40000, and stayed on 232MB until reaching
100000.
One more reason to avoid BCC 5.5 for final apps.
I think I'll now switch to 5.8 for the base
compiler for both Harbour and on my projects
(until switching to GCC that is).
Brgds,
Viktor
On 2008.08.25., at 17:30, Przemyslaw Czerpak wrote:
On Mon, 25 Aug 2008, Szakáts Viktor wrote:
Hi Viktor,
Classic problem with memory fragmentation. Algorithms
used by your C compiler to allocate memory from OS and
then divide it for application do not work well with such
code which is BTW killer for many memory managers.
We can try to reduce this problem by adding code for
array preallocation anyhow programmers should now about
it and try to write more memory manager friendly code.
The problem with this is that such code (AAdd( a, v ))
is fairly common in regular Clipper code and even in
Harbour core .prg code. In tbrowse.prg for example,
there is no way to know the number of columns in advance.
teditor/dirscan are also affected.
There is no problem with AADD() but with the _very_ large
array and memory manager which is simply fatal. F.e. your
code works perfectly on Linux with GLIBC. I added at the
end:
? memory( HB_MEM_USED )
compiling it with FMSTAT and it reports:
4965585
It allocates less then 5MB for Harbour structures.
It means that it should not need more then 20MB from
OS if memory manager does not have some serious internal
problems (in Linux total application memory is 17MB).
I guess you are using BCC. Please try to compile Harbour
with -DHB_FM_WIN32_ALLOC and check the results.
It will be interesting if also default windows memory
manager is effected. This example is very trivial and
should not be a problem for good memory manager.
The end result is that it's difficult to really exploit
the "unlimited" size of arrays in Harbour since it
chokes much earlier because of this. IOW AADD() can
easily be considered as dangerous.
If memory manager does not have some basic protection against
memory fragmentation then any function which allocates memory
can be dangerous in long working programs. Sooner or later
we will have to implement our own memory manager which will
be tuned for Harbour structures.
Maybe we should try to replicate the xhb preallocation
(and maybe lazy shrinking) methods to avoid this. But
you probably have some other and/or better ideas too.
We will add preallocation but I would like to join this
modification with our own memory manager. Please remember
that some CRTLs have quite nice MM and works very well
without user preallocation, f.e. the one from GLIBC in
Linux.
Adding item preallocation like in xHarbour arrays reduces
only the problem. But if your example exploits it with C
compiler you are using then I can easy create other example
which will also exploit current xHarbour code.
[ I also wonder what other Harbour parts could suffer
from the same effect. ]
Pure AADD() is not dangerous as long as total memory consumption
is less then half of total memory available for process.
To recreate the problem it's necessary to resize many times
very large array allocating new memory blocks between resizing
using some primitive or buggy memory manager.
AFAIR we only have one place which may need very large arrays
resized dynamically. It's HB_DirScan().
I even though about changing this code to make preallocation
but I was too lazy ;-( I can update it quite fast.
best regards,
Przemek
_______________________________________________
Harbour mailing list
Harbour@harbour-project.org
http://lists.harbour-project.org/mailman/listinfo/harbour
_______________________________________________
Harbour mailing list
Harbour@harbour-project.org
http://lists.harbour-project.org/mailman/listinfo/harbour
