[fpc-pascal] Question on how to avoid memory trouble

2007-02-03 Thread Jason P Sage
>The error is not from a lack of stack space, either. The error is >EOutOfMemory that indicates a less easily solved problem. >Can anyone recommend a method to search a whole drive, of arbitrary >size, without running out of memory. >Stu Cox This is exactly the kind of thing my string class wo

RE: [fpc-pascal] Question on how to avoid memory trouble

2007-02-05 Thread Cox, Stuart TRAN:EX
-pascal] Question on how to avoid memory trouble >The error is not from a lack of stack space, either. The error is >EOutOfMemory that indicates a less easily solved problem. >Can anyone recommend a method to search a whole drive, of arbitrary >size, without running out of memory

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-05 Thread Wayne Sherman
I think that its great that the hornet's nest that I stirred up might mean that a new unit of container classes gets written... ...My trouble remains... Getting a better containers unit is nice, but the root of the problem seems to be the memory manager. Delphi's old memory manager had fragm

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-06 Thread Micha Nelissen
Wayne Sherman wrote: > Getting a better containers unit is nice, but the root of the problem > seems to be the memory manager. Delphi's old memory manager had No. > fragmentation issues also. That is why NexusMM, BucketMM and FastMM we The memory manager cannot fix fragmentation in TStringList

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-06 Thread Peter Vreman
At 23:39 5-2-2007, you wrote: I think that its great that the hornet's nest that I stirred up might mean that a new unit of container classes gets written... ...My trouble remains... Getting a better containers unit is nice, but the root of the problem seems to be the memory manager. Delphi's

RE: [fpc-pascal] Question on how to avoid memory trouble

2007-02-12 Thread Helmut Hartl
> >FastMM is open source, has anyone ported it to FPC? > > There are users that have got FastMM working with FPC. But > it is not needed. The standard FPC heap manager is for > single-threaded applications as fast as FastMM. A little bit shortsighted (IMHO),as "everyone seriously concern

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-12 Thread Florian Klaempfl
Helmut Hartl schrieb: > > > >FastMM is open source, has anyone ported it to FPC? > > > > There are users that have got FastMM working with FPC. But > > it is not needed. The standard FPC heap manager is for > > single-threaded applications as fast as FastMM. > > A little bit shortsighted

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-12 Thread George Lober
Florian Klaempfl wrote: Helmut Hartl schrieb: > >FastMM is open source, has anyone ported it to FPC? > > There are users that have got FastMM working with FPC. But > it is not needed. The standard FPC heap manager is for > single-threaded applications as fast as FastMM. A little bit

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-12 Thread Jonas Maebe
On 12 Feb 2007, at 20:27, George Lober wrote: One nice side benefit of FastMM is that it automatically lets you know of memory leaks with no fuss and bother on the programmers part. FPC has support for that built in. Just compile your application with -ghl (that also does a lot of other me

RE: [fpc-pascal] Question on how to avoid memory trouble

2007-02-12 Thread Helmut Hartl
>Florian Klaempfl schrieb: > Well, the lockless multithreaded memory managers are a myth. > Atomic exchange operations take a also a lot of time and are > locked by the cpu as well. > The XEN Developers at Cambridge have nice papers to read ... http://www.cl.cam.ac.uk/~kaf24/ (Others are

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-12 Thread Vincent Snijders
Helmut Hartl schreef: http://www.research.ibm.com/people/m/michael/pldi-2004.pdf This one is not lock free, because it uses atomic instructions used by the cpu, which are essentially fine grained locks. Vincent ___ fpc-pascal maillist - fpc-pas

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-12 Thread George Lober
Jonas Maebe wrote: On 12 Feb 2007, at 20:27, George Lober wrote: One nice side benefit of FastMM is that it automatically lets you know of memory leaks with no fuss and bother on the programmers part. FPC has support for that built in. Just compile your application with -ghl (that also does

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-13 Thread Florian Klaempfl
Vincent Snijders schrieb: > Helmut Hartl schreef: > >> http://www.research.ibm.com/people/m/michael/pldi-2004.pdf >> > > This one is not lock free, because it uses atomic instructions used by > the cpu, which are essentially fine grained locks. Exactly, and cmpxchg etc. are really expensive too.

RE: [fpc-pascal] Question on how to avoid memory trouble

2007-02-13 Thread Helmut Hartl
> > > > This one is not lock free, because it uses atomic > instructions used by > > the cpu, which are essentially fine grained locks. > > Exactly, and cmpxchg etc. are really expensive too. Since > the original FPC heap uses already several pools, it should > be easily possible to m

Re: [fpc-pascal] Question on how to avoid memory trouble

2007-02-13 Thread Jonas Maebe
On 13 feb 2007, at 13:12, Helmut Hartl wrote: Exactly, and cmpxchg etc. are really expensive too. Since the original FPC heap uses already several pools, it should be easily possible to make it scale better in multi threaded applications. Fine that sounds really nice - Do you already have a i

[fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-02 Thread Cox, Stuart TRAN:EX
I am trying to EnumerateFiles (create a list of all files that match a given filespec) across and down a whole drive. I've directly used the EnumerateFiles code from TurboPower's SysTools as available on SourceForge. The code works just as TurboPower designed and it and my wrapper code runs find

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-02 Thread Marco van de Voort
> Can anyone recommend a method to search a whole drive, of arbitrary > size, without running out of memory. I don't know seen SysTools, but I worked analysing logfiles for a year. All containertypes (TList TObjectList and TstringList included) that have a single array as internal datastructure be

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Michael Van Canneyt
On Fri, 2 Feb 2007, Marco van de Voort wrote: > > Can anyone recommend a method to search a whole drive, of arbitrary > > size, without running out of memory. > > I don't know seen SysTools, but I worked analysing logfiles for a year. All > containertypes (TList TObjectList and TstringList incl

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Marco van de Voort
> On Fri, 2 Feb 2007, Marco van de Voort wrote: > > > > Can anyone recommend a method to search a whole drive, of arbitrary > > > size, without running out of memory. > > > > I don't know seen SysTools, but I worked analysing logfiles for a year. All > > containertypes (TList TObjectList and Tstr

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Michael Van Canneyt
On Sat, 3 Feb 2007, Marco van de Voort wrote: > > On Fri, 2 Feb 2007, Marco van de Voort wrote: > > > > > > Can anyone recommend a method to search a whole drive, of arbitrary > > > > size, without running out of memory. > > > > > > I don't know seen SysTools, but I worked analysing logfiles f

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Marc Santhoff
Am Freitag, den 02.02.2007, 08:52 -0800 schrieb Cox, Stuart TRAN:EX: > I am trying to EnumerateFiles (create a list of all files that match a > given filespec) across and down a whole drive. I've directly used the > EnumerateFiles code from TurboPower's SysTools as available on > SourceForge. > >

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Marco van de Voort
> Yes it will, because the reallocations don't happen as often. > The sorting introduces an overhead anyway, whether you set capacity or not. Yes, but I was talking about slowness in general, not just from the heap. And TStringList with those huge internal list has to move on avg half of the arr

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Marc Santhoff
Am Samstag, den 03.02.2007, 16:36 +0100 schrieb Marc Santhoff: > Am Freitag, den 02.02.2007, 08:52 -0800 schrieb Cox, Stuart TRAN:EX: ... > > Can anyone recommend a method to search a whole drive, of arbitrary > > size, without running out of memory. > > >From reading this thread I think you must

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Michael Van Canneyt
On Sat, 3 Feb 2007, Marco van de Voort wrote: > > Yes it will, because the reallocations don't happen as often. > > The sorting introduces an overhead anyway, whether you set capacity or not. > > Yes, but I was talking about slowness in general, not just from the heap. > > And TStringList wit

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Marco van de Voort
> On Sat, 3 Feb 2007, Marco van de Voort wrote: > > > > Yes it will, because the reallocations don't happen as often. > > > The sorting introduces an overhead anyway, whether you set capacity or > > > not. > > > > Yes, but I was talking about slowness in general, not just from the heap. > > >

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Luiz Americo Pereira Camara
Michael Van Canneyt wrote: All this doesn't exclude that a specialized class may be more suitable for the job. To be honest, the only good point of TStringList seems to be that it is default available on all Object Pascal. The specialised stuff (splitting strings) is also plagued with odd

Re: [fpc-pascal] Question on how to avoid memory trouble using FindFirst(), FindNext() and FindClose()

2007-02-03 Thread Michael Van Canneyt
On Sat, 3 Feb 2007, Marco van de Voort wrote: > > On Sat, 3 Feb 2007, Marco van de Voort wrote: > > > > > > Yes it will, because the reallocations don't happen as often. > > > > The sorting introduces an overhead anyway, whether you set capacity or > > > > not. > > > > > > Yes, but I was tal