[sqlite] Soft Heap Limit

2007-11-09 Thread Mark Brown
Hi- I'm currently using SQLite 3.4.1 and have been reading with interest the large database thread. I learned about the soft heap limit feature and was considering using it. While reading about the bug fixes for 3.4.2 and soft heap limit, I came across a sample of setting the soft heap limit to

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > Joe Wilson <[EMAIL PROTECTED]> wrote: > > Is this memory pooling going to be compile-time optional? > > > > I find that library-specific memory pools are awkward because each > > library tends to have its own schemes that don't play well with each > > other. If you

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > Joe Wilson <[EMAIL PROTECTED]> wrote: > > --- [EMAIL PROTECTED] wrote: > > > Mostly I am interested in making sure that malloc(1000) does not > > > fail even though you have 5 bytes free and they just happen > > > to be scattered about as 100 discontinguous

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread Eduardo Morras
At 19:41 30/10/2007, you wrote: Mostly I am interested in making sure that malloc(1000) does not fail even though you have 5 bytes free and they just happen to be scattered about as 100 discontinguous blocks of 500 bytes each. On the embebed device i worked (i made only the micro-os with

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread drh
Joe Wilson <[EMAIL PROTECTED]> wrote: > --- [EMAIL PROTECTED] wrote: > > Mostly I am interested in making sure that malloc(1000) does not > > fail even though you have 5 bytes free and they just happen > > to be scattered about as 100 discontinguous blocks of 500 bytes > > each. > > It's a

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > Mostly I am interested in making sure that malloc(1000) does not > fail even though you have 5 bytes free and they just happen > to be scattered about as 100 discontinguous blocks of 500 bytes > each. It's a good goal. You can reduce the likelihood of failure

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread drh
Joe Wilson <[EMAIL PROTECTED]> wrote: > --- [EMAIL PROTECTED] wrote: > > Joe Wilson <[EMAIL PROTECTED]> wrote: > > > The only real way to prevent allocation fragmentation is to move > > > blocks of memory around - > > > > Not true. You can prevent fragmentation, for example, by > > not

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread John Stanton
[EMAIL PROTECTED] wrote: Joe Wilson <[EMAIL PROTECTED]> wrote: The only real way to prevent allocation fragmentation is to move blocks of memory around - Not true. You can prevent fragmentation, for example, by not allocating objects beside each other that will be destroyed at different

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > Joe Wilson <[EMAIL PROTECTED]> wrote: > > The only real way to prevent allocation fragmentation is to move > > blocks of memory around - > > Not true. You can prevent fragmentation, for example, by > not allocating objects beside each other that will be destroyed >

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread drh
Joe Wilson <[EMAIL PROTECTED]> wrote: > The only real way to prevent allocation fragmentation is to move > blocks of memory around - Not true. You can prevent fragmentation, for example, by not allocating objects beside each other that will be destroyed at different times. Or, you can pick a

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread Joe Wilson
The only real way to prevent allocation fragmentation is to move blocks of memory around - i.e., return and manilpulate handles to pointers instead of the pointers themselves. But this adds a lot of runtime overhead and is not C friendly. Anything else is just a compromise. Predictive and

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread drh
Joe Wilson <[EMAIL PROTECTED]> wrote: > Hi Richard, > > This might be worth a read. This paper discusses limitations of custom > memory allocators: > > Reconsidering Custom Memory Allocation > http://www.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf > Interesting paper. Thanks for the

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread Joe Wilson
Hi Richard, This might be worth a read. This paper discusses limitations of custom memory allocators: Reconsidering Custom Memory Allocation http://www.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf This post by Emery Berger outlines the problems with Apache Portable Runtime (APR) memory

Re: [sqlite] Soft heap limit enforcement performance

2007-10-30 Thread drh
patters <[EMAIL PROTECTED]> wrote: > We rely on the SQLite memory management to enforce the memory usage in our > application (running on Windows CE). This has worked quite well for us, but > have found that when we hit the limit, in some circumstances, performance > drops significantly. > >

[sqlite] Soft heap limit enforcement performance

2007-10-30 Thread patters
We rely on the SQLite memory management to enforce the memory usage in our application (running on Windows CE). This has worked quite well for us, but have found that when we hit the limit, in some circumstances, performance drops significantly. Looking into the internals of SQLite, it seems