Joe Wilson <[EMAIL PROTECTED]> wrote: > --- [EMAIL PROTECTED] wrote: > > Joe Wilson <[EMAIL PROTECTED]> wrote: > > > The only real way to prevent allocation fragmentation is to move > > > blocks of memory around - > > > > Not true. You can prevent fragmentation, for example, by > > not allocating objects beside each other that will be destroyed > > at different times. Or, you can pick a single allocation size > > and only do mallocs of exactly that size. The latter approach > > is what we are moving towards for SQLite. The allocation size > > would be the size of what Emery calls a "reap". If you deal > > with large strings and blobs you might need to allocate a chunk > > of memory larger than this, which destroys your fragmentation > > guarantees. But at least you can write testable requirements > > about when you guarantee that fragmentation will not occur. > > Maybe the nomenclature is confusing me, but you are not *preventing* > memory fragmentation with your proposed scheme, but limiting its > effects.
Mostly I am interested in making sure that malloc(1000) does not fail even though you have 50000 bytes free and they just happen to be scattered about as 100 discontinguous blocks of 500 bytes each. > By rounding up memory allocations you will inevitably > still have some unused dead memory areas. It's not fragmentation, > per se, but wasted space nonetheless. Memory lifetime analysis is > great but you have to be very vigilant in your code. > > If you only use small database fields perhaps you can get some sort > of limited memory fragmentation guarantee, but when you have large > blobs and strings that require contiguous memory, as you point out, > all bets are off. Correct. For really large strings and blobs, you can use the new incremental I/O mechanism, though, and still avoid using large contiguous blocks of memory. That is more work for the program, but if you are writing an application where this kind of thing is important, that is what you have to do. > > Also, I'm not sure how many libc functions sqlite uses at this > point. But some of them could malloc memory that is beyond the reach > of your pools. Then there's the application's mallocs to consider > as well. memcpy, memset, strlen. I think that is about the full set. SQLite does not use libc very much, as that limits its portability to embedded platforms. > > But the ultimate test is comparing the total memory arena size > against how many real bytes are allocated and in use. If your > library can increase the percentage of heap used more than a > generic malloc like Lea's, then it will be useful. There is generally a space v. speed tradeoff. Allocators that make more efficient use of memory are slower than those that waste a lot of memory. It is unclear at this point what the best tradeoff will be for SQLite. Probably it will vary from one application to another, suggesting that it should be tunable. > > Are you planning to keep allocations from different connections > from different databases seperate? It would be nice to have > unrelated databases on different threads not share a common memory > pool which would help multi-threaded concurrency. > Partially. My plan is to have a single global memory space that all threads share. But each thread grabs big hunks of that space for its own use on (relatively) infrequent occasions - or at least with less frequency than mallocs currently occur. So the synchronization overhead, while not zero, is reduced. -- D. Richard Hipp <[EMAIL PROTECTED]> ----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
