Did some hashing code which used pooling and preallocate of block and it made the hash way (technical term) faster in insert and delete operations. As well as much less overhead for small hash object. Basically kept it's own pool of hash items. For SQLite might be harder to implement since things are not as fixed by nature (columns/datatypes), but would get rid of lots of malloc() which are slow as well as costly for internal overhead.
Sandy -----Original Message----- From: Andrew Piskorski [mailto:[EMAIL PROTECTED] Sent: Monday, January 10, 2005 10:28 AM To: sqlite-users@sqlite.org Subject: Re: [sqlite] excessive malloc() calls On Mon, Jan 10, 2005 at 09:51:50AM -0800, [EMAIL PROTECTED] wrote: > sqlite3_step() is the function that is making all the excessive calls, one > call per row fetched. > about 3000 of these tables. One complete scan of all 3000 companies will > make roughly 5.5 million malloc() calls! That seems highly non-optimal. If there is no other way around that already present in SQLite, and you don't want to hack one in, then the obvious step might be to link in one of the several high-performance malloc replacement libraries out there, e.g. Hoard: http://www.cs.umass.edu/~emery/hoard/ http://developers.sun.com/solaris/articles/multiproc/multiproc.html Tcl 8.4.x includes a threaded allocator (which is used by AOLserver), which maintains its own thread-local pools of memory and calls the system malloc() underneath only infrequently. It is not intended as a drop in replacement for malloc(), but I THINK the various malloc replacement libraries operate in similar (though more complex) fashions. I'd expect (but have not tested) that these memory pooling techniques would offer you a speed benefit even when you're using only 1 thread, as malloc'ing millions of times is usually not a good idea. -- Andrew Piskorski <[EMAIL PROTECTED]> http://www.piskorski.com/
