I guess I could actually dump the hash table into a blob. I'm also doing something like a bloom filter, and I guess that can just as well go into a blob too.. Basically the system is a big cache, and it must quickly answer the question "Do you have this item in your cache?". The cache is going to receive a lot of queries for which the answer is "NO", and I need the determination of that answer to be fast. That's why I've got a bloom filter-ish thing going which performs that task. Even for a large cache (500k entries), one can keep the entire filter in about 2MB of memory and have less than 1% false positive rate. Unfortunately I can't think of an equivalent data structure that would use B+Tree linear indices to achieve that.
I've done a naive implementation of what I want using straight sqlite, but the performance is not really adequate. It's hard to quantify exactly where the bottlenecks lie though, which is what motivates me to try something else and see what kind of performance delta I get. Thanks. -----Original Message----- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin Sent: 03 November 2010 11:17 AM To: General Discussion of SQLite Database Subject: Re: [sqlite] Using sqlite's WAL with a hash table store On 3 Nov 2010, at 8:30am, Ben Harper wrote: > I know the answer to this question is really "Just try it and see", but I > want to gauge whether the idea is sane or not before I spend/waste time on > the effort: > > I want to build a custom hash table DB, and to solve the > concurrency+durability I need something akin to a WAL, and SQLite's WAL seems > like a perfect fit. I've looked into the wal.c/wal.h a bit and from my brief > perusal it looks like I could quite easily strap the SQLite WAL onto my > custom hash table DB. Modifying SQL, and taking SQL source code and putting it into your own project, are difficult and time-consuming. As a prototype why not /use/ SQL, storing your hash codes in a column ? Use that as a prototype and see if it's fast enough. If it is, stop there. If you find calculating your hashes externally proves too clunky, you could write a custom function to calculate your hash codes http://www.sqlite.org/c3ref/create_function.html , or you could remove the extra column but implement your hash codes as a collating sequence: http://www.sqlite.org/c3ref/create_collation.html Any of the three above ways to do it gets you all the advantages of the WAL code /and/ a SQL engine. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
