On Wed, Jun 24, 2009 at 3:21 PM, Matthew O'Keefe<matthew.oke...@bakbone.com> wrote: > > > We are using SQLite for indexing a huge number (i.e., 100 million to 1 > billion) of key pairs > that are represented by an 88-byte key. We are using a single table with a > very large number of rows (one for each data chunk), and two columns. > > The table has two columns. One is of type ³text² and the other is type > ³integer². >> >> The table is created with: >> >> CREATE TABLE chunks >> ( >> name text primary key, >> pid integer not null > ); > > As expected, as the > table grows, the underlying B-tree implementation for SQLite means that the > number of > disks accesses to (a) find, and (b) add a chunk, grows larger and larger. > We¹ve tested up > to 20 million chunks represented in the table: as expected performance > exponentially > decreases as the number of table entries grows.
Why don't you use, or at least test, with BerkeleyDB? Since you have only one table, you can hardly benefit from the SQLness of an rdb. If nothing, you will have a point of comparison with another technology, and know for sure if SQLite is the appropriate solution for you. You experience would also likely make for a good case study on the SQLite wiki that might help others in the future. > > We wanted to post to the mailing list to see if there are any obvious, > first-order things > we can try to improve performance for such a large table. > > We really appreciate the efforts of the SQLite developer community! > > Matt O¹Keefe > > sqlite-users@sqlite.org > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- Puneet Kishor http://www.punkish.org/ Carbon Model http://carbonmodel.org/ Charter Member, Open Source Geospatial Foundation http://www.osgeo.org/ Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor/ Nelson Institute, UW-Madison http://www.nelson.wisc.edu/ ----------------------------------------------------------------------- collaborate, communicate, compete ======================================================================= _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
