On Sun, Dec 5, 2010 at 12:01 AM, John Nagle <na...@animats.com> wrote: > On 12/4/2010 8:44 PM, Monte Milanuk wrote: >> >> On 12/4/10 3:43 PM, Jorge Biquez wrote: >> >>> I do not see a good reason for not using Sqlite3 BUT if for some reason >>> would not be an option.... what plain schema of files would you use? >> >> Would shelve work? > > There are some systems for storing key-value pairs in files. > > Underneath "shelve" is some primitive database, dbm, gdbm or bsddb. > "bsddb" is deprecated and was removed from Python 3.x. "dbm" has > some classic problems. "gdbm" is an improved version of "dbm". > None of these handle access from multiple processes, or crash > recovery. We're looking at 1979 technology here. > > SQLite works right when accessed from multiple processes. SQLite > is the entry-level database technology for Python today. It handles > the hard cases, like undoing transactions after a crash and > locking against multiple accesses. Lookup performance is good; > simultaneous update by multiple processes, though, is not so > good. When you have a web site that has many processes hitting > the same database, it's time to move up to MySQL or Postgres. > > There's a lot of interest in "non-SQL" databases for very > large distributed systems. You worry about this if you're Facebook > or Google, or are running a big game server farm.
SQLite isn't exactly no SQL. I've used the bsddb and gdbm modules quite a bit. I've found that bsddb tables tend to get corrupted (whether used from CPython or C), EG when a filesystem fills up. I quite like the gdbm module though, and have been using it in my current project. If you find that converting your database keys and values to/from strings is expensive, you could check out http://stromberg.dnsalias.org/~dstromberg/cachedb.html which is a caching wrapper around gdbm and other single-table database interfaces supporting the same API. As far as multiple processes, IINM, gdbm supports a single writer and multiple readers. -- http://mail.python.org/mailman/listinfo/python-list
