On Tue, Jan 18, 2005 at 11:26:46AM -0500, Eric S. Johansson wrote: > I have an application where I need a very simple database, effectively a > very large dictionary. The very large dictionary must be accessed from > multiple processes simultaneously. I need to be able to lock records > within the very large dictionary when records are written to. Estimated > number of records will be in the ballpark of 50,000 to 100,000 in his > early phase and 10 times that in the future. Each record will run about > 100 to 150 bytes. > > speed is not a huge concern although I must complete processing in less > than 90 seconds. The longer the delay however the greater number of > processes must be running parallel in order to keep the throughput up. > It's the usual trade-off we have all come to know and love. > > it is not necessary for the dictionary to persist beyond the life of the > parent process although I have another project coming up in which this > would be a good idea. > > at this point, I know they will be some kind souls suggesting various > SQL solutions. While I appreciate the idea, unfortunately I do not have > time to puzzle out yet another component. Someday I will figure it out > because I really liked what I see with SQL lite but unfortunately, today > is not that day (unless they will give me their work, home and cell > phone numbers so I can call when I am stuck. ;-)
I'm sure we could agree on a fee for me to do so :) > So the solutions that come to mind are some form of dictionary in shared > memory with locking semaphore scoreboard or a multithreaded process > containing a single database (Python native dictionary, metakit, gdbm??) > and have all of my processes speak to it using xmlrpc which leaves me > with the question of how to make a multithreaded server using stock xmlrpc. berkley db (at least version 3, http://pybsddb.sourceforge.net/) supports multiple readers and writers, with fine-grained locking, it looks like a dictionary, and it isn't sql. The use you have in mind is a bit more complicated than the simple create-me-a-dictionary-in-a-file, but is pretty straightforward. The documentation mostly refers you to the C API, but fortunately it (the C API) is clear and well written. HTH -- John Lenton ([EMAIL PROTECTED]) -- Random fortune: Today is National Existential Ennui Awareness Day.
signature.asc
Description: Digital signature
-- http://mail.python.org/mailman/listinfo/python-list