Re: simultaneous multiple requests to very simple database
I agree with you, there's a crying need for something like that and there's no single "one obvious way to do it" answer. Have you looked at bsddb? See also www.sleepycat.com. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
Just learned of this today, so I don't know enough details to judge its suitability for you: Durus http://www.mems-exchange.org/software/durus/ It does not do locking, but alleges to be compact and easy to understand, so perhaps you could modify it to meet your needs, or find some other way to handle that requirement. -Tom -- To respond by email, replace "somewhere" with "astro" in the return address. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
On Tue, 18 Jan 2005 11:26:46 -0500, Eric S. Johansson wrote: > So the solutions that come to mind are some form of dictionary in shared > memory with locking semaphore scoreboard or a multithreaded process > containing a single database (Python native dictionary, metakit, gdbm??) > and have all of my processes speak to it using xmlrpc which leaves me > with the question of how to make a multithreaded server using stock > xmlrpc. Another solution might be to store the records as files in a directory, and use file locking to control access to the files (careful over NFS!). You might also consider berkeley db, which is a simple database to add to an application, (and which I believe supports locks), but I must admit I'm not a fan of the library. I assume that the bottleneck is processing the records, otherwise this all seems a bit academic. Jeremy -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
On Tue, 18 Jan 2005 12:57:21 -0500, Eric S. Johansson <[EMAIL PROTECTED]> wrote: Robert Brewer wrote: Eric S. Johansson wrote: I have an application where I need a very simple database, effectively a very large dictionary. The very large dictionary must be accessed from multiple processes simultaneously. I need to be able to lock records within the very large dictionary when records are written to. Just to clarify, you want shared-read until a write, at which point you want to lock just the item being written? Or would page or table locking be acceptable at that point? just the item/record. I'm doing arrival rate calculations. each record contains a set of arrival times and I am rewriting the record every time a new entry arrives. complete page or table locking will work in the sense that it will prevent collisions but it will have an increasing impact as load and simultaneous table but not record accesses increase. ---eric Use Firebird as sql backend. Is designed as you request (readers not lock writers and writers not lock readers). Google for "firebird optimistic lock". Off course, you have python driver: http://kinterbasdb.sf.net and can deploy on windows and linux with a very little footprint. -- Olaf Zetanien -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
Olaf Zetanien wrote: > > Use Firebird as sql backend. Is designed as you request (readers not lock > writers and writers not lock readers). Google for "firebird optimistic > lock". > > Off course, you have python driver: http://kinterbasdb.sf.net and can > deploy on windows and linux with a very little footprint. > And OS X. But it fails the OP's "no SQL" test. -- Website: www DOT jarmania FULLSTOP com -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
On Tue, Jan 18, 2005 at 11:26:46AM -0500, Eric S. Johansson wrote: > I have an application where I need a very simple database, effectively a > very large dictionary. The very large dictionary must be accessed from > multiple processes simultaneously. I need to be able to lock records > within the very large dictionary when records are written to. Estimated > number of records will be in the ballpark of 50,000 to 100,000 in his > early phase and 10 times that in the future. Each record will run about > 100 to 150 bytes. > > speed is not a huge concern although I must complete processing in less > than 90 seconds. The longer the delay however the greater number of > processes must be running parallel in order to keep the throughput up. > It's the usual trade-off we have all come to know and love. > > it is not necessary for the dictionary to persist beyond the life of the > parent process although I have another project coming up in which this > would be a good idea. > > at this point, I know they will be some kind souls suggesting various > SQL solutions. While I appreciate the idea, unfortunately I do not have > time to puzzle out yet another component. Someday I will figure it out > because I really liked what I see with SQL lite but unfortunately, today > is not that day (unless they will give me their work, home and cell > phone numbers so I can call when I am stuck. ;-) I'm sure we could agree on a fee for me to do so :) > So the solutions that come to mind are some form of dictionary in shared > memory with locking semaphore scoreboard or a multithreaded process > containing a single database (Python native dictionary, metakit, gdbm??) > and have all of my processes speak to it using xmlrpc which leaves me > with the question of how to make a multithreaded server using stock xmlrpc. berkley db (at least version 3, http://pybsddb.sourceforge.net/) supports multiple readers and writers, with fine-grained locking, it looks like a dictionary, and it isn't sql. The use you have in mind is a bit more complicated than the simple create-me-a-dictionary-in-a-file, but is pretty straightforward. The documentation mostly refers you to the C API, but fortunately it (the C API) is clear and well written. HTH -- John Lenton ([EMAIL PROTECTED]) -- Random fortune: Today is National Existential Ennui Awareness Day. signature.asc Description: Digital signature -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
Thomas Bartkus wrote: When you write that "super dictionary", be sure to post code! I could use one of those myself. hmmm it looks like you have just flung down the gauntlet of "put up or quityerwhinging". I need to get the crude implementation done first but I think I can do it if I can find a good XMLRPC multithreading framework. ---eric -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
"Eric S. Johansson" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > 99.9 percent of what I do (and I suspect this could be true for others) > could be satisfied by a slightly enhanced super dictionary with a record > level locking. BUT - Did you not mention! : > Estimated number of records will be in the ballpark of 50,000 to 100,000 in his > early phase and 10 times that in the future. Each record will run about > 100 to 150 bytes. . And > The very large dictionary must be accessed from > multiple processes simultaneously And > I need to be able to lock records > within the very large dictionary when records are written to And > although I must complete processing in less than 90 seconds. And - the hole in the bottom of the hull - all of the above using "a slightly enhanced super dictionary". *Super* dictionary??? *Slightly* enhanced??? Have you attempted any feasability tests? Are you running a Cray? There are many database systems available, and Python (probably) has free bindings to every one of them. Whichever one might choose, it would add simplicity, not complexity to what you are attempting. The problems you mention are precisely those that databases are meant to solve. The only tough (impossible?) requirement you have is that you don't want to use one. When you write that "super dictionary", be sure to post code! I could use one of those myself. Thomas Bartkus -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
Ricardo Bugalho wrote: On Tue, 18 Jan 2005 17:33:26 -0500, Eric S. Johansson wrote: When I look at databases, I see a bunch of very good solutions that are either overly complex or heavyweight on one hand and very nice and simple but unable to deal with concurrency on the other. two sets of point solutions that try to stretch themselves and the developers to fit other application contexts. Have you considerded SQLite/pySQLite ? yep and apparently it won't work http://www.sqlite.org/faq.html#q7 if I had record level locking, the code would do a very common pattern like: if record present: Lock record modify record release lock else: create record atomically (actual method TBB) if I read their opinion correctly, the SQL lite folks are wrong in that only the applications need massive concurrency. Small applications need significant to massive concurrency for very tiny windows on very little data. but I do appreciate the pointer. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
On Tue, 18 Jan 2005 17:33:26 -0500, Eric S. Johansson wrote: > When I look at databases, I see a bunch of very good solutions that are > either overly complex or heavyweight on one hand and very nice and simple > but unable to deal with concurrency on the other. two sets of point > solutions that try to stretch themselves and the developers to fit other > application contexts. > Have you considerded SQLite/pySQLite ? -- Ricardo -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
On Tue, 18 Jan 2005 17:33:26 -0500, Eric S. Johansson <[EMAIL PROTECTED]> wrote: > so in conclusion, my only reason for querying was to see if I was > missing a solution. So far, I have not found any work using because > they add orders of magnitude more complexity than simple dbm with file > locking. Obviously, the simple solution has horrible performance right > now I need simplicity implementation. > > thanks for your commentary. Maybe you can just get the best of both worlds. Have a look at SQLObject. You can ignore the fact that underneath the SQLObject there's a postgres (or mysql, or whatever) database, and get OO based persistance. SQLObject is crippled in that there are degrees of freedom that SQL gives you that SQLObject takes away/makes hard to use, but what you're trying to do, and what most people actually do with databases, can be easily wrapped around with a simple, pythonic wrapper. It even has a .createTable() function for those times when you don't even want to log into the database. Regards, Stephen Thorne. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
Thomas Bartkus wrote: "Eric S. Johansson" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] at this point, I know they will be some kind souls suggesting various SQL solutions. While I appreciate the idea, unfortunately I do not have time to puzzle out yet another component. Someday I will figure it out because I really liked what I see with SQL lite but unfortunately, today is not that day (unless they will give me their work, home and cell phone numbers so I can call when I am stuck. ;-) Forgive me if this reply sounds a bit glib. But I do mean it without malice. understood and taken in that spirit. Do you seriously expect to write your own (database) solution and that this will save you time and effort over learning an existing (SQL) solution? Because - If you are seeking to "save time" on "puzzles", you are certainly going about it the wrong way. one thing I learned a long time ago was to respect the nagging voice in the back of my head that says "there is something wrong". Right now with databases, that voice is not nagging but screaming. So I made my query to try and prove that intuition wrong. So far, that has not happened. When I look at databases, I see a bunch of very good solutions that are either overly complex or heavyweight on one hand and very nice and simple but unable to deal with concurrency on the other. two sets of point solutions that try to stretch themselves and the developers to fit other application contexts. 99.9 percent of what I do (and I suspect this could be true for others) could be satisfied by a slightly enhanced super dictionary with a record level locking. but, the database world does not fit this model. It has a great deal more complication then what is frequently necessary. If I ever find the time, I will try to build such a beast probably around Metakit. The only reason for reluctance is that I have spent too many hours tracking down concurrency problems at the OS level way to many years ago and so I do not create multithreaded applications lightly. so in conclusion, my only reason for querying was to see if I was missing a solution. So far, I have not found any work using because they add orders of magnitude more complexity than simple dbm with file locking. Obviously, the simple solution has horrible performance right now I need simplicity implementation. thanks for your commentary. ---eric -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
"Eric S. Johansson" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > at this point, I know they will be some kind souls suggesting various > SQL solutions. While I appreciate the idea, unfortunately I do not have > time to puzzle out yet another component. Someday I will figure it out > because I really liked what I see with SQL lite but unfortunately, today > is not that day (unless they will give me their work, home and cell > phone numbers so I can call when I am stuck. ;-) Forgive me if this reply sounds a bit glib. But I do mean it without malice. Do you seriously expect to write your own (database) solution and that this will save you time and effort over learning an existing (SQL) solution? Because - If you are seeking to "save time" on "puzzles", you are certainly going about it the wrong way. Best of luck Thomas Bartkus -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous multiple requests to very simple database
Robert Brewer wrote: Eric S. Johansson wrote: I have an application where I need a very simple database, effectively a very large dictionary. The very large dictionary must be accessed from multiple processes simultaneously. I need to be able to lock records within the very large dictionary when records are written to. Just to clarify, you want shared-read until a write, at which point you want to lock just the item being written? Or would page or table locking be acceptable at that point? just the item/record. I'm doing arrival rate calculations. each record contains a set of arrival times and I am rewriting the record every time a new entry arrives. complete page or table locking will work in the sense that it will prevent collisions but it will have an increasing impact as load and simultaneous table but not record accesses increase. ---eric -- http://mail.python.org/mailman/listinfo/python-list
RE: simultaneous multiple requests to very simple database
Eric S. Johansson wrote: > I have an application where I need a very simple database, > effectively a very large dictionary. The very large > dictionary must be accessed from multiple processes > simultaneously. I need to be able to lock records within > the very large dictionary when records are written to. Just to clarify, you want shared-read until a write, at which point you want to lock just the item being written? Or would page or table locking be acceptable at that point? Robert Brewer MIS Amor Ministries [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
simultaneous multiple requests to very simple database
I have an application where I need a very simple database, effectively a very large dictionary. The very large dictionary must be accessed from multiple processes simultaneously. I need to be able to lock records within the very large dictionary when records are written to. Estimated number of records will be in the ballpark of 50,000 to 100,000 in his early phase and 10 times that in the future. Each record will run about 100 to 150 bytes. speed is not a huge concern although I must complete processing in less than 90 seconds. The longer the delay however the greater number of processes must be running parallel in order to keep the throughput up. It's the usual trade-off we have all come to know and love. it is not necessary for the dictionary to persist beyond the life of the parent process although I have another project coming up in which this would be a good idea. at this point, I know they will be some kind souls suggesting various SQL solutions. While I appreciate the idea, unfortunately I do not have time to puzzle out yet another component. Someday I will figure it out because I really liked what I see with SQL lite but unfortunately, today is not that day (unless they will give me their work, home and cell phone numbers so I can call when I am stuck. ;-) So the solutions that come to mind are some form of dictionary in shared memory with locking semaphore scoreboard or a multithreaded process containing a single database (Python native dictionary, metakit, gdbm??) and have all of my processes speak to it using xmlrpc which leaves me with the question of how to make a multithreaded server using stock xmlrpc. so feedback and pointers to information would be most welcome. I'm still exploring the idea so I am open to any and all suggestions (except maybe SQL :-) ---eric -- http://mail.python.org/mailman/listinfo/python-list
