Re: [ZODB-Dev] Twisted, ZODB and zc.twist
On Jul 29, 2011, at 4:38 AM, Erik Allik wrote: > Thanks for another clear answer! I'll simply ignore zc.twist (for now > at least) and set up a classic thread pool. > I'll still probably use zc.twist's source as an example of how to > write transaction retrial on ConflicErrors and ClientDisconnecteds as > I haven't found anything similar elsewhere. Cool. FWIW, http://pypi.python.org/pypi/zc.queue does something similar without threads. I prefer it (despite the fact that I said that "this testing approach has known weaknesses") if your use cases allow it. Gary > > Erik > > > > 2011/7/29 Gary Poster : >> >> On Jul 28, 2011, at 6:38 PM, Erik Allik wrote: >> >>> Gary, since you're the author of zc.twist (as I understand), I take >>> this as an exhaustive answer, and thank you :) >>> >>> However, I'm still curious to know zc.twist would provide any benefit >>> when used together with a thread pool. >>> The thread examples in its README (which I find hard to understand) >>> indicate that there must be some meaning to using zc.twist with >>> threads. >> >> The threads in the README are there to let me explore a running reactor >> within the doctest. They are not anything to emulate. >> >> The README overstates zc.twist's usefulness. As you point out, it blocks. >> If you assume that the ZODB is functioning normally, it won't block "much" >> but with standard definitions of blocking, it fails. For zc.async, that's >> fine--but a "while True" loop and a time.sleep() to give some breathing room >> probably would have been fine, with hindsight, and much simpler. The use of >> Twisted is a very questionable convenience, and the way zc.async uses >> Twisted is not to be copied 99% or more of the time. >> >> Gary >> >> >>> >>> Thanks again and best regards, >>> Erik >>> >>> >>> >>> 2011/7/29 Gary Poster : >>>> Hello Erik. As the author, I'll suggest that you move on to a ZODB >>>> connection pool used by the usual Twisted-provided thread pool. zc.twist >>>> is maybe interesting and maybe clever, but of very limited usefulness. I >>>> suspect it is only used in zc.async, and that's probably as it should be. >>>> It's a case of premature generalization. >>>> >>>> If you are going to use Twisted in anything like a normal way, and the >>>> ZODB in anything like a normal way, use a connection pool and threads. >>>> >>>> Best, >>>> >>>> Gary >>>> >>>> >>>> On Jul 28, 2011, at 4:12 PM, Erik Allik wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> I hope this is the right mailing list to post this question to. >>>>> >>>>> I'm trying to build an Twisted application that uses ZODB and I've >>>>> also found zc.twist. >>>>> I've read its README as well as the source code to try to understand >>>>> how exactly and why I should use zc.twist, and also asked the same >>>>> questions on irc.freenode.net #zope but without a conclusive answer. >>>>> >>>>> What I'm not able to understand is whether using zc.twist will allow >>>>> me to do all ZODB access in the main Twisted thread, or if still need >>>>> to set up a thread pool and deferToThread any code that accesses the >>>>> DB. >>>>> >>>>> The README of zc.twist has: >>>>> "Everything can be done within the main thread, so it can be full-bore >>>>> Twisted usage, without threads." >>>>> >>>>> However, looking at the source code of zc.twist, I cannot find any >>>>> reference to async communication (callbacks/polling) with ZODB, so >>>>> it's difficult to conclude that all DB access is non-blocking and can >>>>> be done in a single thread. Also, there are some examples with threads >>>>> in the README. >>>>> >>>>> I would highly appreciate if somebody with a clearer understanding of >>>>> this could explain to me whether I in fact need to set up a thread >>>>> pool, and in any case, what exactly is zc.twist for regardless of >>>>> whether it needs to be used with a threadpool or not. >>>>> >>>>> Thanks in advance! >>>>> >>>>> Regards, >>>>> Erik Allik >>>>> >>>>> P.S. I'd be very thankful for any references to open source >>>>> projects/code using ZODB with Twisted, or even zc.twist (besides >>>>> zc.async). >>>>> ___ >>>>> For more information about ZODB, see the ZODB Wiki: >>>>> http://www.zope.org/Wikis/ZODB/ >>>>> >>>>> ZODB-Dev mailing list - ZODB-Dev@zope.org >>>>> https://mail.zope.org/mailman/listinfo/zodb-dev >>>> >>>> >> >> ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Twisted, ZODB and zc.twist
On Jul 28, 2011, at 6:38 PM, Erik Allik wrote: > Gary, since you're the author of zc.twist (as I understand), I take > this as an exhaustive answer, and thank you :) > > However, I'm still curious to know zc.twist would provide any benefit > when used together with a thread pool. > The thread examples in its README (which I find hard to understand) > indicate that there must be some meaning to using zc.twist with > threads. The threads in the README are there to let me explore a running reactor within the doctest. They are not anything to emulate. The README overstates zc.twist's usefulness. As you point out, it blocks. If you assume that the ZODB is functioning normally, it won't block "much" but with standard definitions of blocking, it fails. For zc.async, that's fine--but a "while True" loop and a time.sleep() to give some breathing room probably would have been fine, with hindsight, and much simpler. The use of Twisted is a very questionable convenience, and the way zc.async uses Twisted is not to be copied 99% or more of the time. Gary > > Thanks again and best regards, > Erik > > > > 2011/7/29 Gary Poster : >> Hello Erik. As the author, I'll suggest that you move on to a ZODB >> connection pool used by the usual Twisted-provided thread pool. zc.twist is >> maybe interesting and maybe clever, but of very limited usefulness. I >> suspect it is only used in zc.async, and that's probably as it should be. >> It's a case of premature generalization. >> >> If you are going to use Twisted in anything like a normal way, and the ZODB >> in anything like a normal way, use a connection pool and threads. >> >> Best, >> >> Gary >> >> >> On Jul 28, 2011, at 4:12 PM, Erik Allik wrote: >> >>> Hello everyone, >>> >>> I hope this is the right mailing list to post this question to. >>> >>> I'm trying to build an Twisted application that uses ZODB and I've >>> also found zc.twist. >>> I've read its README as well as the source code to try to understand >>> how exactly and why I should use zc.twist, and also asked the same >>> questions on irc.freenode.net #zope but without a conclusive answer. >>> >>> What I'm not able to understand is whether using zc.twist will allow >>> me to do all ZODB access in the main Twisted thread, or if still need >>> to set up a thread pool and deferToThread any code that accesses the >>> DB. >>> >>> The README of zc.twist has: >>> "Everything can be done within the main thread, so it can be full-bore >>> Twisted usage, without threads." >>> >>> However, looking at the source code of zc.twist, I cannot find any >>> reference to async communication (callbacks/polling) with ZODB, so >>> it's difficult to conclude that all DB access is non-blocking and can >>> be done in a single thread. Also, there are some examples with threads >>> in the README. >>> >>> I would highly appreciate if somebody with a clearer understanding of >>> this could explain to me whether I in fact need to set up a thread >>> pool, and in any case, what exactly is zc.twist for regardless of >>> whether it needs to be used with a threadpool or not. >>> >>> Thanks in advance! >>> >>> Regards, >>> Erik Allik >>> >>> P.S. I'd be very thankful for any references to open source >>> projects/code using ZODB with Twisted, or even zc.twist (besides >>> zc.async). >>> ___ >>> For more information about ZODB, see the ZODB Wiki: >>> http://www.zope.org/Wikis/ZODB/ >>> >>> ZODB-Dev mailing list - ZODB-Dev@zope.org >>> https://mail.zope.org/mailman/listinfo/zodb-dev >> >> ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Twisted, ZODB and zc.twist
Hello Erik. As the author, I'll suggest that you move on to a ZODB connection pool used by the usual Twisted-provided thread pool. zc.twist is maybe interesting and maybe clever, but of very limited usefulness. I suspect it is only used in zc.async, and that's probably as it should be. It's a case of premature generalization. If you are going to use Twisted in anything like a normal way, and the ZODB in anything like a normal way, use a connection pool and threads. Best, Gary On Jul 28, 2011, at 4:12 PM, Erik Allik wrote: > Hello everyone, > > I hope this is the right mailing list to post this question to. > > I'm trying to build an Twisted application that uses ZODB and I've > also found zc.twist. > I've read its README as well as the source code to try to understand > how exactly and why I should use zc.twist, and also asked the same > questions on irc.freenode.net #zope but without a conclusive answer. > > What I'm not able to understand is whether using zc.twist will allow > me to do all ZODB access in the main Twisted thread, or if still need > to set up a thread pool and deferToThread any code that accesses the > DB. > > The README of zc.twist has: > "Everything can be done within the main thread, so it can be full-bore > Twisted usage, without threads." > > However, looking at the source code of zc.twist, I cannot find any > reference to async communication (callbacks/polling) with ZODB, so > it's difficult to conclude that all DB access is non-blocking and can > be done in a single thread. Also, there are some examples with threads > in the README. > > I would highly appreciate if somebody with a clearer understanding of > this could explain to me whether I in fact need to set up a thread > pool, and in any case, what exactly is zc.twist for regardless of > whether it needs to be used with a threadpool or not. > > Thanks in advance! > > Regards, > Erik Allik > > P.S. I'd be very thankful for any references to open source > projects/code using ZODB with Twisted, or even zc.twist (besides > zc.async). > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] big list (was Re: Making ZODB / ZEO faster)
On Dec 7, 2009, at 11:08 AM, Erik Dahl wrote: ... > Our slow loading object was a persistent with a regular list inside of > the main pickle. Objects that the list pointed to were persistent > which I believe means that hey will load separately. In general we > have tried to make our persistent objects reasonably large to lower > the amount of load round trips. I haven't actually checked its size > yet but it will be interesting to see. Even if the list holds just persistent objects, if there are a lot of them, just loading the references can still be a big problem. That would point to refactoring that part of the app, and/or investigating if the characteristics of things like zc.queue or zc.blist would help. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Runaway cache size
I'm the primary zc.catalog author. He is running read-only, so savepoints shouldn't be very pertinent. The original poster is correct that the cache is cleaned out at transaction boundaries (or savepoints, as Alan said). I don't have experience with running the db in readonly mode in production. That said, if you have obvious boundaries for your searches, I suspect that beginning a transaction at the boundary would probably do the trick, and also work (with invalidations and the like) if you ever have a readonly connection to a ZEO database server that is getting updates from another source. While it's interesting to contemplate the knob that Alan mentioned, I'm reasonably confident that this logic belongs in the app, not the index or catalog. Gary On Jul 31, 2008, at 11:55 AM, Alan Runyan wrote: I believe it is common to have a theshold (say every 1) objects to do a savepoint and then you can run cacheMinimize after the savepoint. surprised the zc.catalog does not provide this sort of knob. maybe it was a problem in the original ZCatalog implementation? cheers alan On Wed, Jul 30, 2008 at 1:21 PM, <[EMAIL PROTECTED]> wrote: I have a database consisting mainly of an IOBTree of ~700,000 items (each persistent mappings), and an zc.catalog indexing them by one of the mapping's fields (a date). I am opening the FileStorage in read_only mode. For each day in the index, I get the day's mappings and read the contents of another field. As I go through each day, the memory usage explodes (over 32 GB). Is there a way to configure the cache to automatically keep itself under the value of the cache size parameter? To avoid this problem in most cases, I wrap the IOBTree in another object which does nothing more than call db.cacheMinimize after every 1 items are iterated over. But for random access, that's not an option. From reading the archives, it sounds like cache cleaning does not happen while running a transaction. Is that my problem? I'm in read-only mode, so I can't perform a transaction anyway, but could the ZODB think I want to? Or is my problem something else? Maybe if someone can point me to a description of how the caching works. -- Anthony Foglia Princeton Consultants (609) 987-8787 x233 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Uh-oh, I'm implicated! (see below) On Jun 3, 2008, at 12:53 PM, Sean Allen wrote: ... If you do that in gemstone, there is only one copy of Order B, no matter what variable in what dictionary you come at it from. And its drop dead simple. I looked at implementing that with zodb and moved along. I'm confused. This has been the way the ZODB worked for a long time, unless I'm really missing something in your description. i tried to do this: create customer that has order so that i can have different extents type situations... store customer in one dictionary. store order in another. if i pulled the order back out from the order dictionary and modified it then pulled the customer out, the customers order was no longer in sync with what came out of the order dictionary. the reference was lost on serialization. original in memory objects were fine, those that came back out from zodb werent. i'm going to quote the initial email i sent with the idea in general and the followup i got and i then tried it out to make sure i hadnt asked the question wrong, and yeah... what i wanted to do, wasnt easily done. the quotes: The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back references are not maintained automatically. I'd identify two classic solutions to this sort of thing. One is to make a custom mapping (using a BTree as the inner data structure) that maintains back-references when objects are placed in them or removed. zope.app(.container? .folder? I'd have to look) has code that does this, along with firing events. For simple stories like the one you describe here, that's what I'd probably recommend. It works to the strengths of the ZODB, which particularly shines in terms of readability when you just need to walk a tree of attributes to get what you want. The other is to keep an external index, a la zc.extrinsicreference or zc.relation. zc.extrinsicreference does not have too many dependencies beyond ZODB, and as long as zope.app.keyreference doesn't drag much along with it, might be usable as a library. That said, it's also very simple, and could be used as a model for you, even if you don't use it directly. It would also be a reasonable choice for a simple situation like the one you describe. It relies on events to update its data structures. zc.relation an almost-released-revision of zc.relationship that drastically reduces dependencies--actually, it has no additional dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup . It's also a bit overwhelming and low-level: see the README:http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto . It doesn't hook anything up for you: you set the relationship catalog up and you arrange for it to be updated, via events or direct messages. That said, if you need its power, it is well- tested and would be a good choice for some jobs from at least some perspectives (caveat read-or: I'm the author). Now in the context of this discussion, I see that I misread you. I apologize. This works out of the box: You have a Product class and a Brand class. Both inherit from persistent.Persistent. You have a persistent-aware mapping such as a BTree. In the root of your ZODB, put two BTrees, one for your products and one for your brands. Create some Brand instances and put them in the brand mapping. Create some Product instances and put them in the product mapping. Assign some of the brands you have made as attributes of the products. Now, product.brand.foo = 'bar' (in any thread or any ZEO client) will be changing the same effective object (let's call it a record) as the one in the brand mapping you have in the root of your db. I believe that this is what you are talking about. Again, I apologize for not reading your original question closely enough. What you *can't* do out of the box is ask "hey, what products have an attribute that points to this brand?". That's a back-reference, and that needs solutions like the ones to which I was ref
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 20, 2008, at 11:04 PM, Sean Allen wrote: I've been playing with this and I'm still having a disconnect 3 week later. 1. I love the zc.relation stuff it answers one area of stuff I hadn't even gotten to yet. cool :-) 2. I'm just totally missing something less cool. :-/ If I have Customers and Orders and I want to be able update all of them independently of their relationships ( so that if an order is updated, when i get the customer at some later time, it has the updated order amongst that relation ) but I can't figure it out. I tried looking at the zope.app.folder stuff but I keep getting lost in the zope aspects of it and am having a hard time seeing the forest for the trees. Is there some more general ready on the patterns used that you know of? I think if I understood the idea more in an abstract sense, I could get a lot more out of the folder implementation. Here's a simple, dumb example that parallels the folder stuff. See the Dict class in http://svn.zope.org/zc.async/branches/dev/src/zc/async/utils.py?rev=84657&view=auto . You put something in the dict, and __setitem__ slams a name and a parent. The other mutating methods should do the right thing as well in terms of updating the back reference. So, completely randomly and arbitrarily, but to try and make a parallel, what if customers were a dict of orders, and when you made an order it was associated with only one customer, and you put the order "in" the customer. This may be bizarre--what's the key? can more than one customer be associated with an order? But that would mean that customer.values() would get all of the customer's orders, and order.parent would be the associated customer. This example subclasses zc.dict, a super simple package that only depends on ZODB, btw. It does not have full dict behavior, as Jim likes to point out, because items are stored by BTree sorting, not hashes, but it looks like a dict otherwise. Maybe it can help you out. If you want to try out this example, copy the code out and get the zc.dict egg and give it a try. Anyway, this pattern of directly manipulating back-references is good for intrinsic relationships like customers and orders. Hope that helps a bit, Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Anyone mind looking over a bit of code?
Hi Kenneth. First, make sure you do a transaction.begin() (or transaction.abort()) before you view your database for a given operation. This will sync invalidations for a given transaction, even if it will only be a read transaction, and reset the MVCC view. The partial-view behavior you describe sounds like it might have been possible pre-MVCC, but seems pretty unlikely now, unless you are committing after each individual insertion rather than after the aggregated insertion. ...Actually, looking through your code in newData, that appears to be what you are doing: if the object is iterable, you recursively call newData on the sub-elements, with a commit for each insertion. In that case, it would be reasonable for a concurrent reader to get part of the insertions while the writer continues to write. Because of MVCC, the reader will not see the subsequent insertions until it makes another transaction commit/abort/begin. There are some various other oddities to my eye--why do a ``while`` on the BTree insertion when you don't mutate the key?--but they don't look like the cause of the behavior you describe. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo Server as a single point of failure
On Mar 10, 2008, at 6:34 PM, Kenneth Miller wrote: Hello all, I'm currently developing my application ontop of zodb/zeo, and the idea of having the Zeo server the only single point of failure still bothers me. My question is, has anyone thought of any solutions for this. My idea was to have a few or more background processes that kept multiple servers synced and concurrent. Any thoughts? - Zope Corp sells ZRS (http://www.zope.com/products/zope_replication_services.html ) - Shane Hathaway's RelStorage (http://wiki.zope.org/ZODB/RelStorage) promises to accomplish some of the same sort of thing with (for instance) Postgres tools though the page I listed says that this is untested. - Christian Theune is working on ZEO RAID: http://pypi.python.org/pypi/gocept.zeoraid/dev Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Feb 27, 2008, at 10:03 AM, Sean Allen wrote: On Feb 27, 2008, at 5:33 AM, Laurence Rowe wrote: Sean Allen wrote: been looking for anything along those lines. in particular, strategies and gotchas for how to store objects. everything i've found is basically just a single type of object being stored. i'm really interested in tutorials and information on the best ways to setup large complicated hierarchies and all the various gotchas etc. anything like that exist? A good jumping off point is the wiki: http://wiki.zope.org/ZODB/Documentation The beauty of using the ZODB is that it really makes no difference whether you are storing homogeneous or heterogeneous hierarchies of objects. If you are likely to have a large number of objects in any one folder then make sure it is BTree based. This is a scalable data structure, so the whole list does not have to be loaded into memory to access a single child object. Been over that. Still have serious questions. And yes, we have a large number of objects, something in the area of 16 million at present if you were to map each relational table row to an object. The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back references are not maintained automatically. I'd identify two classic solutions to this sort of thing. One is to make a custom mapping (using a BTree as the inner data structure) that maintains back-references when objects are placed in them or removed. zope.app(.container? .folder? I'd have to look) has code that does this, along with firing events. For simple stories like the one you describe here, that's what I'd probably recommend. It works to the strengths of the ZODB, which particularly shines in terms of readability when you just need to walk a tree of attributes to get what you want. The other is to keep an external index, a la zc.extrinsicreference or zc.relation. zc.extrinsicreference does not have too many dependencies beyond ZODB, and as long as zope.app.keyreference doesn't drag much along with it, might be usable as a library. That said, it's also very simple, and could be used as a model for you, even if you don't use it directly. It would also be a reasonable choice for a simple situation like the one you describe. It relies on events to update its data structures. zc.relation an almost-released-revision of zc.relationship that drastically reduces dependencies--actually, it has no additional dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup . It's also a bit overwhelming and low-level: see the README: http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto . It doesn't hook anything up for you: you set the relationship catalog up and you arrange for it to be updated, via events or direct messages. That said, if you need its power, it is well-tested and would be a good choice for some jobs from at least some perspectives (caveat read-or: I'm the author). HTH Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: memory exhaustion problem
On Jan 17, 2008, at 11:57 AM, Flavio Coelho wrote: Some progress! Apparently the combination of: u._p_deactivate() transaction.savepoint(True) transaction.commit() helped. Memory consumption keeps growing but much more slowly (about 1/5 of the original speed). Please correct me if I am wrong, but I believe that ideally memory usage should stay constant throughout the loop, shouldn't it? Are you still using the example code you gave in a previous message? If so, you are generating and storing user objects in a list, which grows. If not, maybe give a snippet of your current example code. Moreover, I shouldn't need to commit either, since I am not modifying the objects... You need to tell the ZODB about the transaction boundary so it knows when to "let go". transaction.abort() would work too. You could also manually much with the persistent object cache I suppose, though _p_deactivate should be doing most of what would help... "ghosts" (deactivated objects) still take up some memory. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote: David Binger mems-exchange.org> writes: On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote: Ie we perhaps look at a catalog data structure in which writes are initially done to some kind of queue then moved to the BTrees at a later point. A suggestion: use a pair of BTrees, one with a high branching factor (bucket size) and one with a very low branching factor. Force all writes into the tree with little buckets. Make every search look in both trees. Consolidate occasionally. An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items in the original BTree with the regular bucket size/ branch factor. maybe. haven't thought it through, but worth thinking about. idle thought I should probably not share: you could use a Bucket directly for that--it will never split at all, and has the conflict resolution behavior. (strangely, I'm not idle at all, but rather overwhelmingly busy ;-) ) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Memory Errors reading large ZODB
On Aug 2, 2007, at 9:33 AM, Jim Fulton wrote: On Aug 2, 2007, at 7:51 AM, Chris Withers wrote: Gary Poster wrote: you can call cache minimize after a threshold.. maybe every 100 iterations. sounds good, assuming you know you are not writing. I've used this trick loads, especially for huge datastructure migrations where writing is happening. I wonder why I haven't bumped into problems? I'm not sure what Gary was referring to. I don't think there are any problems with writing. OTOH, modified objects won't be removed from memory. Perhaps that was what he was referring to. Yup. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Memory Errors reading large ZODB
On Aug 1, 2007, at 4:16 PM, Alan Runyan wrote: snip... line 96, in __setstate__ Persistent.__setstate__(self, state) MemoryError means you have run out of memory. check your jail or how much memory you have allocated. you can call cache minimize after a threshold.. maybe every 100 iterations. sounds good, assuming you know you are not writing. If you think you might be writing but don't intend to, and are willing to be sloppy, you probably could use transaction.abort() instead. If you were writing, the old trick was transaction.commit(1) and the new one is transaction.savepoint(). there are finer grain mechanisms to do this but i cant remember them off the top of my head. http://svn.zope.org/ZODB/trunk/src/ZODB/interfaces.py You are working with an old version of ZODB though, you said, so be careful looking at the trunk's interface-it might not describe precisely what you are using (savepoint is particularly pertinent here). Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems with ZODB3-3.9.0_dev_r77011
On Jul 19, 2007, at 6:36 PM, Tobias Rodäbel wrote: Hi Gary, thanks a lot for your response! Am 19.07.2007 um 15:52 schrieb Gary Poster: - Is this consistently reproduceable, or intermittent? Unless you are intentionally creating a conflict in a test, any errors in the changes in 3.9 would be more likely to be intermittent. It is consistently reproduceable. - Even better, can you construct a small, distributable test case? That would certainly invite more help. - Have you tried to reproduce with the most recent zope.app.keyreference in the 3.4 line and the most recent ZODB 3.8 line? If so, that might get Jim's attention, and would rule out the relatively small changes in the 3.9 dev egg. Unless you like riding the bleeding edge, I might suggest using those earlier versions for now anyway. I did. It worked well. Here are some code snippets I use concerning to this problem, I guess. Maybe it has something to do with that location.LocationProxy thing. (I found most of the code here http:// readlist.com/lists/zope.org/zope3-users/0/2518.html ) Hi Tobias Sounds even more like Roger was right, then. Make sure that the object you persist is not actually wrapped in the location proxy. Here's the most pertinent snippet from what Roger sent you: def set(self, photo): if photo is not None: # remove location proxy because the ZODB doesn't like it anymore photo = zope.proxy.getProxiedObject(photo) self._photo = photo (notice that `getProxiedObject` call, and the comment) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: AW: [Zope3-dev] Re: [ZODB-Dev] Problems with ZODB3-3.9.0_dev_r77011
On Jul 19, 2007, at 10:36 AM, Roger Ineichen wrote:
Hi Tobias
Auftrag von Gary Poster
Gesendet: Donnerstag, 19. Juli 2007 15:53
An: Tobias Rodäbel
[...]
zope.app.keyreference-3.5.0_dev_r77018-py2.4.egg requires
ZODB3>=3.9.0-dev-r77011
But there might be a caching problem within ZODB3-3.9.0 dev
r77011,
my debug session tells:
/development/Zope3/MyProject/eggs/ZODB3-3.9.0_dev_r77011-py2.4-
macosx-10.4-ppc.egg/ZODB/Connection.py(644)_store_objects()
-> raise
(Pdb) obj
(Pdb) oid
'\x00\x00\x00\x00\x00\x00\x00\xc6'
(Pdb) self._cache[oid]
*** KeyError: '\x00\x00\x00\x00\x00\x00\x00\xc6'
(Pdb) self._cache[oid] = obj
*** TypeError: Cache values must be persistent objects.
But zope.app.file.image.Image should be persistent.
[...]
Looking forward to some hints or help,
Hi Tobias. The ZODB 3.9 dev version is only different from 3.8 in
some conflict resolution code, for which I am responsible. Some
thoughts.
I'm pretty shure you ve got a LocationProxy arround your object
because the zope.app.file doesn't implement ILocation.
Hey Roger. I bet you are right.
...
Gary
-
This happens because of the call in
ZODB.Connection.py
$Id: Connection.py 75693 2007-05-11 22:18:37Z jim $
cool that we have the revision info in the file header ;-)
line: 627
except:
# Dang, I bet it's wrapped:
# TODO: Deprecate, then remove, this.
if hasattr(obj, 'aq_base'):
self._cache[oid] = obj.aq_base
else:
raise
My question is, can we improve the method:
if hasattr(obj, 'aq_base')
and remove the proxy with a method that works?
I mentioned this to Jim. If the masking of the original exception
doesn't appear to lose any useful information, he'd prefer something
like
if type(obj) is not obj.__class__:
raise ValueError('object appears to be proxied', obj)
else:
raise
IOW, don't guess, but give a more helpful error message
Gary___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems with ZODB3-3.9.0_dev_r77011
On Jul 19, 2007, at 7:11 AM, Tobias Rodäbel wrote: Hi, zope.app.keyreference-3.5.0_dev_r77018-py2.4.egg requires ZODB3>=3.9.0-dev-r77011 But there might be a caching problem within ZODB3-3.9.0 dev r77011, my debug session tells: > /development/Zope3/MyProject/eggs/ZODB3-3.9.0_dev_r77011-py2.4- macosx-10.4-ppc.egg/ZODB/Connection.py(644)_store_objects() -> raise (Pdb) obj (Pdb) oid '\x00\x00\x00\x00\x00\x00\x00\xc6' (Pdb) self._cache[oid] *** KeyError: '\x00\x00\x00\x00\x00\x00\x00\xc6' (Pdb) self._cache[oid] = obj *** TypeError: Cache values must be persistent objects. But zope.app.file.image.Image should be persistent. (Pdb) dir (obj) ['__annotations__', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__getstate__', '__hash__', '__implemented__', '__init__', '__module__', '__new__', '__providedBy__', '__provides__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__slotnames__', '__str__', '__weakref__', '_data', '_getData', '_height', '_p_activate', '_p_changed', '_p_deactivate', '_p_delattr', '_p_getattr', '_p_invalidate', '_p_jar', '_p_mtime', '_p_oid', '_p_serial', '_p_setattr', '_p_state', '_setData', '_size', '_width', 'contentType', 'data', 'getImageSize', 'getSize'] Looking forward to some hints or help, Hi Tobias. The ZODB 3.9 dev version is only different from 3.8 in some conflict resolution code, for which I am responsible. Some thoughts. - I haven't seen any errors like this yet. That's just a data point, and certainly does not necessarily invalidate your report. - Is this consistently reproduceable, or intermittent? Unless you are intentionally creating a conflict in a test, any errors in the changes in 3.9 would be more likely to be intermittent. - Even better, can you construct a small, distributable test case? That would certainly invite more help. - Have you tried to reproduce with the most recent zope.app.keyreference in the 3.4 line and the most recent ZODB 3.8 line? If so, that might get Jim's attention, and would rule out the relatively small changes in the 3.9 dev egg. Unless you like riding the bleeding edge, I might suggest using those earlier versions for now anyway. Gary___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Failing tests of ZODB on Linux 64-bit
On May 9, 2007, at 1:40 PM, Gary Poster wrote: On May 9, 2007, at 1:35 PM, Christian Theune wrote: Hey, my new buildbot just discovered this: Failure in test test32 (BTrees.tests.testBTrees.FamilyTest) Traceback (most recent call last): File "/usr/lib64/python2.4/unittest.py", line 260, in run testMethod() File "/data/buildbot/slave/ZODB/build/src/BTrees/tests/ testBTrees.py", line 1712, in test32 self.assert_(BTrees.family32.maxint + 1 not in s) File "/usr/lib64/python2.4/unittest.py", line 309, in failUnless if not expr: raise self.failureException, msg AssertionError That's mine. I'll try to get my 64 bit machine working again-- upgrade issues have killed it ATM. 64 bit machine was alive enough to repro and fix, at least. I checked in the fix (just a test change) to SVN (r75659). I have not released a ZODB egg because I don't think you need one for buildbot and Jim says he will be releasing one with some more important changes soon. Thanks Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Failing tests of ZODB on Linux 64-bit
On May 9, 2007, at 1:35 PM, Christian Theune wrote: Hey, my new buildbot just discovered this: Failure in test test32 (BTrees.tests.testBTrees.FamilyTest) Traceback (most recent call last): File "/usr/lib64/python2.4/unittest.py", line 260, in run testMethod() File "/data/buildbot/slave/ZODB/build/src/BTrees/tests/ testBTrees.py", line 1712, in test32 self.assert_(BTrees.family32.maxint + 1 not in s) File "/usr/lib64/python2.4/unittest.py", line 309, in failUnless if not expr: raise self.failureException, msg AssertionError That's mine. I'll try to get my 64 bit machine working again-- upgrade issues have killed it ATM. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] DisconnectedError
Hi. ZODB does not currently work with Py2.5. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: another reason to stop supporting versions
I assume you are more interested in seeing if you can get someone to raise a compelling -1 than in hearing a chorus of +1s. But FWIW, +1 in getting rid of versions. It sounds like it would simplify the code significantly, which would be its own "feature". Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob test failure on FreeBSD
On Mar 27, 2007, at 10:51 AM, Christian Theune wrote: Thanks for the notice, I'll look into it. Might take a while to get a FreeBSD installed as a sandbox somewhere ... FWIW, It smells more like a 64 bit vs. 32 bit machine/OS question, but that's just on the basis of fly-by-diagnosis. :-) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Feb 12, 2007, at 10:01 AM, Manuel Vazquez Acosta wrote: Hi Gary, Although the shared buffer indeed has a queue-like interface there's a catch for my needs. I need that pull method to retrieve the item in the head and to move the head forward, and not to remove the item from the buffer. Just like zc.queue I need a single copy of a item in the queue, but I also need this uniqueness over the time, so that any item which has being consumed already, does not get back to buffer. Ah, OK. FWIW, you could get the same functionality with a wrapper around a zc.queue pretty easily, of course, with at least a couple of variants off the top of my head (both involving an additional data structure that you maintain, and supplying your own `put` and `pull` that wrap the zc.queue methods, either with composition or inheritance). If, instead, you move to a multi-threaded model following the same approach you outlined before (sequential int keys, IIUC) you'll need to be careful of write conflict errors. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote: My scenario is akin a consumer-producer with shared buffer. Consumers pull items from the buffer whilst producers put items in the buffer. The buffer is an OOBTree along with an IOBTree which gives "serial" numbers to the keys of the OOBTree. I'm not sure if this is a match, but if you follow Chris' suggestions (which I recommend generally) then you might want to look at zc.queue: http://cheeseshop.python.org/pypi/zc.queue/ . Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
On Oct 5, 2006, at 9:37 AM, Jim Fulton wrote: Russ Ferriday wrote: Do you have some favorite documents that you would point me to as most accurate or up to date? No, because I haven't used ZODB documentation much myself. :) Decent clues will be enough to get started. - Doc in test or source files - web pages, - forums chats, - email threads The ZODB wiki which is here, in case you didn't know http://www.zope.org/Wikis/ZODB/FrontPage (or Google for ZODB wiki -> "I'm feeling lucky" ;-) has some pointers In particular, I usually point people here (which is a link from the wiki): http://www.zope.org/Wikis/ZODB/FrontPage/guide/index.html Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Newbie questions re persistent objects and connections: Summary
On Sep 13, 2006, at 3:42 AM, Edward K. Ream wrote: Thanks to all who replied. The pointer to the correct documentation will be most helpful. I just awoke (at 2 a.m.) with the way forward. In true prototype fashion, Leo's new zodb-read and zodb-write commands will simply disallow multiple connections. So at most one 'zodb outline' can be open at any time and a zodb/Leo outline will use a single, long-lasting connection. This scheme is dead simple (and can easily be tested!) and it will give Leo's users at least 90% of what they want, namely the ability to store and retrieve any particular Leo outline in a zodb. Indeed, this prototype is likely to be all that is ever needed. Excellent! I'm glad you feel like you have a way forward. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Newbie questions re persistent objects and connections: [Was: can Leo...]
On Sep 12, 2006, at 8:24 AM, Edward K. Ream wrote: This is a repost of my original: Can Leo use zodb as a file system. Both the title of the thread and the botched formatting may have put people off, so let's try again :-) I know Leo. :-) It's cool. Benji's email is the right reply. I'll try to give you a few pointers to your specific questions. In my reply, I warn you that I'll say things like "I would be worried about this approach" and "hypothetically" and other worrisome things. My intent is to convey that the small amount of time I have to spend on this email just gives me time for gut feelings, and I haven't written an app like Leo or used the ZODB outside of message based systems, so I don't want to say "bad idea" outright. Sorry for the wishy-washy replies. ... I want to know if this scheme can be used if there are multiple Leo outlines open. In my (zodb newbie's) mind this question translates to the following zodb-specific questions: 1. Is it possible associate a particular connection with a particular Leo outline? Only in application logic. I would be worried about this approach. 2. Is it possible to **leave a connection open** for as long as a particular Leo outline is open? Yes. I would be worried about this approach too. 3. If multiple Leo outlines (and multiple connections) are open, is it possible to define transactions that are specific to particular connection? In particular, does something like connection.commit() exist, and if so, how are connection-specific transactions defined? Give each connection its own transaction manager. For instance, see this file: http://svn.zope.org/zc.queue/trunk/src/zc/queue/queue.txt? rev=67933&view=auto starting with "NOTE: "... (the warning is just because the test depends on the ZODB when all it technically needs to depend on is persistence). Doing this can be tricky, and as per usual, I would be worried about this approach. ;-) The reason I ask these particular questions is this: If my prototype opened an outline as above and then *closed* the connection, my prototype threw this exception soon after I started to use the outline: Traceback (most recent call last): File "C:\prog\tigris-cvs\leo\src\leoGlobals.py", line 5010, in getScript elif p == c.currentPosition(): File "C:\prog\tigris-cvs\leo\src\leoNodes.py", line 1370, in __cmp__ if p1.childIndex() != p2.childIndex(): File "C:\prog\tigris-cvs\leo\src\leoNodes.py", line 1580, in childIndex if not v or not v._back: File "C:\Python24\Lib\site-packages\ZODB\Connection.py", line 587, in setstate raise RuntimeError(msg) RuntimeError: Shouldn't load state for 0x60af when the connection is closed Here p is an instance of a Leo position, a short-lived object that *refers* to vnodes and tnodes but is *not* itself persistent. So is it possible for non-persistent objects to refer to persistent objects? Yes, while the connection is open. After that, you get errors, as demonstrated above. Summary 1. It appears that zodb does not update persistent objects properly when the connection that created those objects is closed. I say this because of the traceback given above. Thus, I assume that *long-lasting* connections are required to support persistent objects. Is this assumption correct? No. That's one approach, and is hypothetically reasonable. Another approach would be to have in-memory copies of the objects in the zodb (they could even be the same class, just without a database connection). Saving the file pushes the copy's data to the ZODB and commits a transaction. This would work with a single long running connection or with short connections, opened just for save and load. That would be my first take on ZODB integration with Leo, FWIW. 2. With multiple Leo outlines connected to multiple connections there must be some way to **save one outline without saving them all**. (or to revert one outline without reverting them all.) How do I define outline-specific transactions? Again, in a fly-by opinion, I think saving use case needs in-memory copies. I think the reverting use case needs a mechanism other than ZODB undo. If I had worlds enough and time, I would try to help you use zc.vault, which I think would be ideal for this purpose. As it is, I'll just point you to this rather ovewhelming document and run away. :-( http://svn.zope.org/zc.vault/trunk/src/zc/vault/README.txt? rev=69750&view=auto I hope this helps a bit. The main point of the email is to let you know that people care, and want to help out as possible. ;-) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads and Connections
On Jul 26, 2006, at 12:52 PM, Chris McDonough wrote: FWIW I believe by default at least, open ZODB connections are tied to the thread which did the opening (they are returned to a pool when closed and reused, possibly in another thread). And indeed each connection does have a cache; caching is one of the primary responsibilities of a connection object. See the documentation in ZODB/interfaces.py for "IConnection". I *think* what is happening here is that you are committing the transaction devoted to the current connection/thread, and trying to close a connection that has pending changes from another thread. So when you do transaction.abort(), it's aborting the transaction involving the current thread, not the one associated with the other connection. That said, I'm not entirely sure what to tell you to do here; there used to be an API named "setLocalTransaction" or somesuch that allowed you to control the one-thread-per-connection policy wrt to a transaction minimally, IIRC. This API has disappeared, probably replaced with something more flexible, but I'm not sure what that is. I suspect it may have something to do with the transaction_manager parameter to DB.open() however. Open the database with an explicit transaction_manager if you want to manage these yourself, instead of using the thread default. If you have a connection, get the transaction_manager off of the connection. Can't find it in the interface, unfortunately, but I believe it to be reliable. All that said, Chris S, it sounds like you might be attacking a solved problem--and *might* be adding some serious unnecessary complexity. Maybe you ought to take a step back and see if anyone has a general approach to what you want? For instance, you said The reason why I'm doing this is because I'm trying to update the classes of persistent objects loaded into memory. Want to elaborate on this a bit? It sounds like stuff other folks have done, but you might want to give more detail. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: getting the object ID (_p_oid attribute)
On Jun 26, 2006, at 3:16 PM, Florent Guillaume wrote: On 26 Jun 2006, at 21:11, Robert Gravina wrote: On 2006/06/27, at 3:49, Benji York wrote: Robert Gravina wrote: I just tried loading a persisted object interactively and noticed that although the _p_oid doesn't print out as anything (and hence I always thought it was empty in my debugging prints), it isn't actually None! Can anyone explain this? (here "p" is my persisted object) >>> p._p_oid '\x00\x00\x00\x00\x00\x00\x00\x08' >>> print p._p_oid >>> p._p_oid is None False What would you expect to see if you printed out seven null characters and a backspace? Hahaha - that's a good point! I was expecting IDs to look, well, something like "asdf23asdf". Well, anyway thankyou! I seemed to have solved this problem. I was able to write a __eq__ function like this: def __eq__(self, other): if isinstance(other, ): if hasattr(other,"_p_oid") and other._p_oid != None and (other._p_oid == self._p_oid): return True else: return False and now can compare objects for equality after the (Twisted) client edits them and sends them back. You can shorten that to def __eq__(self, other): return aq_base(self) is aq_base(other) Heh, good point. And you can ditch the aq_base if you don't use acquisition-based classes. (or don't have proxies generally--Zope 3 has a number of them, for instance.) (this wouldn't work if you had the same persistent object from different connections...but don't go there!) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: getting the object ID (_p_oid attribute)
On Jun 26, 2006, at 3:11 PM, Robert Gravina wrote: I seemed to have solved this problem. I was able to write a __eq__ function like this: def __eq__(self, other): if isinstance(other, ): if hasattr(other,"_p_oid") and other._p_oid != None and (other._p_oid == self._p_oid): return True else: return False and now can compare objects for equality after the (Twisted) client edits them and sends them back. To be *really* careful in the face of multi-databases, you'd check database names too. See the zope.app.keyreference.persistent code, for instance. If you don't have multidatabases, then this code looks like it will do the job. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB
On Apr 29, 2006, at 6:08 AM, Christian Theune wrote: Hi everybody, the ZODB team is proud to announce a technical preview of a Blob implementation for ZODB. Hey. This seems like there's no way that it will make it into the next ZODB release, right? I assume no one is using it in production, on the basis of the branch TODO. Does anyone have a hard need to get this production-ready (address the TODOs) by a certain date? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] persistent queue gets a project
I proposed putting a persistent queue in ZODB (actually persistent) and had gotten some positive feedback here. Jim and I talked over some of the pros and cons today and I decided to just put it in a separate package, in a separate product. It's here: http://svn.zope.org/zc.queue/ The current changes between its previous incarnation in my svn.zope.org sandbox and its new official home are minimal and superficial. I didn't change the tests as we had discussed. Maybe later. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] edge case question about _p_deactiveate and modified objects
On Mar 24, 2006, at 9:11 AM, Chris Withers wrote: Hi All, What happens if you call _p_deactivate() on a modified object but before the modifications have been committed? Do the modifications get lost or will they still be saved when the next commit happens after the _p_deactivate()? Under normal operation, the _p_deactivate call is ignored on a modified object, so the changes are not lost (but _p_deactivate accomplished nothing). However, there is currently a bug that if you call _p_deactivate on a new persistent object that has a database connection then the whole thing will just be lost. I have a TODO to make a small test case of this and put it in the collector. AIUI from Jim, fixing this reliably will require some API changes. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Conflict resolving persistent queue?
On Mar 23, 2006, at 3:21 PM, Chris Withers wrote: Hi All, I was wondering whether anyone had implemented a FIFO persistent queue class which has the following conflict resolution strategy: two concurrent adds: adds both new items to the end of the queue in a time-based order one add and one remove happening concurrently: add the new item and remove the removed item two concurrent removes: re-raise the conflict error. http://svn.zope.org/Sandbox/poster/zope.zasync/queue.py? rev=41465&view=auto Slight differences: - if you use the bucket-based one (better for potentially large queues) then it's not reliably time based order; if you use the simpler one then you get what you want. - it only pukes if you remove the same one (since, though it defaults to FIFO, it allows you to remove ones later down the queue if desired). I can't imagine it'd be hard to knock one up in python if not, in which case: - is there any similar python code I can look at for examples? - how would I write unit tests for the above 3 cases? This was the approach I preferred: http://svn.zope.org/Sandbox/poster/zope.zasync/queue.txt? rev=41464&view=auto I want to put this in the ZODB release (though time is running out). When I proposed it, Florent suggested that it go in persistent and that the tests no longer rely on ZODB, but test the conflict resolution code with artificial entries. This makes sense in that context, and it's what I plan/hope to do. That said, the approach I took in the doctest is also useful, since it is more real-world. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Mar 2, 2006, at 10:03 PM, Chris McDonough wrote: persistence-3.6.0.egg (which would include persistent and BTrees modules). zodblib-3.6.0.egg (which would include ZODB and ZODB packages). transaction-3.6.0.egg (which would include only the transaction package). zconfig-2.3.1.egg (which would include only the ZConfig package). zodb-3.6.0.egg (which might be an empty "dependency" package that depended on persistence, transaction, zodblib, and zconfig). Anyone care? First, "rah rah!". Second, zodblib seems unlikely to be useful alone, but maybe I'm wrong, and even if I'm right there's precedent for that sort of thing in packaging land, so what the hey. Third, I'd guess that zodb would contain the stitching for zconfig-to- zodb. It makes me wonder if a zodb-without-zconfig egg would be feasible/useful, but that's probably for later navel-staring. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Possible contributions to ZODB
On Feb 3, 2006, at 9:12 AM, Florent Guillaume wrote: Gary Poster wrote: On Jan 26, 2006, at 10:16 PM, Gary Poster wrote: I have some code that might be worth putting into ZODB. There are three bits. - Two flavors of persistent queue. [...] I'm a bit surprised there was so little discussion: ZC has needed a persistent queue several times in the past. Whether the queue would be based on what I proposed or on something else someone else has, it seems like it would be a good thing to have. However, with only Julien replying I don't think that's enough to push it further. I'll keep it in zasync until/unless some better option comes around. Like Julien I'd like to encourage you to put it in the core. I'd say, in the "persistent" module. OK: I'll push this a bit more then. Thanks to you and Julien for your encouragement, and for Julien's offer of help. If you want unit testing without ZODB dependency, I think you should just synthesize states by hand, and test the conflict resolution method. True, that would work; and since I agree that the queue belongs in persistent, that's probably what I should do. Because in fact in you current unit tests, you're testing much more, including the ZODB mechanisms that detect concurrently modified objects in different connections and regenerate old state from the database, etc. It's not strictly needed (and is a lot of setup). True. I still like what I did, because I think it gives a fuller story for someone reading the doctest (and I don't mind the fact that the test relies on ZODB, despite classic unit test advice), but it can't go in persistent, so I think the point is moot. I'll push this farther along. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Possible contributions to ZODB
On Jan 26, 2006, at 10:16 PM, Gary Poster wrote: I have some code that might be worth putting into ZODB. There are three bits. - Two flavors of persistent queue. [...] I'm a bit surprised there was so little discussion: ZC has needed a persistent queue several times in the past. Whether the queue would be based on what I proposed or on something else someone else has, it seems like it would be a good thing to have. However, with only Julien replying I don't think that's enough to push it further. I'll keep it in zasync until/unless some better option comes around. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Possible contributions to ZODB
I have some code that might be worth putting into ZODB. There are three bits. - Two flavors of persistent queue. Both have a conflict resolution policy to be about as lenient as I could make it. They don't expect to ever hold multiple instances of equivalent values at the same time. One flavor puts all of the contents in a single data structure, while the other divides it up into smaller persistent objects, a la BTree/Buckets. These might go in the persistent package, but tests require ZODB, so I'm not sure. See http://svn.zope.org/Sandbox/poster/zope.zasync/queue.py?view=auto and http://svn.zope.org/Sandbox/poster/zope.zasync/queue.txt?view=auto - Simple, somewhat inefficient approach to pickling methods. I'm not sure where this might go; maybe the persistent package again? See http://svn.zope.org/Sandbox/poster/zope.zasync/methodpickle.py?view=auto and http://svn.zope.org/Sandbox/poster/zope.zasync/methodpickle.txt? view=auto - Hacked version of MappingStorage (barely) appropriate for testing conflict resolution code. It might go in ZODB/tests/utils. I'd need something like it to include the persistent queue tests. See http://svn.zope.org/Sandbox/poster/zope.zasync/tests.py?view=auto Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 12:04 PM, Tim Peters wrote: Note that we have yet to use a new strategy for shrinking pickle sizes: a few years ago Python's pickle code grew support for "extension codes", a registry of class/type names that _can_ be referenced by short (as short as 2 bytes) new pickle codes, instead of embedding the module and class name into every pickle, over and over again. I don't recall the exact numbers numbers, but some years ago Jeremy analyzed a customer Data.fs, and discovered that at least half of it consisted of repetitions of the string "BTrees.OOBTree.OOBTree" ;-) That's the kind of thing the "extension code" pickle mechanism was intended to address; it's a simple and cheap compression gimmick, but so far unused. Yes, I remembered this, and just refreshed my memory. This is the last mention I see in the archives as to ZODB use of protocol 2 (i.e., it doesn't, and prior to Py 2.3.4 it couldn't). http://mail.zope.org/pipermail/zodb-dev/2004-December/008259.html Is that still accurate--that is, does ZODB still not use protocol 2? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 11:38 AM, Tim Peters wrote: ... I know Gary Poster gave a lot of thought to making pickles for the timezone info in Zope3 efficient too. For some definition of "a lot of thought". :-) The pickle for pytz.utc is now relatively small (though still adds a non-trivial percentage addition--30%ish?--to a naive datetime IIRC). That's as far as that bit goes. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On Aug 5, 2005, at 2:33 PM, Tim Peters wrote: [Julien Anguenot] ... The order argument could be an integer with a default value to 0 added to the registration method. [Gary Poster] I usually like the word 'priority' for this sort of value, rather than 'order'. To me 'order' implies unique value--i.e., with 'order' I am mildly surprised to learn that two entries may have the same order. I am not surprised that two entries may share a priority. It's wholly unclear to me, given a callback with associated value 0 and another with associated value 999, which one is supposed to be invoked first. If the parameter is called "priority", then I'd expect 999 to "win"; if it's called "order", then 0. To each his own, I suppose--'first priority' and 'last priority' is my model, in which, at the least, a priority of 1 would be first priority and a priority of 999 is a reasonable last priority (acknowledging a positive infinity as something that would take the honor of being reliably last priority, then). A priority of 0, and a negative priority, start to muddle the model a bit, certainly. But anyway, if you don't see 'priority' in that way, then yes, it is perhaps unclear in a different dimension than 'order' is unclear. :-) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On Aug 5, 2005, at 6:20 AM, Julien Anguenot wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, I would like to order the before commit hooks on a transaction by being able to specify an order for a subscriber at registration time. For the moment, they are called in the order they've been registred. I like this idea, and was just idly wishing for it this morning. The order argument could be an integer with a default value to 0 added to the registration method. I usually like the word 'priority' for this sort of value, rather than 'order'. To me 'order' implies unique value--i.e., with 'order' I am mildly surprised to learn that two entries may have the same order. I am not surprised that two entries may share a priority. If negative values are acceptable then a default value of 0 is OK by me. We could just insert the subscriber wihin the queue according to this order parameter if found *or* sorting the queue according to this parameter before flushing the queue and calling the hooks. I think it would be ideal if before commit hooks with the same priority continue to be in the order in which they were added. There are a number of ways to accomplish this and I expect you can choose a very reasonable one :-) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Reloading product in 2.8
On Jul 12, 2005, at 1:46 PM, Jens Vagelpohl wrote: On 12 Jul 2005, at 18:17, Lennart Regebro wrote: previous Zope version, which I used, was 2.6.1, but it was very old version and I decided to jump to 2.8.0. I took problems with autorefresh. What can I do to solve my problems? If it creates problems then simply don't use it. That's not a very good answer. I disagree. I agree with Jens that it is unlikely to ever work correctly 100% of the time. I agree with Lennart that leaving it not working is not a good answer. I think it should be removed as a feature. Making every developer using ZODB eventually learn not to trust the feature seems unnecessarily painful. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Invalidations
On May 5, 2005, at 11:24 PM, Gary Poster wrote: I can't get Zope 3 to run Sorry, that should have been "I can't get Zope 3 to serve pages"... ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Invalidations
On May 5, 2005, at 8:40 PM, Tim Peters wrote: ...I'd be happier not finishing this exercise <0.5 wink>, so TM.begin() it is. I'm guessing this thread has to do with the recent _synch changes in transaction._manager.py. I can't get Zope 3 to run without the following change to the HEAD (which seems right enough, but this is just on the basis of a quick look). Is this right? Gary Index: transaction/_manager.py === --- transaction/_manager.py (revision 30261) +++ transaction/_manager.py (working copy) @@ -101,6 +101,9 @@ if txn is not None: txn.abort() synchs = self._synchs.get(tid) +if synchs is None: +from ZODB.utils import WeakSet +synchs = self._synchs[tid] = WeakSet() txn = self._txns[tid] = Transaction(synchs, self) _new_transaction(txn, synchs) return txn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
