Re: [Zope-dev] Session Errors (read conflicts)
On Mon, 2003-03-17 at 20:34, John Eikenberry wrote: John Eikenberry wrote: Toby Dickenson wrote: Read conflicts occur if a change is committed in between the start of a transaction, and the transaction needing to load the object. A workaround to reduce the number of read conflicts is to touch the objects that are likely to change early in the transaction. Thanks for the tip. My sessions test script had a sleep() call right after the transaction commit, thus making it right at the beginning of the next transaction. Moving it to right before the commit (after all the session tests) pretty much eliminated all the read conflict errors. Just get a standard ConflictError occasionally now (not nearly as often). This turned out to be better than I thought. After nearly 3 hours of constant abuse I have yet to see a ReadConflictError and have yet to get either the load() or get() KeyErrors, previously I could force one of these in a 2-3 minutes of abuse. Our live sessions code uses the sessions about half to two-thirds of the way through the transaction. Given what can happen in that first half, there is easily plenty of time for read conflicts. I think I might be able to move our session use to the beginning of the transaction (just storing stuff in the REQUEST object for later usage) and hopefully fix our issue. Interesting, and I'm very glad you got it working reliably. But I'm not sure I understand why touching an object at the start of a transaction would fix the consistency problem. If a object A is read at the beginning of a transaction that uses the data from object A during write of object B and later object A changes and is invalidated by another transaction (before the first finishes), no conflict errors will be raised. Isn't there the potential for the first transaction to write inconsistent data into object B? In my mind, this is the same thing as using a low conflict connection (ie. you are essentially turning off read conflicts, albeit by circumstance rather than by code). I guess the question boils down to: why *don't* you see consistency problems now? I would expect to continue to see the get errors you reported earlier. I think this exact problem is being discussed on ZODB-Dev under the title Database Read Conflicts -- neither safe nor robust. The KeyErrors happen under similar circumstances to the ReadConflictErrors. The significant difference being that the KeyErrors happen after the transience timeout has occured. When I am running with the LowConflictConnection disabled the KeyErrors occur in the Connection.setstate method at line 509, before the ReadConflictError check. Is this the same KeyError you reported as coming out of TemporaryStorage.load? So say you have 2 threads; A and B. If A starts first, hits the session code and triggers a _housekeep() call during which time B has started but has not reached the Sessions code until after _housekeep() has finished. When it does reach the sessions code, you get the KeyErrors. Would you mind restating that? I think this is important, and I'm not sure I understand the second sentence above. Thanks! - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors (read conflicts)
On Tuesday 18 March 2003 3:24 pm, Chris McDonough wrote: Our live sessions code uses the sessions about half to two-thirds of the way through the transaction. Given what can happen in that first half, there is easily plenty of time for read conflicts. I think I might be able to move our session use to the beginning of the transaction (just storing stuff in the REQUEST object for later usage) and hopefully fix our issue. Depending on what your sessioning involves, you might not need to change when you do all the work. Possibly just touching (an attribute of) the persistent session object will be enough. But I'm not sure I understand why touching an object at the start of a transaction would fix the consistency problem. If a object A is read at the beginning of a transaction that uses the data from object A during write of object B and later object A changes and is invalidated by another transaction (before the first finishes), no conflict errors will be raised. Isn't there the potential for the first transaction to write inconsistent data into object B? Today you are guaranteed that your transaction reads a consistent initial state of A and B. That is, there is no chance that you only see half the changes of a recent transaction that modified both. Yes, it is possible for one transaction to modify A and a concurrent transaction to modify B. This is what Deiter describes as not safe in that zodb-dev thread. IMO, it is only not safe if you are not aware of it. And it is critical for performance with concurrency - consider these two transactions occuring on different zeo nodes. In my mind, this is the same thing as using a low conflict connection (ie. you are essentially turning off read conflicts, albeit by circumstance rather than by code). There is a significant difference. The low conflict connection permits the possibility of seeing half the changes of a recent transaction that modified both objects. low conflict connection is a misleadnig name. The conflicts are still there - just not reported as exceptions. low consistency connection would be more accurate ;-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors (read conflicts)
Today you are guaranteed that your transaction reads a consistent initial state of A and B. That is, there is no chance that you only see half the changes of a recent transaction that modified both. Yes, it is possible for one transaction to modify A and a concurrent transaction to modify B. This is what Deiter describes as not safe in that zodb-dev thread. IMO, it is only not safe if you are not aware of it. And it is critical for performance with concurrency - consider these two transactions occuring on different zeo nodes. Right, I understand now, thank you. In my mind, this is the same thing as using a low conflict connection (ie. you are essentially turning off read conflicts, albeit by circumstance rather than by code). There is a significant difference. The low conflict connection permits the possibility of seeing half the changes of a recent transaction that modified both objects. low conflict connection is a misleadnig name. The conflicts are still there - just not reported as exceptions. low consistency connection would be more accurate ;-) Yup. Maybe I should rename it? ;-) So... I'm wondering what the hell I should do. ;-) Obviously I need to deactivate the low consistency connection in the default setup. I'm thinking that I also may just need to move the housekeeping duties to a separate scheduled thread that only happens when the system is not busy (e.g. when the asyncore poll select timeout is reached maybe) in order to reduce the potential for conflicts. But I'm not sure what else to do, and I still don't understand what is causing the KeyErrors in the storage code. Sigh. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors (read conflicts)
On Tuesday 18 March 2003 6:12 pm, Chris McDonough wrote: I'm thinking that I also may just need to move the housekeeping duties to a separate scheduled thread that only happens when the system is not busy (e.g. when the asyncore poll select timeout is reached maybe) in order to reduce the potential for conflicts. Thats hard to define, when zeo is installed. But I'm not sure what else to do, and I still don't understand what is causing the KeyErrors in the storage code. Sigh. Ive never looked at nor used your Sessioning support, but I am interested in things that can cause (POS)KeyErrors in Storages. This might not be a storage bug. There are several known corner cases in zodb that mean semi-legitimate applications can cause KeyErrors. DirectoryStorage tries to guard against them, and will raise a DanglingReferenceError rather than commit a transaction that will cause a POSKeyError when read. Can the problem be reproduced using DirectoryStorage? (without the low consistency connection) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors (read conflicts)
On Tue, 2003-03-18 at 14:01, Toby Dickenson wrote: On Tuesday 18 March 2003 6:12 pm, Chris McDonough wrote: I'm thinking that I also may just need to move the housekeeping duties to a separate scheduled thread that only happens when the system is not busy (e.g. when the asyncore poll select timeout is reached maybe) in order to reduce the potential for conflicts. Thats hard to define, when zeo is installed. It would only make sense for shared session data if the housekeeping thread was enabled on a single ZEO client in a cluster. Of course this scenario has its own configuration and management difficulties. But I'm not sure what else to do, and I still don't understand what is causing the KeyErrors in the storage code. Sigh. Ive never looked at nor used your Sessioning support, but I am interested in things that can cause (POS)KeyErrors in Storages. This might not be a storage bug. There are several known corner cases in zodb that mean semi-legitimate applications can cause KeyErrors. DirectoryStorage tries to guard against them, and will raise a DanglingReferenceError rather than commit a transaction that will cause a POSKeyError when read. Can the problem be reproduced using DirectoryStorage? (without the low consistency connection) That's a fine question. ;-) John, did you see these Key Errors happen under a storage other than TemporaryStorage (e.g. if the session data container is in the main database)? I will check myself as well when possible. Dieter also observed that this may be a behavior specific to mounted storages, so I will need to try that as well. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors (read conflicts)
Chris McDonough wrote: On Mon, 2003-03-17 at 20:34, John Eikenberry wrote: The KeyErrors happen under similar circumstances to the ReadConflictErrors. The significant difference being that the KeyErrors happen after the transience timeout has occured. When I am running with the LowConflictConnection disabled the KeyErrors occur in the Connection.setstate method at line 509, before the ReadConflictError check. Is this the same KeyError you reported as coming out of TemporaryStorage.load? Yes. I'm referring to the traceback I reported in a previous mail: http://mail.zope.org/pipermail/zope-dev/2003-March/019118.html The error occurs on the same line in TemporaryStorage.load(), whether it gets called from ZODB/Connection.py or TemporaryFolder/LowConflictConnection.py. So say you have 2 threads; A and B. If A starts first, hits the session code and triggers a _housekeep() call during which time B has started but has not reached the Sessions code until after _housekeep() has finished. When it does reach the sessions code, you get the KeyErrors. Would you mind restating that? I think this is important, and I'm not sure I understand the second sentence above. I'm working off the idea that the load() KeyErrors went away when the ReadConflictErrors did. So it seemed like they were probably triggered by similar scenarios. The difference being that for the former, the timeout had been reached and the _housekeep() code had been run (and possibly some additional code in _getCurrentBucket). Now given Toby's description of how ReadConflicts occur [1], it seems that instead of a change being committed to the object it is instead deleted (in _housekeep). Thus instead of getting the object and seeing it is marked as invalid, it cannot get the object at all when it expects to be able to... resulting in the KeyError in load(). [1] Read conflicts occur if a change is committed in between the start of a transaction, and the transaction needing to load the object. A workaround to reduce the number of read conflicts is to touch the objects that are likely to change early in the transaction. -- John Eikenberry [EMAIL PROTECTED] __ A society that will trade a little liberty for a little order will deserve neither and lose both. --B. Franklin ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
