Re: [Zope-dev] Session Errors (read conflicts)

2003-03-18 Thread Chris McDonough
On Mon, 2003-03-17 at 20:34, John Eikenberry wrote: John Eikenberry wrote: Toby Dickenson wrote: Read conflicts occur if a change is committed in between the start of a transaction, and the transaction needing to load the object. A workaround to reduce the number of read

Re: [Zope-dev] Session Errors

2003-03-18 Thread Chris McDonough
On Mon, 2003-03-17 at 15:02, Dieter Maurer wrote: You know that we use a Transience implementation with minimal (and not very essential) administrative data which is immune to inconsistencies and can be used across several ZEO clients (although we do not use that). We have good experience

Re: [Zope-dev] Session Errors (read conflicts)

2003-03-18 Thread Toby Dickenson
On Tuesday 18 March 2003 3:24 pm, Chris McDonough wrote: Our live sessions code uses the sessions about half to two-thirds of the way through the transaction. Given what can happen in that first half, there is easily plenty of time for read conflicts. I think I might be able to move our

Re: [Zope-dev] Session Errors (read conflicts)

2003-03-18 Thread Chris McDonough
Today you are guaranteed that your transaction reads a consistent initial state of A and B. That is, there is no chance that you only see half the changes of a recent transaction that modified both. Yes, it is possible for one transaction to modify A and a concurrent transaction to

Re: [Zope-dev] Session Errors (read conflicts)

2003-03-18 Thread Toby Dickenson
On Tuesday 18 March 2003 6:12 pm, Chris McDonough wrote: I'm thinking that I also may just need to move the housekeeping duties to a separate scheduled thread that only happens when the system is not busy (e.g. when the asyncore poll select timeout is reached maybe) in order to reduce the

Re: [Zope-dev] Session Errors (read conflicts)

2003-03-18 Thread Chris McDonough
On Tue, 2003-03-18 at 14:01, Toby Dickenson wrote: On Tuesday 18 March 2003 6:12 pm, Chris McDonough wrote: I'm thinking that I also may just need to move the housekeeping duties to a separate scheduled thread that only happens when the system is not busy (e.g. when the asyncore poll

Re: [Zope-dev] Session Errors (read conflicts)

2003-03-18 Thread John Eikenberry
Chris McDonough wrote: On Mon, 2003-03-17 at 20:34, John Eikenberry wrote: The KeyErrors happen under similar circumstances to the ReadConflictErrors. The significant difference being that the KeyErrors happen after the transience timeout has occured. When I am running with the

Re: [Zope-dev] Session Errors

2003-03-17 Thread Toby Dickenson
On Saturday 15 March 2003 5:40 pm, Chris McDonough wrote: - The second is an unidentified (yet) bug in TemporaryStorage. As a cause of the KeyError exception? Could be I have seen equivalent exceptions with plain FileStorage in applications that do not use sessions, so I suspect there

Re: [Zope-dev] Session Errors

2003-03-17 Thread Dieter Maurer
Hi Chris, Chris McDonough wrote at 2003-3-15 12:40 -0500: ... - The first is that turning off read conflicts allows for the potential for the session data bucket and the session data index to become out of sync. I admit that I knew this already, but haven't figured out a way

Re: [Zope-dev] Session Errors

2003-03-17 Thread John Eikenberry
Toby Dickenson wrote: Read conflicts occur if a change is committed in between the start of a transaction, and the transaction needing to load the object. A workaround to reduce the number of read conflicts is to touch the objects that are likely to change early in the transaction. Thanks

Re: [Zope-dev] Session Errors

2003-03-17 Thread John Eikenberry
John Eikenberry wrote: Toby Dickenson wrote: Read conflicts occur if a change is committed in between the start of a transaction, and the transaction needing to load the object. A workaround to reduce the number of read conflicts is to touch the objects that are likely to change

Re: [Zope-dev] Session Errors

2003-03-15 Thread Chris McDonough
Thanks so much for helping to pin this down. Thanks for the observation that the error occurs more frequently when the timeout is small. This is likely because the timeout value directly impacts the frequency of housekeeping operations, where old buckets are garbage collected and new buckets are

Re: [Zope-dev] Session Errors

2003-03-14 Thread John Eikenberry
Chris McDonough wrote: OK, thanks John. Thank you for helping. I hate to ask this (I should have done this to start with), but would you be willing to use the following patch --against the original file, not your recently patched version-- and try again? I only checked one of the two

Re: [Zope-dev] Session Errors

2003-03-14 Thread Chris McDonough
OK, thanks John. Let's try one more thing... currently the mounted database used to store the session data uses a connection that ignores read conflicts. This is known to be bad because the machinery which deals with keeping the sessioning index data will also ignore read conflicts, which may

Re: [Zope-dev] Session Errors

2003-03-14 Thread John Eikenberry
Sorry for the length of this one... but I'm trying to braindump to give you as much info about the problem as possible. To be sure it doesn't get lost in my below ramblings, there is probably important peice of information I haven't mentioned yet... that is that these errors seem to coincide

Re: [Zope-dev] Session Errors

2003-03-14 Thread John Eikenberry
John Eikenberry wrote: Once you start a second thread ReadConflictErrors start getting raised. Which thread gets the conflict and which one keeps working seems variable (probably just a timing thing). If I start enough of these threads I can cause the error to happen. But only once the

Re: [Zope-dev] Session Errors

2003-03-13 Thread Chris McDonough
John, Which Zope 2.6? Zope 2.6.1? Here's what line 807 of the current Transience.py looks like: v = self._data[b].get(k, notfound) Does yours look like that? What is the value of the __version__ variable at the top of the Transience.py file? On Thu, 2003-03-13 at 07:11, John

Re: [Zope-dev] Session Errors

2003-03-13 Thread John Eikenberry
Sorry, its Zope 2.6.1. Chris McDonough wrote: John, Which Zope 2.6? Zope 2.6.1? Here's what line 807 of the current Transience.py looks like: v = self._data[b].get(k, notfound) Does yours look like that? Yes. What is the value of the __version__ variable at the

Re: [Zope-dev] Session Errors

2003-03-13 Thread Chris McDonough
Hi John, Can you apply the attached diff to your Transience.py file and run with it in place for a couple of days? It will not fix the problem (the symptoms will remain) but it should print some diagnostic information to the Zope event log (the STUPID_LOG_FILE, hopefully you've got that going)

Re: [Zope-dev] Session Errors

2003-03-13 Thread John Eikenberry
Patch applied and the first results are in... so far its a lot of these: 2003-03-13T15:18:07 PROBLEM(100) Transience KeyError raised in get, checking _data BTree -- 2003-03-13T15:18:07 PROBLEM(100) Transience BTree check succeeded Chris McDonough wrote: Hi John, Can you apply the

Re: [Zope-dev] Session Errors

2003-03-13 Thread Chris McDonough
OK, thanks John. I hate to ask this (I should have done this to start with), but would you be willing to use the following patch --against the original file, not your recently patched version-- and try again? I only checked one of the two BTrees that might be at the heart of the problem with the