[ZODB-Dev] ReadConflictErrors with ZODB3.10.1 and Relstorage 1.4.1

2011-01-26 Thread Jürgen Herrmann
i everybody! About a week ago i migrated our FileStorages to Relstorage instances. We have 5 databases, 4 of them mounted via ZodbMountPoints in a plain zope 2.13 installation. Since the migration i experience sporadic ReadConflictErrors (they occur about 1-4 hours after restarting the

Re: [ZODB-Dev] ReadConflictErrors with ZODB3.10.1 and Relstorage 1.4.1

2011-01-26 Thread Shane Hathaway
On 01/26/2011 05:29 AM, Jürgen Herrmann wrote: _readCurrent is obviously reused and not cleared at transaction boundaries, is that expected?): No! Thanks for the great analysis. This insight is key. RelStorage has a monkey patch of the Connection.sync() method, which has not changed

Re: [ZODB-Dev] ReadConflictErrors with ZODB3.10.1 and Relstorage 1.4.1

2011-01-26 Thread Jürgen Herrmann
On Wed, 26 Jan 2011 07:08:14 -0700, Shane Hathaway sh...@hathawaymix.org wrote: On 01/26/2011 05:29 AM, Jürgen Herrmann wrote: _readCurrent is obviously reused and not cleared at transaction boundaries, is that expected?): No! Thanks for the great analysis. This insight is key.

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Stefan H. Holek
On 26.01.2011, at 21:15, Matt Hamilton wrote: I'm not sure how the current MAX_BUCKET_SIZE values were determined, but looks like they have been the same since the dawn of time, and I'm guessing might be due a tune? Go Matt go! :-) -- Stefan H. Holek ste...@epy.co.at

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Shane Hathaway
On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Jim Fulton
On Wed, Jan 26, 2011 at 3:15 PM, Matt Hamilton ma...@netsight.co.uk wrote: All,  I have been doing some performance investigation into a large Plone site we have running. The site in question has approx 300,000 items of content. Each piece of content is indexed by ZCatalog. The main thing I

Re: [ZODB-Dev] ReadConflictErrors with ZODB3.10.1 and Relstorage 1.4.1

2011-01-26 Thread Jürgen Herrmann
On Wed, 26 Jan 2011 07:08:14 -0700, Shane Hathaway sh...@hathawaymix.org wrote: On 01/26/2011 05:29 AM, Jürgen Herrmann wrote: _readCurrent is obviously reused and not cleared at transaction boundaries, is that expected?): No! Thanks for the great analysis. This insight is key.

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Chris Withers
On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Jürgen Herrmann
On Wed, 26 Jan 2011 21:54:52 +, Chris Withers ch...@simplistix.co.uk wrote: On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Chris Withers
On 26/01/2011 21:57, Jürgen Herrmann wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Googling for cross database references zodb should give you all the necessary background. I don't

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Chris Withers
On 26/01/2011 21:54, Chris Withers wrote: On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Shane Hathaway
On 01/26/2011 03:21 PM, Chris Withers wrote: Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? My guess is not. Shane, I went to try your suggestion: If you're sure you have found all of the corruption, you can

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Laurence Rowe
On 26 January 2011 21:57, Jürgen Herrmann juergen.herrm...@xlhost.de wrote:  is there a script or some example code to search for cross db  references?  i'm also eager to find out... for now i disabled my packing cronjobs. Packing with garbage collection disabled (pack-gc = false) should

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Chris Withers
On 26/01/2011 22:49, Laurence Rowe wrote: On 26 January 2011 21:57, Jürgen Herrmannjuergen.herrm...@xlhost.de wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Packing with garbage

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Laurence Rowe
On 26 January 2011 23:11, Chris Withers ch...@simplistix.co.uk wrote: On 26/01/2011 22:49, Laurence Rowe wrote: On 26 January 2011 21:57, Jürgen Herrmannjuergen.herrm...@xlhost.de  wrote:  is there a script or some example code to search for cross db  references?  i'm also eager to find

Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors

2011-01-26 Thread Shane Hathaway
On 01/26/2011 04:07 PM, Chris Withers wrote: On 26/01/2011 22:38, Shane Hathaway wrote: The odd thing is I have seen no reports of pack issues for a long time. Why two at once? I still wonder if there might be no bug at all. I'm afraid my wondering is on the other side of that argument ;-)

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-26 Thread Roché Compaan
On Wed, 2011-01-26 at 20:15 +, Matt Hamilton wrote: So... has anyone tried increasing the size of MAX_BUCKET_SIZE in real life? I tried back in 2007 but didn't see a big difference. https://mail.zope.org/pipermail/zodb-dev/2007-November/011279.html My benchmark was focussed on writes not