Re: [ZODB-Dev] ZODB 3.8.3b1
Jim Fulton wrote: > It would be great if people who have databases with version data would > try the utility out. Where's tfm for the utility? :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.8.3b1
Jim Fulton wrote: > On Sat, Aug 15, 2009 at 8:04 AM, Chris Withers wrote: >> Jim Fulton wrote: >>> It would be great if people who have databases with version data would >>> try the utility out. >> Where's tfm for the utility? :-) > > Use the -h option. There really isn't much to it. (There's also a > doctest in the ZODB/scripts directory.) -h assumes I have a script to run, which is what I was hoping the manual would lead me too... I have a buildout with version 2.12.0a3 of zope2 specified as an egg, extending http://svn.zope.org/*checkout*/Zope/tags/2.12.0a3/versions.cfg?rev=99292. I guess I need to specify ZODB3=3.8.3b1 in my [versions] section, but then what do I do? (hmmm, zope2 2.12.0a3 relies on ZODB3 3.9.0a12, is it safe to downgrade this for the duration of running this script? will this script not work with ZODB3 3.9.x?) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.8.3b1
Jim Fulton wrote: > If you install ZODB3 3.8.3b1 normally, you'll get a strip_versions > script in your bin directory. I'm not familar with the Zope 2 buildout > or whether you can use ZODB 3.8 with Zope 2.12. Hmm, okay, I'll play it safe then... If I have a normal source install of Zope 2.9.10, how do I install ZODB 3.8.3b1 such that the zope instance will actually use it? Or do I just need to install it into site-packages of an appropriate python version so I can run the script on the data.fs file? (sorry, having to guess here since I don't know how the script works - is it over a zeo connection? does it modify the .fs in place? does in generate a new .fs from an old .fs? Is it safe to use on a .fs in use by a storage server?) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.8.3b1
Jim Fulton wrote: > $ bin/strip_versions -h > usage: strip_versions input output > > Remove version records by copying input to output, stripping any > version records seen in input. > > The input and output arguments are file-storage file names. If the -c > option is used, then the input and output arguments must be storage > configuration files. So is this safe to use on a Data.fs that's in use by a storage server? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.8.3b1
Jim Fulton wrote: > Yup, where I assume you mean using the in-use storage as input. The > script makes a new copy. After making the copy, you can change the > server to use that. Of course, changes made after copying won't be in > the copy, but if you just want to try the script out, you can just > point it at an in-use file. I did. Initial testing suggests this works just fine :-) I used the following buildout to get the right version of ZODB in isolation and so didn't have to worry about affecting either the old or new Zope instances involved: [buildout] parts = fixer versions = versions [versions] ZODB3 = 3.8.3b1 [fixer] recipe = zc.recipe.egg eggs = ZODB3 cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] instrumenting a ZEO server?
Hi All,
I'm wondering if there's any way to get a ZEO server to log about how
it's performing?
(eg: how many connections, how many clients are waiting, where they're
waiting, etc)
I'm trying to debug a problem where app servers are being dropped out of
a load balancer pool for being unresponsive, and I think they're waiting
on the ZEO server so I'm trying to get some stats on the ZEO server side
as to what's going on...
As far as the clients go, DeadlockDebugger is providing some insights:
File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 749, in loadEx
self._lock.acquire()# for atomic processing of invalidations
I assume this is the client waiting for the storage server's lock?
Is this the ZEO server's single threadedness or is this a lock between
threads on the app server?
File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 769, in loadEx
data, tid, ver = self._server.loadEx(oid, version)
File "/opt/Zope-2.9/lib/python/ZEO/ServerStub.py", line 192, in loadEx
return self.rpc.call("loadEx", oid, version)
File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 531, in call
r_flags, r_args = self.wait(msgid)
File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 638, in wait
asyncore.poll(delay, self._singleton)
File "/usr/local/lib/python2.4/asyncore.py", line 122, in poll
r, w, e = select.select(r, w, e, timeout)
What does this mean? I'm guessing it's the actual transfer of data from
the storage server to the app server, right?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] instrumenting a ZEO server?
Alan Runyan wrote: > What version of ZODB are you using? Whatever ships with Zope 2.9.8 > If you have zeo on in debugging What does this mean? > How many clients are waiting: > You should be able to see this in zeo log files Not currently... maybe this relates to your mystical "zeo on in debugging" comment? ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.0 released
Jim Fulton wrote: > The changes from ZODB 3.8: > > http://pypi.python.org/pypi/ZODB3/3.9.0#id1 There are some typos in this list: - should probably be `with` when describing the with statement - "If control exists the block" I'm guessing should be "If control exits the block" - effecient -> efficient Questions: - when using `with` blocks and a ConflictError is raised, what happens? In my fantasy world, the `with` block would be tried again ;-) - if using cache-size-bytes, is the parameter that limits the number of objects by number ignored? It would be ideal to specify a size in bytes and an unlimited number of objects... - where can I find out more about invalidation-age? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.0 released
Jim Fulton wrote: >> There are some typos in this list: >> >> - should probably be `with` when describing the with statement > > I don't know what you mean. Databases have a new method, transaction, that can be used with the Python (2.5 and later) with statement: -> Databases have a new method, transaction, that can be used with the Python (2.5 and later) `with` statement: >> Questions: >> >> - when using `with` blocks and a ConflictError is raised, what happens? > > The transaction is aborted. That might be surprising for old-time Zope users... Still, I assume the ConflictError propagates through so they'll see it raised, right? >> - if using cache-size-bytes, is the parameter that limits the number of >> objects by number ignored? > > No. Is there any way to get this to happen? Would seeing the number limit extremely high have the right effect? >> - where can I find out more about invalidation-age? http://svn.zope.org/ZODB/trunk/src/CHANGES.txt?rev=103676&r1=103674&r2=103676 There's a "that" missing from the first of the new lines ;-) > src/ZEO/tests/invalidation-age.txt Ah cool, now I get it :-) typo: verificatioin -> verification I take it this is irrelevant if you're not using persistent client caches? Are those safe to use now? [1] cheers, Chris [1] http://www.mail-archive.com/zodb-dev@zope.org/msg03349.html http://www.mail-archive.com/zodb-dev@zope.org/msg01377.html -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] CRC check failed when running repozo
Hi All, Anyone ever see this? Traceback (most recent call last): File "/opt/Zope-2.9//bin/repozo.py", line 517, in ? main() File "/opt/Zope-2.9//bin/repozo.py", line 510, in main do_backup(options) File "/opt/Zope-2.9//bin/repozo.py", line 450, in do_backup reposz, reposum = concat(repofiles) File "/opt/Zope-2.9//bin/repozo.py", line 263, in concat bytesread += dofile(func, ifp) File "/opt/Zope-2.9//bin/repozo.py", line 200, in dofile data = fp.read(todo) File "/usr/local/lib/python2.4/gzip.py", line 225, in read self._read(readsize) File "/usr/local/lib/python2.4/gzip.py", line 273, in _read self._read_eof() File "/usr/local/lib/python2.4/gzip.py", line 309, in _read_eof raise IOError, "CRC check failed" IOError: CRC check failed Zope 2.9.8... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] How do I find out the size of a subtree of the object graph?
Hi All, As the subject line says with the following definitions: - size = size on disk, in Mb - subtee = Zope folderish type things, in this case... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] CRC check failed when running repozo
Adam GROSZER wrote: > Hello, > > +1 on that. Sometimes copy operations (over the network) screw up some > bits. Happens rarely but then it hits hard. > > Monday, September 14, 2009, 10:13:39 AM, you wrote: > > CT> Hmm. Haven't seen that. My guess would be some low-level data corruption. > > CT> Christian My suspicion is running repozo where the previous repozo run failed and the output directory is corrupted. I'm gonna try blowing away all the previous repozo output and try again... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How do I find out the size of a subtree of the object graph?
Jim Fulton wrote: > ZODB doesn't provide any direct support because it has no notion of tree. Does it have any notion of object size on disk for a given object? > In Zope 2, you could export the folder and look at the size of the export. Sadly, when you're trying to figure out which folder is using all the space on a disk that's running out of space, that's not practical... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How do I find out the size of a subtree of the object graph?
Jens Vagelpohl wrote: > >> Sadly, when you're trying to figure out which folder is using all the >> space on a disk that's running out of space, that's not practical... > > How about simply writing a script that crawls the ZODB and gives you > paths and sizes? Indeed, and what's the magic incantation to get the size of the object as it is on disk? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Problems packing Zope 2.9.8 storage
Hi All, I've got a ZODB that's ballooned and I'm now having problems packing with zeopack. I tried with zeopack, which appeared to just hang (still in top, but no cpu usage for an hour or two), so I ctrl-c'ed it. Now when I try, I just get the text "Already packing" back. There is a .old file there, but it was last modified over a week ago. Looking in FileStorage.py, it appears there's a possibility for an exception to be raised after self._pack_is_in_progress has been set to True but before the try/finally clause that clears self._pack_is_in_progress has been entered. Has anyone experienced this? If so, what should I look to do now? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Shane Hathaway wrote: >> There is a .old file there, but it was last modified over a week ago. > > The analysis phase of packing can take a long time. A week is too much, > though. :-) Yeah, this is off the back of an automated pack/backup/unpack/verify to keep two storages in sync on two database servers, one live, one hot standby. The process went wrong because one of the storages has ballooned in size... >> If so, what should I look to do now? > > If I were solving it, I would see what happens if I copy the storage to > RelStorage and then pack it. Maybe that's just me though. I thought RelStorage was slow at packing? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Shane Hathaway wrote: > Chris Withers wrote: >> I thought RelStorage was slow at packing? > > Not anymore. I optimized packing last December/January and it has been > plenty fast ever since, AFAIK. Interesting. Got any metrics on its speed compared to FileStorage, including packing? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
Shane Hathaway wrote: > Look at this chart in particular, which suggests that RelStorage with > PostgreSQL or MySQL is several times faster than ZEO for heavy read loads: Cool, now all I need is something like zeoraid for postgres (or, at a push, mysql)... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] tools for analysing the tail of a filestorage?
Hi All, I have a filestorage being used by Zope 2 that is mysteriously growing. I don't have confidence in the Undo tab, since this setup has two storages, once mounted into the other. I tried fstail.py, and while it tells me the same info as the Undo tab (except with more certainty that it's showing the right storage results ;-) it doesn't say much about the objects in question... Are there any other tools that might tell me more? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Cleaning up a ZODB storage from 'versions'
Jim Fulton wrote: >> Is there a way for cleaning up a storage and removing the related >> versions in some way? > > Yes. Use the ZODB 3.8 strip_versions script. I used this buildout to get this script available in isolation: [buildout] parts = fixer versions = versions [versions] ZODB3 = 3.8.3b1 [fixer] recipe = zc.recipe.egg eggs = ZODB3 ...it doesn't need an instance home or any such, as it just copies a source .fs file (which can be in use) to a new .fs fiel but minus the versions. cheers, Chris PS: Andreas, I could have taken your usual approach and just shouted at you to use Google since Jim already helped me through this about a month ago ;-) -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] "tid to old for invq" in zeo server logs?
Hi All, I saw a few entries like these in the event log of a storage server that stopped responding for a couple of minutes: INFO ZEO.StorageServer (30332) tid to old for invq 252618838173181593 < 252618856850589832 -- INFO ZEO.StorageServer (30332) tid to old for invq 252618838177204172 < 252618856850589832 What do these mean? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] "tid to old for invq" in zeo server logs?
Jim Fulton wrote: > It means that the server wasn't able to provide "quick" invalidations > because a client was disconnected too long. The message isn't very > well worded, even aside from the misspelling. Has that been fixed in the latest zodb? If not, if I get a chance, I'm happy to patch... > Normally, in this situation, a client would then perform a full cache > verification, which puts lots of pressure on the server, possibly > making it unresponsive. There's a client option > "drop-cache-rather-than-verify" that can prevent this. Was the present in Zope 2.9.8? > Arguably, this > is a "don't be stupid" option that should be enabled by default. Happy to patch this too, if it's the right thing to do and I get a chance and I'm told it's okay to do so... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 released
Jim Fulton wrote: > situations when deleting items. I recommend that people running 3.9 > update to the new release as soon as reasonable. (If you're using ZEO, > this only applies to ZEO clients, but servers. Just wanted to check this last sentence: what needs updating? clients, servers or both? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] why use zc.recipe.filestorage?
Hi All,
I'm wondering what the point of this recipe is?
Why use:
[storage]
recipe = zc.recipe.filestorage
[zeo]
recipe = zc.zodbrecipes:server
zeo.conf =
path ${storage:path}
...when you could just use:
[zeo]
recipe = zc.zodbrecipes:server
zeo.conf =
path ${buildout:directory}/databases/Data.fs
I feel like I'm missing something...
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] why use zc.recipe.filestorage?
Jim Fulton wrote:
>> [zeo]
>> recipe = zc.zodbrecipes:server
>> zeo.conf =
>>
>> path ${buildout:directory}/databases/Data.fs
>>
>>
>> I feel like I'm missing something...
>
> You aren't missing anything. The benefit is very small.
Excellent. I shall not feel guilty about not using it then :-)
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zodb 3.8.3 strip_versions missing some versions?
Hi All, I'm upgrading a Zope 2.9.8 project to Zope 2.12 + zeoraid. As I've done with previous projects, I run the .fs files through zodb 3.8.3's strip_versions script. For both the storages in this project, the script found no version records. As a first step, I'm currently running the 2.9.8 app server against a pair of zeo storage servers running zodb 3.9.3 and zeoraid 1.0b3 However, when I try and view the history tab of *any* page template within the .fs files, even a newly created one, I get an assertion error from zeoraid as gocept.zeoraid.storage.RAIDStorage's history method is getting called with version passed as an integer 20. Anyone have any idea what that 20 means and why it's being passed like that? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Christian Theune wrote: > There's a size argument following the version which probably is not > being passed as a keyword argument so it looks like a slight API > incompatibility. I wonder whether ZEORaid should hack around that or > whether the caller should be changed. Hmm... looking more closely at the traceback, the caller is: File "ZODB3-3.9.3-py2.6-linux-i686.egg/ZEO/StorageServer.py", line 1382, in history return self.storage.history(oid, size) Now, ZODB 3.9 doesn't support versions, which is why the versions parameter has vanished, I guess? However, I wonder if maybe this call should be: self.storage.history(oid, size=size) ...to support storages which still accept a version parameter *and* newer ones that don't? Looking at RelStorage: http://svn.zope.org/relstorage/trunk/relstorage/storage.py?rev=105120&view=auto ...I can see it's going to have the same problem. Would anyone object if I wrote a test and fixed this on the 3.9 branch of ZODB? I'm not really sure what to do about the trunk... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Christian Theune wrote: >> Would anyone object if I wrote a test and fixed this on the 3.9 branch >> of ZODB? I'm not really sure what to do about the trunk... > > I think that's the best idea. It's not exactly part of the policy of > supporting old protocols but if we want to support and encourage 3rd > party storage implementations we should leverage this simple fix for > better backwards compatibility. Hmmm, I'm less sure now. The problematic code is here: http://zope3.pov.lt/trac/browser/ZODB/branches/3.9/src/ZEO/StorageServer.py#L1379 It looks like ZEOStorage308Adapter is used when the protocol specified is less than Z309, which it is in my case. But, the assumption here is that the storage being served by StorageServer effectively matches the Z309 API. zeoraid's RAIDStorage does not... However, I can't see what `history` method is called if the protocol >= Z309. Christian, I'm guessing you guys have used zeoraid with 3.9 from some time now, does this issue not occur if everything is talking Z309? If not, why not? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Test time on 3.9 branch
Hi All, Should I really expect the tests to take an hour to run on Windows? Total: 3825 tests, 0 failures, 0 errors in 56 minutes 49.469 seconds. As you can see, no errors or failures were reported. However, there were some error-ish messages interspersed in the test output. Is this to be expected? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Test time on 3.9 branch
Jim Fulton wrote: >> Should I really expect the tests to take an hour to run on Windows? > > Yes. Erk. Is there a subset I can look to run when looking to patch the ZEOStorage308Adapter? >> However, there were >> some error-ish messages interspersed in the test output. Is this to be >> expected? > > You didn't show the extra output, so I can't say. I wouldn't be surprised. Here's the full output... cheers, Chris Running .!no tests here! tests: Set up .!no tests here! in 0.016 seconds. Ran 1 tests with 0 failures and 0 errors in 0.000 seconds. Running .BlobAdaptedFileStorageBlobTests tests: Tear down .!no tests here! in 0.000 seconds. Set up .BlobAdaptedFileStorageBlobTests in 0.000 seconds. Ran 87 tests with 0 failures and 0 errors in 18.062 seconds. Running .BlobAdaptedFileStorageTests tests: Tear down .BlobAdaptedFileStorageBlobTests in 0.000 seconds. Set up .BlobAdaptedFileStorageTests in 0.000 seconds. error: uncaptured python exception, closing channel <__main__.ZEOTestServer localhost:21997 at 0x11205a8> (:[Error 32] The process cannot access the file because it is being used by another process: 'Data.fs' [C:\P ython26\lib\asyncore.py|read|74] [C:\Python26\lib\asyncore.py|handle_read_event|408] [c:\localsvn\zodb\src\ZEO\tests\zeoser ver.py|handle_accept|97] [c:\localsvn\zodb\src\ZEO\tests\zeoserver.py|cleanup|34] [c:\localsvn\zodb\src\ZODB\FileStorage\Fi leStorage.py|cleanup|1314]) warning: unhandled close event error: uncaptured python exception, closing channel <__main__.ZEOTestServer localhost:27769 at 0x11205a8> (:[Error 32] The process cannot access the file because it is being used by another process: 'Data.fs' [C:\P ython26\lib\asyncore.py|read|74] [C:\Python26\lib\asyncore.py|handle_read_event|408] [c:\localsvn\zodb\src\ZEO\tests\zeoser ver.py|handle_accept|97] [c:\localsvn\zodb\src\ZEO\tests\zeoserver.py|cleanup|34] [c:\localsvn\zodb\src\ZODB\FileStorage\Fi leStorage.py|cleanup|1314]) warning: unhandled close event Ran 107 tests with 0 failures and 0 errors in 9 minutes 29.578 seconds. Running .BlobFileStorageBlobTests tests: Tear down .BlobAdaptedFileStorageTests in 0.000 seconds. Set up .BlobFileStorageBlobTests in 0.000 seconds. Ran 87 tests with 0 failures and 0 errors in 16.563 seconds. Running .BlobWritableCacheTests tests: Tear down .BlobFileStorageBlobTests in 0.000 seconds. Set up .BlobWritableCacheTests in 0.000 seconds. error: uncaptured python exception, closing channel <__main__.ZEOTestServer localhost:23335 at 0x11225a8> (:[Error 32] The process cannot access the file because it is being used by another process: 'Data.fs' [C:\P ython26\lib\asyncore.py|read|74] [C:\Python26\lib\asyncore.py|handle_read_event|408] [c:\localsvn\zodb\src\ZEO\tests\zeoser ver.py|handle_accept|97] [c:\localsvn\zodb\src\ZEO\tests\zeoserver.py|cleanup|34] [c:\localsvn\zodb\src\ZODB\FileStorage\Fi leStorage.py|cleanup|1314]) warning: unhandled close event error: uncaptured python exception, closing channel <__main__.ZEOTestServer localhost:24274 at 0x11225a8> (:[Error 32] The process cannot access the file because it is being used by another process: 'Data.fs' [C:\P ython26\lib\asyncore.py|read|74] [C:\Python26\lib\asyncore.py|handle_read_event|408] [c:\localsvn\zodb\src\ZEO\tests\zeoser ver.py|handle_accept|97] [c:\localsvn\zodb\src\ZEO\tests\zeoserver.py|cleanup|34] [c:\localsvn\zodb\src\ZODB\FileStorage\Fi leStorage.py|cleanup|1314]) warning: unhandled close event Ran 106 tests with 0 failures and 0 errors in 9 minutes 12.500 seconds. Running .ClientStorageNonSharedBlobsBlobTests tests: Tear down .BlobWritableCacheTests in 0.015 seconds. Set up .ClientStorageNonSharedBlobsBlobTests in 0.016 seconds. Ran 73 tests with 0 failures and 0 errors in 1 minutes 20.750 seconds. Running .ClientStorageSharedBlobsBlobTests tests: Tear down .ClientStorageNonSharedBlobsBlobTests in 0.031 seconds. Set up .ClientStorageSharedBlobsBlobTests in 0.000 seconds. Ran 73 tests with 0 failures and 0 errors in 1 minutes 23.734 seconds. Running .DemoStorageTests tests: Tear down .ClientStorageSharedBlobsBlobTests in 0.015 seconds. Set up .DemoStorageTests in 0.000 seconds. Ran 50 tests with 0 failures and 0 errors in 4 minutes 16.000 seconds. Running .FileStorageTests tests: Tear down .DemoStorageTests in 0.000 seconds. Set up .FileStorageTests in 0.000 seconds. error: uncaptured python exception, closing channel <__main__.ZEOTestServer localhost:25849 at 0x110bc88> (:[Error 32] The process cannot access the file because it is being used by another process: 'Data.fs' [C:\P ython26\lib\asyncore.py|read|74] [C:\Python26\lib\asyncore.py|handle_read_event|408] [c:\localsvn\zodb\src\ZEO\tests\zeoser ver.py|handle_accept|97] [c:\localsvn\zodb\src\ZEO\tests\zeoserver.py|cleanup|34] [c:\localsvn\zodb\src\ZODB\FileStorage\Fi leStorage.py|cleanup|1314]) warning: unhandled close event error: uncaptured python exception, closi
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Jim Fulton wrote: > I'm fine with change the storage server to pass the history size > argument as a keyword parameter. Okay, but where is the Z309 equivalent of ZEOStorage308Adapter's history method? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Test time on 3.9 branch - windows?
Jim Fulton wrote: Should I really expect the tests to take an hour to run on Windows? >>> Yes. >> Erk. Is there a subset I can look to run when looking to patch the >> ZEOStorage308Adapter? > > Feel my windows pain. Do the tests take much less time to run on Linux? If so, any idea as to why? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ValueError: corrupted record when calling GetObject
Izak Burger wrote: > A restart "fixes" the problem. Its running version 3.7.1 of ZEO/ZODB. I have a feeling some problems in this area were fixed recently. Can you try ZODB 3.8 or 3.9? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zeoraid backends dropping out
Hi All,
Since this is the first post to this list on the issue, the setup I have is:
zeoraid1 and zeoraid2 are zeoraid servers
zeo1 and zeo2 are normal storage servers serving two file storages:
packed and unpacked.
zeoraid1 and zeo1 are on one box, zeoraid2 and zeo2 are on the other
both. A variety of zeo clients connect to the zeoraid servers, and each
client has the ip addresses of both zeoraid servers in its clientstorage
config, so they round robin in the event of a zeoraid server failure.
I'm using zodb 3.9.3 and zeoraid 1.0b3 for the servers, the clients are
all currently Zope 2.9.8.
So, diagrammatically, the configuration is roughly:
zeoraid1 -> zeo1 zeo2
clients ->
zeoraid2 -> zeo1 zeo2
Now, last week, the packed storage on zeo1 started showing as failed in
the zeoraids. After that, a load of data was committed to the storage,
which all ended up in zeo2. Recovering the 10GB or so of data took a
couple of days.
I just went to check that the recovery was finished, only to find the
unpacked storage on zeo2 is now showing as failed in zeoraid2 (which is
the zeoraid currently in use by the clients) and when I try to do
"bin/zeoraid -S packed details" against zeoraid1, I get:
(20659) CW: error in notifyConnected (('127.0.0.1', 6001))
Traceback (most recent call last):
File "ZEO/zrpc/client.py", line 476, in notify_client
self.client.notifyConnected(self.conn)
File "ZEO/ClientStorage.py", line 621, in notifyConnected
self.verify_cache(stub)
File "ZEO/ClientStorage.py", line 1308, in verify_cache
ltid = server.lastTransaction()
File "ZEO/ServerStub.py", line 89, in lastTransaction
return self.rpc.call('lastTransaction')
File "ZEO/zrpc/connection.py", line 703, in call
raise inst # error raised by server
RAIDClosedError: Storage has been closed.
This feels exactly what happened with the packed storage on zeo1 before.
I've looked in the zeoraid and zeo event logs on both servers and didn't
see anything logged as an error. The only weird (and worrying!) thing I
did see was in the zeo2 event log:
--
2009-11-11T02:52:19 WARNING ZODB.FileStorage Packed.fs truncated,
possibly due to damaged records at 14487840095
...but the packed storage shows as optimal in both zeoraid1 and
zeoraid2, so what does this mean?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] help with pinning zeo plone zodb3 to 3.8.1
rafael wrote: > I am experiencing the following problem: i need to pin zodb3 = 3.8.1 > so I can pack (cross reference issue). I use the following mini buildout to get 3.8.1 on its own. In my case it was to run the strip_versions script, but if you run it, you'll see you have a bin/runzeo and bin/zeopack that you can use to get what you want. [buildout] parts = fixer versions = versions [versions] ZODB3 = 3.8.3 [fixer] recipe = zc.recipe.egg eggs = ZODB3 cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] repozo uses md5 lib, which is deprecated in python 2.6
Hi All, ZODB's repozo script uses the md5 library, which is deprecated in python 2.6, resulting in annoying emails from cron jobs. A few questions: - Where's the tracker for ZODB nowadays to report this issue? - Anyone know what to replace md5 with? The warning suggests hashlib, which landed in 2.5. I assume it's okay to just use hashlib rather than worry about conditionally import md5 or hashlib, given that trunk ZODB and 3.9 branch are only targeted at Python 2.5 and above, right? - Does repozo have any tests? If so, how do I run them. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo uses md5 lib, which is deprecated in python 2.6
Christian Theune wrote: >> - Anyone know what to replace md5 with? The warning suggests hashlib, >> which landed in 2.5. I assume it's okay to just use hashlib rather than >> worry about conditionally import md5 or hashlib, given that trunk ZODB >> and 3.9 branch are only targeted at Python 2.5 and above, right? > > Not sure about that. Most people I see that need to support 2.4 and 2.6 > tend to provide a wrapper-function combined with a conditional import. My understanding was that ZODB 3.9 (and therefore trunk too) only targeted 2.5+, perhaps Jim can confirm? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo uses md5 lib, which is deprecated in python 2.6
Jim Fulton wrote: > On Wed, Nov 18, 2009 at 12:00 PM, Chris Withers > wrote: >> - Anyone know what to replace md5 with? The warning suggests hashlib, > > Yes > >> which landed in 2.5. I assume it's okay to just use hashlib rather than >> worry about conditionally import md5 or hashlib, given that trunk ZODB >> and 3.9 branch are only targeted at Python 2.5 and above, right? > > No, ZODB works with Python 2.4 too. > > It's easy to handle this with a test on sys.version_info. OK, this is all done on trunk and 3.9 branch, I ran the test script with Python 2.4 and 2.6, no errors. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Chris Withers wrote:
> Jim Fulton wrote:
>> I'm fine with change the storage server to pass the history size
>> argument as a keyword parameter.
>
> Okay, but where is the Z309 equivalent of ZEOStorage308Adapter's history
> method?
Ug, the answer is much less pleasant :-(
So, a ZODB 3.9.3 ClientStorage calls the ServerStub's history method,
this does an rpc call:
self.rpc.call('history', oid, length)
Which comes in on the server and gets despatched straight out in:
/ZEO/zrpc/connection.py", line 581, in handle_request
This, of course, means we can't pull the keyword parameters trick, since
ZEO's rpc doesn't support keyword parameters from what I saw...
I'm not sure what to suggest here...
- release a new version of zeoraid targeted at zodb 3.9+ that doesn't
include any version parameters anywhere. (I dunno how Christian would
feel about the maintenance burden)
- drop support for zodb < 3.9 in zeoraid and remove all version
parameters. That'd be fine by me, dunno how Christian would feel.
- use some kind of StorageInterface to indicate what parameters a
storage expects. (this would require work on both zeoraid and zeoraid :-/)
- make zrpc.connection.Connection.handle_request do some fancier work
around 581 to check what parameters the method is expecting. I worry
that might impact performance though...
Help?
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo uses md5 lib, which is deprecated in python 2.6
Jim Fulton wrote: >>> It's easy to handle this with a test on sys.version_info. >> OK, this is all done on trunk and 3.9 branch, I ran the test script with >> Python 2.4 and 2.6, no errors. > > Cool, but can we please get an automated test? I don't run the manual > tests. I never will. :) I'm afraid that's a bit beyond my capabilities... Also feels like a bit of an ask for fixing a deprecation warning bug ;-) This would probably be something to do next time the .fs format changes in such a way that repozo needs updating... ...unless, of course, there's another tool for backing up FileStorages now that I should be using instead? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Jim Fulton wrote:
> What's wrong with the original idea of having the server pass the size
> as a keyword parameter? The ZEO protocol is irrelevant.
Okay, from the top:
>> So, a ZODB 3.9.3 ClientStorage calls the ServerStub's history method, this
>> does an rpc call:
>>
>> self.rpc.call('history', oid, length)
>>
>> Which comes in on the server and gets despatched straight out in:
>>
>> /ZEO/zrpc/connection.py", line 581, in handle_request
...which ends up calling .history(oid,length) on zeoraid on RAIDStorage
which is implemented as follows:
def history(self, oid, version='', size=1):
"""Return a sequence of history information dictionaries."""
assert version is ''
return self._apply_single_storage('history', (oid, size))[0]
...because it's designed to work with ZODB 3.8 and 3.9, but doesn't
support versions.
We can't use the trick of making the Z308-->Z309 adapter pass the size
as a keyword parameter, because the call in this case is actually in
ZODB 3.9's ClientStorage, so the adapter is never used, because all
components involved are speaking Z309.
We also can't change the ClientStorage call to pass the size as a
keyword parameter because ZEO's rpc doesn't appear to support keyword
parameters.
That leaves the ideas I posted in my last reply...
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo uses md5 lib, which is deprecated in python 2.6
Jim Fulton wrote: >> Also feels like a bit of an ask for fixing a deprecation warning bug ;-) > > I don't follow that. Isn't that what you did? No, I ran an existing test script with two versions of python and verified that it worked as expected in both cases, with no deprecation warning being emitted. >> This would probably be something to do next time the .fs format changes in >> such a way that repozo needs updating... > > That's not what happened this time is it? No. md5 became deprecated, the FileStorage format hasn't changed between Python 2.4 and 2.6. As far as I'm aware, python versions and .fs formats are orthogonal ;-) > AFAICT, repozo doesn't have automated tests, or this would have been > caught. This means that repozo is susceptible to breakage at any time. > I didn't add repozo to ZODB, but without tests, I'd be happy to take > it out. ;) Then how do you backup filestorages? What's the official, supported way of backing up FileStorages? I'd be more than happy to switch to using that instead... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Jim Fulton wrote: > So why not add a history method to ZEOStorage that passes size as a > keyword argument? I don't know what "ZEOStorage" is... zrpc.connection.Connection appears to despatch direct to the storage, please correct me if I'm wrong. In this case, that's a RAIDStorage. It doesn't have any choice in what parameters its history method gets called with, that's controlled by the ClientStorage emitting the zrpc call... So I'm afraid I don't really understand what you're trying to suggest... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
Jim Fulton wrote: > There's nothing official or supported about a backup solution without > automated tests. > > So I guess there isn't one. Right, so what does Zope Corp use? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Jim Fulton wrote: > On Thu, Nov 19, 2009 at 6:07 PM, Chris Withers wrote: >> Jim Fulton wrote: >>> So why not add a history method to ZEOStorage that passes size as a >>> keyword argument? >> I don't know what "ZEOStorage" is... > > Then we shouldn't be having this conversation. I'm working off this traceback: Traceback (most recent call last): File "ZODB3-3.9.3-py2.6-linux-i686.egg/ZEO/zrpc/connection.py", line 581, in handle_request ret = meth(*args) File "gocept.zeoraid-1.0b6-py2.6.egg/gocept/zeoraid/storage.py", line 219, in history assert version is '' There is no ZEOStorage instance involved in this as far as I can tell. What am I missing? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Christian Theune wrote: >> Traceback (most recent call last): >> File "ZODB3-3.9.3-py2.6-linux-i686.egg/ZEO/zrpc/connection.py", line >> 581, in handle_request >> ret = meth(*args) >> File "gocept.zeoraid-1.0b6-py2.6.egg/gocept/zeoraid/storage.py", line >> 219, in history >> assert version is '' >> >> There is no ZEOStorage instance involved in this as far as I can tell. >> What am I missing? > > Check the "setup_delegation" method: the history method is patched > through directly so your traceback doesn't see the ZEOStorage anymore. OK, I've committed tests and patches to deal with this issue and the similar issue in ZEOStorage308Adapter on the 3.9 branch. Should I merge these changes to the trunk too? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] 3.9.4 release?
Hi Jim, It would be great if the two recent fixes I committed could make it into a 3.9.4 release some time soon... What can I do to help this happen? Assuming I do anything necessary, when could you do this release? (or give Christian permission to do it :-) ) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
Jim Fulton wrote: > On Thu, Nov 19, 2009 at 7:01 PM, Chris Withers wrote: >> Jim Fulton wrote: >>> There's nothing official or supported about a backup solution without >>> automated tests. >>> >>> So I guess there isn't one. >> Right, so what does Zope Corp use? > > We use ZRS, of course. Well, ZRS solves the HA challenge the same way as zeoraid, if I understand correctly, but what about offsite backups and the like? The project I'm currently working on uses repozo to create backups that: - get hoovered by the hosting provider's backup mechanisms and rotated offsite daily - get sprayed by rsync over ssh to a DR site on another continent How would ZRS solve these problems? > I'd prefer that there be a file-storage backup solution out of the box. > repozo is the logical choice. It sounds like it needs some love though. > This isn't something I'd likely get to soon. I'm not sure how much love repozo needs. It works, and it won't need changing until FileStorage's format changes, which I don't see happening any time soon. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
Jim Fulton wrote: > I need to review the changes before the release. I'll probably reject the > repozo change without an automated test. Are you serious? You'd rather have a broken tool than one that isn't broken on the basis that the existing tests aren't part of the test suite that gets run by default?! > In the future, please don't check changes directly into trunk or a > release branch. > Check them into a development branch. I'll review and merge. Sure, although these are both miniscule changes... I've learned my lesson, I won't try and contribute to ZODB development in future... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
Tim Peters wrote: > That's what he said -- and you made him repeat it several times by now. Yes, I find it hard to believe that someone would deliberately break something that someone else has taken the trouble to fix (and run the tests for!)... >> I've learned my lesson, I won't try and contribute to ZODB development >> in future... > > I think that's the wrong lesson to take. repozo had several fatal > bugs when I inherited it, and part of the reason is that there were no > tests of any kind. As the checkin comment said when I added > testrepozo.py: > > Better Than Nothing -- which is what we had before. > > There wasn't time then to finish the job (i.e., to automate the > testing and dope out some way to make failure output more /helpful/ > than just "guts don't match"). What I checked in then was essential, > though, to verify the slew of bugfixes that went in around the same > time. > > Alas, 5 1/2 years later, repozo testing apparently remains just barely > "Better Than Nothing", but that's really not good enough for a > supported approach. I'm not arguing with the above, and I'm not asking for anything more supported than already exists. However, requiring someone to completely rewrite a test suite for software that they're never needed to understand on the basis that they corrected some imports to make them compatible with a newer version of python seems unreasonable. > So the right lessons are: (1) to do development on a development > branch; Where is the development process that requires this documented? > and, (2) to finish the job I started if repozo is to be > treated as more than just another random piece of flotsam in the ZODB > ocean. I'm fine with the status quo... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
Jim Fulton wrote: > On Fri, Nov 20, 2009 at 9:32 AM, Chris Withers wrote: > ... >> I'm not sure how much love repozo needs. It works, and it won't need >> changing until FileStorage's format changes, which I don't see happening any >> time soon. > > It just occurred to me that repozo doesn't support blobs. Indeed, I've never been comfortable with blobs due to their spotty support in older tools and the need to make a filesystem HA for them to be used in a HA environment. As a result, I don't use them, so I'm okay with repozo not suppporting them. Yes, I'd love it if repozo *did* support them, but I wouldn't have the faintest idea of how to do that, even if I did have time to work on it... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
Jeff Shell wrote: > For what it's worth, I've dealt with broken / unloved / > mildly-out-of-date ZODB scripts by making a new package that my > little company uses internally. I don't agree with this because it's fragmenting a tiny development group even further, no to mention creating private forks of tools that mean everyone loses in terms of eyeballing... > It allowed me to get some useful > tools available again, Such as? And what stopped you working on them back in the ZODB source tree? > and was independent from any ZODB release > cycle. I have no problems with the ZODB release cycle... > You could always explore this option, and even do it better than me > by sharing with the community. This just feels wrong to me. If I *was* changing a script's behaviour such that it needed new tests, I'd make sure those were automatically run. The changes to repozo didn't require any new tests, the bug was provoked just by running the existing tests with python 2.6. > So if 'repozo' needs love, even if it's simple love, I would > recommend that someone make a 'z3c.zodbtools' or 'z3c.repozo' or > 'z3c.fsbackup' project. I couldn't agree less. This kind of fragmentation helps no-one. It's something I'd only consider as a last resort, and then the most likely action is just to roll my own sdist with a funky version number and stick it on the customer's index server. > Hell, I'll look at the package I've made and > see if I can do something along those lines as a starting/example > point. I'd much prefer to see your fixes worked back into the 3.9 branch and trunk of ZODB... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
Jim Fulton wrote: >> +class RepozoTests(unittest.TestCase): >> + >> +def test_importability(self): >> +from ZODB.scripts import repozo >> + > > I appreciate your dedication to quality. > > That test passes under ZODB 3.9.3 with Python 2.6. ...but it will emit a deprecation warning, which I thought we considered to be a test failure? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zope 2.12 and blobs
Jim Fulton wrote: > No, we should tell them they should use a mostly untested tool to do backups > that doesn't work with blobs. That should inspire confidence. > > Does Zope 2.12 use Blobs? Not by default, no. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs
Jeff Shell wrote: > On Nov 20, 2009, at 11:52 AM, Jim Fulton wrote: >>> If configured, sure. >> I would not trust repozo + blob backups. > > I don't trust Blobs. A year ago, I reverted all of the code I had for our CMS > that supported Blobs as it was impossible to copy their data in a basic Zope > 3 copy/past/move. I think that may have been fixed, but since there's > practically zero documentation on how to use blobs, use them wisely, use them > well, and use them in fairly plain Zope 3-ish applications, I lost confidence. That fairly accurately sums up how I feel about blobs... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] errors logged at info in ZEO's connection
Hi All (although I guess predominantly Jim),
ZEO/zprc/connection.py contains this rather odd piece of code:
raise
except Exception, msg:
if not isinstance(msg, self.unlogged_exception_types):
self.log("%s() raised exception: %s" % (name, msg),
logging.INFO, exc_info=True)
Any objections to me changing this on the 3.9 branch and trunk to:
raise
except Exception, msg:
if not isinstance(msg, self.unlogged_exception_types):
self.logger.exception(
"%s() raised exception: %s" % (name, msg),
)
...and making sure there's a test of some sort that demonstrates this?
It seems a bit bizarre that exceptions are logged at INFO rather than
ERROR...
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] errors logged at info in ZEO's connection
Jim Fulton wrote: >> It seems a bit bizarre that exceptions are logged at INFO rather than >> ERROR... > > Agreed. A simple fix and test on a branch would be appreciated. Done: http://svn.zope.org/ZODB/branches/chrisw-error_logging/ It would be great if this, too, could make it in a 3.9.4 release... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] client cache bugs when a client switches from one zeo server to another
Hi All, So, I got the following errors when using a setup I'll describe shortly: o Module Products.ZCatalog.CatalogBrains, line 86, in getObject o Module OFS.Traversable, line 250, in restrictedTraverse o Module OFS.Traversable, line 226, in unrestrictedTraverse __traceback_info__: ([], 'tvi-101108-rb3') o Module OFS.ObjectManager, line 728, in __getitem__ KeyError: 'An object id' This catalog is maintained in sync with the content as changes to one are always committed in the same transaction as changes to the other. o Module Products.BTreeFolder2.BTreeFolder2, line 293, in getBatchObjectListing IndexError: 124 This looks like an impossible error that occurs when a BTreeFolder's object count doesn't match the length of its objectIds(). These errors would be bizarre and disturbing were it not that they went away as soon as I restarted the Zope instance. That suggests client cache corruption to me as the client in question has no persistent cache. So anyway, the setup is one Zope 2.9.8 client connected to two zeoraid 1.0b7 zeo servers. The client has been switching between the two zeoraid servers while I've been testing zeoraid by bringing it up and down. So, I'm left wondering: - have there been any client cache bugs fixed with Zope 2.9.8 that would cause this kind of behaviour when a client has multiple zeo servers and switches between the two due to failure of one of them? - (mainly at Christian) can you think of anything in zeoraid that might cause this behaviour? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jim Fulton wrote: > Well, not really fair enough. I just looked at the change that Chris > made. It has a bug > that would be caught by your test if anyone tried to port repozo to > Python 3. Which alternate reality are you talking about here? ;-) > I'm going to back out these changes. Wow. Well, okay, whatever. I have an sdist cut from a revision where those changes do exist. It's annoying 'cos it means I won't be using 3.9.4 if that ever lands, but I'm not sure I care. > We are doing users a > serious disservice > giving them such an important tool with minimal tests and no > automated tests. Yes, because effectively deleting work that people have contributed is a *really* great way of encouraging people to contribute... Nevermind... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Martin Aspeli wrote: > - there were no tests for the existing code There are tests, they're just manual and spew a lot of output even when passing, but they do (as best I could tell) exercise the backup and restore cycle quite heavily. I ran these before and after to verify they didn't fail but did exhibit the behaviour I was trying to fix before I started and didn't fail *and* didn't exhibit the behaviour I was trying to fix after. > - Tres wrote a trivial test for the trivial fix Indeed. His test exhibited the problem prior to the fix and showed it to be fixed after. (since the problem was a warning generated on import) > - You now found some different problem in the same code, ...but didn't say what it is. Jim seemed to hint it was a Python 3 problem. I'd warrant there are much more serious problems to be worrying about for Python 3 first. If Python 3 compatibility is now a requirement for any patches submitted, then we're well and truly fucked. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jim Fulton wrote: > BTW, offline back ups if file-storage even with Blobs is > straightforward without repozo. I couldn't parse this, could you restate particularly wrt to what ou mean by offline? If there's a way other than repozo, then it would be great to know what it was... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jim Fulton wrote: > If you take the database down, Right, that was what I was hoping you didn't mean ;-) That's useless then. > Repozo gives you, for FileStorage: > > - Online backups -- you can back up a database while it is running > - Incremental backups > > Of course, it isn't being actively supported. Of course, it's having the snot hammered out of it by what I would guess is the largest group of ZODB users who run backups of any sort. Myself, I use it to reconstitute a 20GB database on a backup server once an hour, followed by running one of the myriad of zodb checker tools on the backup. It may not be actively supported but I'd wager we as a community would find out soon enough if it developed problems. I'll reiterate that in the universe the majority of zodb users inhabit, dropping repozo is simply not an acceptable thing to do. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jeff Shell wrote: > I'm not sure whose expectations you're satisfying as the items in > ZODB.scripts are a wild and inconsistent mess. It's been trial and error just > to find ones that seem to work, outside of repozo. Oh the irony of the one script that Jim is looking to remove, because someone dared to do some maintenance on it ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Making ZODB / ZEO faster
Jim Fulton wrote: > On Fri, Dec 4, 2009 at 11:31 AM, Erik Dahl wrote: > ... >> I haven't dove into the relstorage yet but have heard it will perform >> better. > > It doesn't. See: > > https://mail.zope.org/pipermail/zodb-dev/2009-October/012758.html Jim, That's a little sweeping and some might see you as a little biased in this ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
Marius Gedminas wrote: > * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well > as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 > it's gone. > > Solved by peeking into the method signature with inspect.getargspec() > and supplying a version only if it's needed. I believe this may be fixed with 3.9.4 is released. ...but it sounds like your code isn't ending up using the Z303 adapter, which is why you're running into the problem. How are you mixing versions? 3.8 on client and 3.9 on server or? (sorry, I lost the previous thread and saw this as a new one when Jim changed the subject line...) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: compatibility of ZEO clients and servers running different Python versions
Jim Fulton wrote: > Because of the complexity of the solution, I'm, sadly, inclined to > leave things as they are: > > - Can't use Python 2.4 clients with Python 2.5 or 2.6 servers > - Can use Python 2.6 clients with 2.4 servers. > - Can't use Python 2.5 or 2.6 servers with clients running ZODB 3.7 or > earlier. > > I wonder what other people think. Personally, I'm certainly okay with this. It'd be great to have it advertised somewhere prominent though... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problem with handling of data managers that join transactions after savepoints
Jim Fulton wrote: >>> I plan to implement A soon if there are no objections. >>> >>> Unless someone somehow convinced me to do D, I'll also add an >>> assertion in the Transaction.join method to raise an error if a >>> data manager joins more than once. >> Option A sounds sensible. It also means I won't have to change >> anything in the zope.sqlalchemy data manager. > > Very cool. I was hoping non-ZODB-data-manager authors > were paying attention. :) > > If anyone knows of any other, I would appreciate someone forwarding > this thread to them. zope.sendmail and MaildropHost have data managers. I've seen some file-based things that was a data manager and a few others in people's BFG stacks, maybe they can pipe up and/or let the others of those wsgi components know. I'm also likely about to write one, but I'm dumb, so can't comment meaningfully on the options ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] The database root: It's a trap!
Jim Fulton wrote: > A. Change ZODB so that new databases have BTrees as root objects. > >This has the advantage that BTrees are scalable. It has a number >of down sides: > >- BTrees should only be used when keys are known to have a stable > ordering. If we had a scalable hash data structure, that would > be better. > >- This encourages mistake 2 above. > > B. Change ZODB so that the root object is a variant of persistent >mapping that either refuses to store more than a small number of >objects, or at least issues a warning when more than a small >number of objects is stored. C. I'd really like to be able to configure what the root object is as database creation time. That way it can be application specific. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] The database root: It's a trap!
Jim Fulton wrote: >> C. I'd really like to be able to configure what the root object is as >> database creation time. That way it can be application specific. > > That sounds reasonable, but doesn't address the default behavior. Default would be an OOBTree for me... Scales, no thinking or scary warnings required :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Restoring from repozo and reusing an index file?
Paul Winkler wrote: > 3 should be pretty fast as its an incremental backup. > 4 may take a little while but iirc it's not too bad for this db - > maybe 10 minutes. > Most of the downtime is caused by rebuilding the index which takes way > longer (30 min?) > > So I had two questions: > > 1) would it be safe to copy the index file from the old host and just > use that with the filestorage generated by repozo? Almost certainly not. > 2) should I maybe ditch repozo and just rsync everything? That'd > replace step 3 with a second call to rsync, and eliminate steps 4 and > 6. But I'm not sure how fast it is at updating a big binary file. I > think I'll do an experiment today and report back on this option. I suppose you could look at zeoraid. Have you tried just starting with an empty index file? Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
Christian Theune wrote: > On 06/11/2010 05:50 PM, Alan Runyan wrote: I suppose you could look at zeoraid. >>> Hadn't thought of that, will look at it, thanks! >> Anyone using ZEORaid in production? > > I hope not. There's still open issues with split brain situations IIUC, that can be mitigated by only having one zeoraid server live at a time? > and a > mysterious bug WRT oid generation This one's new to me, where can I find out more? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
Christian Theune wrote: > On 06/12/2010 09:47 AM, Chris Withers wrote: >> Christian Theune wrote: >>> On 06/11/2010 05:50 PM, Alan Runyan wrote: >>>>>> I suppose you could look at zeoraid. >>>>> Hadn't thought of that, will look at it, thanks! >>>> Anyone using ZEORaid in production? >>> I hope not. There's still open issues with split brain situations >> IIUC, that can be mitigated by only having one zeoraid server live at a >> time? > > Yes. Our goal is to mitigate even splits by making the system read-only > and manually allowing to specify which parts are active in that case. > > An automatic voting procedure would then require more than 2 > participants, but with even splits you at least won't fail horribly. *nods sagely while looking a little blank* ;-) >>> and a >>> mysterious bug WRT oid generation >> This one's new to me, where can I find out more? > > The one you reported quite a while ago that ended in mixed-up OID results. Have you got a url to the tracker issue? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
Christian Theune wrote: > and a > mysterious bug WRT oid generation This one's new to me, where can I find out more? >>> >>> The one you reported quite a while ago that ended in mixed-up OID >>> results. >> >> Have you got a url to the tracker issue? > > I think it was this one: > https://bugs.edge.launchpad.net/gocept.zeoraid/+bug/485976 Okay, but again, this one seems to be mitigated by only running one zeoraid server at a time, right? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] tracking back to what code caused a conflicting object change
Hi All, Forwarding here as there may be more people with the relevant knowledge around. So, the key thing I'm trying to find is what code is changing the conflicting objects? (primarily since I'm not aware of any doing so, so it must be inadvertantly getting executed) Any ideas gratefully received! Chris PS: apologies for the inline forwarding, zope.org's MTAs are still incorrectly configured to reject messages forwarded as attachments... Original Message Subject: [Zope] ConflictError from Products.Transience.TransientObject.TransientObject Date: Tue, 27 Jul 2010 11:45:26 +0100 From: Chris Withers To: zope list user Hi All, I have an old Zope 2.9.8 instance which has recently started throwing the following unresolved ConflictErrors: Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 479, in commit Module ZODB.Connection, line 1050, in _commit_savepoint Module tempstorage.TemporaryStorage, line 202, in store ConflictError: database conflict error (oid 0x046a, class Products.Transience.TransientObject.TransientObject, serial this txn started with 0x0387c9f23edea155 2010-07-27 10:26:14.735074, serial currently committed 0x0387c9f28a1ff766 2010-07-27 10:26:32.373016) To my knowledge, the request that throws this isn't touching the session or temp_folder, but I guess something is. How can I find out what code is actually touch this and so causing this conflict? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - z...@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev ) Original Message Subject: [Zope] (also) ConflictError from Products.Transience.Transience.Increaser Date: Tue, 27 Jul 2010 11:47:09 +0100 From: Chris Withers To: zope list user References: <4c4eb8c6.8050...@simplistix.co.uk> Chris Withers wrote: > To my knowledge, the request that throws this isn't touching the session > or temp_folder, but I guess something is. > > How can I find out what code is actually touch this and so causing this > conflict? In similar circumstances, I also see: Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 479, in commit Module ZODB.Connection, line 1050, in _commit_savepoint Module tempstorage.TemporaryStorage, line 202, in store ConflictError: database conflict error (oid 0x29, class Products.Transience.Transience.Increaser, serial this txn started with 0x0387c9ffb271bc44 2010-07-27 10:39:41.822878, serial currently committed 0x0387ca0108963711 2010-07-27 10:41:02.012526) How can I find out what's going on to cause this conflict? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - z...@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev ) -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Zope-dev] Undo tab in Zope 2.12
Chris Withers wrote: > ...often appears to be empty when it shouldn't be. > > Has anyone else observed this? The answer turned out to be ZODB 2.10b4 being used on the storage server side. I found a clue in the ChangeLog but it sure would be nice if there was some logging of some sort to warn what was going on... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] attempting to undo an un-undoable transaction gives "NotImplementedError" with ZODB 3.9.5
Here's an example: 2010-08-26 22:49:01 ERROR txn.-1338986496 Failed to abort resource manager: at 47516176> Traceback (most recent call last): File "transaction-1.1.0-py2.6.egg/transaction/_transaction.py", line 475, in abort rm.abort(self) File "transaction-1.1.0-py2.6.egg/transaction/_transaction.py", line 548, in abort self.manager.abort(o, txn) File "ZODB3-3.9.5-py2.6-macosx-10.3-fat.egg/ZODB/DB.py", line 985, in abort raise NotImplementedError NotImplementedError This used to give a meaningful error message. Anyone know why it no longer does? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] attempting to undo an un-undoable transaction gives "NotImplementedError" with ZODB 3.9.5
On 09/09/2010 21:47, Jim Fulton wrote: >> 2010-08-26 22:49:01 ERROR txn.-1338986496 Failed to abort resource >> manager:> object at 0x2d509f0> at 47516176> >> Traceback (most recent call last): >>File "transaction-1.1.0-py2.6.egg/transaction/_transaction.py", line >> 475, in abort >> rm.abort(self) >>File "transaction-1.1.0-py2.6.egg/transaction/_transaction.py", line >> 548, in abort >> self.manager.abort(o, txn) >>File "ZODB3-3.9.5-py2.6-macosx-10.3-fat.egg/ZODB/DB.py", line 985, in >> abort >> raise NotImplementedError >> NotImplementedError >> >> This used to give a meaningful error message. > > In 3.7 and earlier. > >> >> Anyone know why it no longer does? > > The resource manager abort method used to be a pass. In 3.9 in raises > an exception, which hides the undo exception. That's bad, right? > An app should capture > the undo exception information before calling abort. "the app" here is Zope 2's publisher process, right? > My guess is that this works much better in ZODB 3.10. Any particular reason? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Stripping version data from databases
On 09/09/2010 15:10, Jim Fulton wrote: > I suspect we'll see more cases where people have databases that have > versioned records > that they don't know how to get rid of. (We had a case like this > internally, but we stopped > caring about the database in question before we had to deal with it. :) > > I'd be happy to throw together a script that stripped version records using > the storage iteration and restore protocols. I would want someone else to > review and (possibly :) test it. I'm can't commit to much effort on it > though. > I'm not willing to include this in ZODB, but would probably put it in my > public script "zoo", http://svn.zope.org/Sandbox/J1m/scripts/. Didn't you do this already? https://mail.zope.org/pipermail/zodb-dev/2009-August/012676.html https://mail.zope.org/pipermail/zodb-dev/2009-August/012684.html ...and the buildout I use to run it: https://mail.zope.org/pipermail/zodb-dev/2009-August/012690.html Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] "Versions aren't supported" message in Plone 4 upgrade
On 09/09/2010 23:31, David Glick wrote: > I meant that we exported the database under Plone 3.x and reimported to > a new database in the same Zope instance (still Plone 3.x with the same > versions of all software). *Then* proceeded with a normal upgrade to > 4.0 using the normal method and the new Data.fs. ...or you could just use the micro-buildout I linked to previously which lets you just strip out the versions. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB history problem with Zope 2.12.7
On 28/09/2010 12:55, Tres Seaver wrote:
> On 09/28/2010 03:36 AM, Chris Withers wrote:
>> On 28/09/2010 00:12, Marius Gedminas wrote:
>>> --- ./OFS/History.py.orig 2010-09-28 02:11:56.535745440 +0300
>>> +++ ./OFS/History.py2010-09-28 02:12:00.043764683 +0300
>>> @@ -151,6 +151,9 @@
>>>base = aq_base(self)
>>>base._p_activate() # make sure we're not a ghost
>>>base.__setstate__(state) # change the state
>>> +for attr in dir(base):
>>> +if attr.startswith('_v_'):
>>> +delattr(base, attr)
>>>base._p_changed = True # marke object as dirty
>>>self.manage_afterHistoryCopy()
>>
>> Thanks, I guess I'll monkey patch for now, here's the bug:
>>
>> https://bugs.launchpad.net/zope2/+bug/649605
>>
>> However, I'm curious, so the above will fix the object in the current
>> thread, but what about objects in other threads?
>>
>> (or do _v_ attributes get killed off at the start of each transaction?)
>
> Only when objects are ghostified (due to an invalidation from another
> thread or process) or evicted from the cache. I'm not quite sure how
> the case you are triggering occurs, but if that code saves the new-old
> version of the template to the ZODB, then any instances in other
> threads' connection caches should be invalidated.
Feel free to repeat the steps I originally posted and are in the bug
description ;-)
Looks like the History.py code isn't doing something it should, but I
don't know when or what changed to make this so. I'll CC zodb-dev in
case Jim knows of any changes in ZODB (I'm using 3.9.6) that might be
relevant...
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] semantics of _v_ attributes?
Hi All, I'm trying to fix this bug: https://bugs.launchpad.net/zope2/+bug/649605 ...and I'm struggling to find documentation on the semantics of _v_ attributes. In particular, surely this code: base._p_activate() # make sure we're not a ghost base.__setstate__(state) # change the state base._p_changed = True # marke object as dirty ...should be enough to indicate that all _v_ attributes should be dropped? If it isn't, what is the correct way to indicate to Persistent/ZODB that all _v_ attribute should be dropped not only for the current object but for all occurrences of that object across all connections to the current ZODB? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] semantics of _v_ attributes?
On 06/10/2010 15:19, Jim Fulton wrote: > _v_ attributes should be dropped whenever an object's state is > cleared. It would be great if there was some clear documentation covering all this in the ZODB package... > So, you want to update an object using old state. > > I would use: > >base._p_invalidate() # remove state, including volatiles, no matter what >base._p_activate() # make sure we're not a ghost >base.__setstate__(state) # change the state >base._p_changed = True # marke object as dirty > > This isn't foolproof, depending on how an object uses _v_. For > example if __setstate__ sets _v_s if they aren't already set, then the > above code would likely fail, as the _v_s would reflect the old state, > not the new. Okay, it works for the Zope 2.12 problem at hand: https://bugs.launchpad.net/zope2/+bug/649605 > If this use case is important, it would be better for ZODB tp provide > an API to handle this directly, so you didn't have to hack together > existing APIs. Well, I have customers who think it's important, but it depends on whether you still think Zope 2 with TTW editing is important ;-) > I assume this code will ultimately be followed by a commit. Correct. > Well, for your use case, you're going to be committing a change to the > object. That will cause instances of the object in all other > connections to be invalidated, dropping their volatile attributes. Cool :-) I'll take this on trust, I don't want to contemplate trying to write a test that proves it ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zodb 3.9.6 performance tuning?
Hi, After recently moving a medium-sized customer project from Zope 2.9.8 to Zope 2.12.7 sitting on ZODB 3.9.6 by way of zeoraid b7, I'm now seeing some fairly weird and bad behaviour that we didn't have before. So, everything was fine after the migration. However, today I turned on a batch job which pumps content into the storage. This appears to be *much* slower than on Zope 2.9.8 (with its included ZODB release) and also appears to cause the web clients to be slow enough when its running that they start dropping out of the load balancer pool. How can I debug what's going on? Is there any way to get diagnostics out of a 3.9.6 zeo server to find out what it's up to? cheers, Chris PS: If it helps, it looks like the zeoraid process is using an order of magnitude more cpu than the zeo process, according to top, but even that's only 30% of the CPU load... -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
Hi All, I'm wondering if some of my issues might be alleviated by persistent client caches. Now, I have the impression that these were unreliable and buggy and had been so for some time. Is this still the case? If not, where are the docs on how to configure them nowadays? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
On 12/10/2010 17:11, Jim Fulton wrote: >> Now, I have the impression that these were unreliable and buggy and had >> been so for some time. > > They became much better in later 3.8 releases and still much better lately. > > We've been using them in production for quite a while. Any docs on recommended settings around? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] blocked transactions?
Hi All, What do the following error message mean: -- 2010-10-12T17:05:58 INFO ZEO.StorageServer (24471/192.168.100.100:54407) Transaction blocked waiting for storage. Clients waiting: 3. -- 2010-10-12T17:07:16 INFO ZEO.StorageServer (24471/192.168.100.105:50112) Blocked transaction restarted. Clients waiting: 2 ...and would ZODB 3.10 help fix them? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage and ZODB 3.10?
Hi Shane, Does RelStorage 1.4 work with ZODB 3.10? (In particular, the Undo changes and the tightened storage API..) I see the mention of 3.10.0b7, but just wanted to check that means you're okay with 3.10 final ;-) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blocked transactions?
On 13/10/2010 14:06, Jim Fulton wrote: > On Wed, Oct 13, 2010 at 4:02 AM, Chris Withers wrote: >> Hi All, >> >> What do the following error message mean: >> >> -- >> 2010-10-12T17:05:58 INFO ZEO.StorageServer (24471/192.168.100.100:54407) >> Transaction blocked waiting for storage. Clients waiting: 3. >> -- >> >> 2010-10-12T17:07:16 INFO ZEO.StorageServer (24471/192.168.100.105:50112) >> Blocked transaction restarted. Clients waiting: 2 >> >> ...and would ZODB 3.10 help fix them? > > These aren't error messages. You can tell because they say "INFO". If you say so ;-) Their occurences coincided with an app server cluster of 4 clients locking up completely... I would dearly love to know why that happened. Any way to interrogate a running zeo server to find out what it thinks it's up to? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blocked transactions?
On 13/10/2010 15:23, Jim Fulton wrote:
>> Their occurences coincided with an app server cluster of 4 clients locking
>> up completely...
>
> Did the numbers get much bigger than 3?
Nope.
>> I would dearly love to know why that happened.
>
> Typically, it's due to a client that votes and fails to call finish.
Wouldn't I see storage errors in that case?
The only errors I saw were these and quite a few:
File
"/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/ServerStub.py",
line 377, in stub
raise ValueError("Timeout waiting for protocol handshake")
ValueError: Timeout waiting for protocol handshake
>> Any way to interrogate a
>> running zeo server to find out what it thinks it's up to?
>
> You can connect to the monitor port in 3.9 and earlier,
When that's configured, what information does it provide and how do I
get it? (if there are docs, lemme know and I'll go read them instead)
> You almost certainly want to set transaction-timeout in
> your server configuration. This will cause transactions that take too
> long to be terminated. We use a transaction timeout of 300 seconds.
5 mins is pretty high, though, right? Surely if all clients end up
hanging for 230 seconds, they'll all be dropped by a load balancer?
Related: how can I find out how long transactions are taking?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blocked transactions?
On 13/10/2010 15:40, Jim Fulton wrote:
>> Wouldn't I see storage errors in that case?
>
> No. Unless you set a transaction timeout, the storage will
> wait for the finish indefinitely.
...I meant on the client side. There were no errors other than the
protocol timeout ones below:
>> File
>> "/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/ServerStub.py",
>> line 377, in stub
>> raise ValueError("Timeout waiting for protocol handshake")
>> ValueError: Timeout waiting for protocol handshake
>
> These were on the client, not the server.
>
> That's odd.
>
> This suggests that there's something else going on with your storage
> server and that the transactions waiting messages are a red herring.
Indeed. I'd dearly love to know what, sadly, the storage server is
zeoraid, and I can't expect you to help with that ;-)
(zeoraid then talks to back end zeo servers, and the only
out-of-the-ordinary stuff logged was on the zeoraid servers, and it was
those blocked messages)
>> When that's configured, what information does it provide and how do I get
>> it? (if there are docs, lemme know and I'll go read them instead)
>
> Go read the code or try it.
Will do.
>> Related: how can I find out how long transactions are taking?
>
> Note that we're really talking about how long commits are taking,
> specifically the time between vote and finish. You can determine that
> from the waiting messages.
So there's no logging I can turn on to see how long commits are taking,
unless I can get more of those 'transaction blocked' messages to show up?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blocked transactions?
On 13/10/2010 15:59, Jim Fulton wrote: >> So there's no logging I can turn on to see how long commits are taking, >> unless I can get more of those 'transaction blocked' messages to show up? > > You can set the log level to debug. This will log the entire ZEO > protocol. Analysis will take require a bit of work. (Note that ZEO > 3.10 no longer does this level of logging even at the debug level, as > it slowed the server down.) yeah, slowing the server down is what I'm worried about so I'm not gonna do this unless someone tells me I really need to ;-) > You could temporarily hack the ZEO storage server code to collect > statisics. That would probably be easier than debug logging. It worries me a little that this kind of thing, which feels important for ZODB tuning, isn't available without hackery. Are there any plans to get some storage server stats accumulation? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 13/10/2010 15:23, Jim Fulton wrote: > You can connect to the monitor port in 3.9 and earlier, > if the monitor port is configured. In 3.10, the monitor server is > replaced by a ZEO client method, server_status. This tells you > much the same information that's in the log messages. Okay, monitor port up and running now. I see commits listed when I'm not expecting any. Do we have any kind of "tail -f /some/filestorage.fs" yet? (or have we always had such a tool) to see what the last few transactions in the underlying file storage look like in a human-readable form? cheers, Chrus -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
On 12/10/2010 17:56, Jim Fulton wrote:
> Currently the only configuration "docs" are in the .xml Alan provided a link
> to.
>
> The best size will depend on the app.
Is there a downside to "too big", provided there's plenty of disk space
free on the app servers?
> There are some docs in
> http://svn.zope.org/ZODB/trunk/doc/, although I don't know if they're
> accurate.
...yeah, those look a lot like the original docs that Guido(?) wrote.
> In ZODB 3.10, the cache tracing analysis tools actually work and produce
> meaningful numbers if you start tracing with an empty cache.
Okay, does that mean the stuff that's documented should be ignored for
ZODB 3.9?
I went for the following in the end:
...
cache-size 200Mb
client app
wait on
mount-point /
cache-size 5
cache-size-bytes 500Mb
Any obvious issues with that?
One problem I did hit, though, was if I restarted an app server, I got:
File
"/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/config.py",
line 210, in open
**options)
File
"/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/ClientStorage.py",
line 395, in __init__
self._cache = self.ClientCacheClass(cache_path, size=cache_size)
File
"/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/cache.py", line
194, in __init__
self._lock_file = zc.lockfile.LockFile(path + '.lock')
File
"/var/buildout-eggs/zc.lockfile-1.0.0-py2.6.egg/zc/lockfile/__init__.py", line
76, in __init__
_lock_file(fp)
File
"/var/buildout-eggs/zc.lockfile-1.0.0-py2.6.egg/zc/lockfile/__init__.py", line
59, in _lock_file
raise LockError("Couldn't lock %r" % file.name)
LockError: Couldn't lock 'zec.lock'
...and yet the app server came up fine. What does this error message imply?
The above did not, however, occur if I stopped and then started the app
server...
Race condition?
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 14/10/2010 01:28, Darryl Dixon - Winterhouse Consulting wrote: >> Do we have any kind of "tail -f /some/filestorage.fs" yet? (or have we >> always had such a tool) to see what the last few transactions in the >> underlying file storage look like in a human-readable form? > > fsdump.py gets you pretty close (ZODB/scripts/fsdump.py). Between that and > the Undo log for the DB inside Zope, you might be able to figure it out... ..well, the Undo log shows no entries ;-) ...and fsdump appears to have no way of saying "show me the last few transactions in this 35GB file" :-( Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 14/10/2010 01:30, Laurence Rowe wrote: > There's also fstail: > > $ bin/zopepy -m ZODB.scripts.fstail var/filestorage/Data.fs That looks more like it.. Any ideas what options there are to it? From reading the source, I see a -n, but no idea what it should do. I'm looking for a "more detail" knob, and also a 'tail -f' equivalent... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction commits don't have to write data?
On 14/10/2010 01:21, Chris Withers wrote: > On 13/10/2010 15:23, Jim Fulton wrote: >> You can connect to the monitor port in 3.9 and earlier, >> if the monitor port is configured. In 3.10, the monitor server is >> replaced by a ZEO client method, server_status. This tells you >> much the same information that's in the log messages. > > Okay, monitor port up and running now. > I see commits listed when I'm not expecting any. ...but using fstail, I see no new transactions in the last 6 hrs. Is it normal/ok/etc to commits in the monitor server output but without and actual transaction showing up in fstail? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 14/10/2010 02:40, Darryl Dixon - Winterhouse Consulting wrote: >> ...and fsdump appears to have no way of saying "show me the last few >> transactions in this 35GB file" :-( > > Yeah it's pretty verbose. I usually just pump the output to a text file ...yeah, but a 35GB zodb is going to result in, what, a 120? 200? GB text file. I don't think I actually have that much disk space lying around on these servers, let alone how long fsdump would take to produce it ;-) I wonder how hard it'd be to add some kind of start-at-the-end feature to fstail? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage Questions
Hi Shane, For all of these, if there are docs, please point me in their direction. I've been working from: http://pypi.python.org/pypi/RelStorage In my case, I'm looking at ZODB 3.9.6, Zope 2.12.7 onto MySQL 5.1 provided as a two machine cluster maintained by a hosting company. This sound good? Any different versions I should look at? From the features section: "multiple ZODB instances can share the same database"; does this mean I can put more than one storage in the same mysql database? Is this a good idea or should I stick to one storage per mysql database? From the RelStorage options section: - I'll have one storage with keep-history true and one with it false. Both will require packing and garbage collection, right? What's the recommended method and frequency of doing that? I'm guessing the zodbpack script but wanted to doubt check that does both pack and garbage collection. - How do you pick the right pack-duty-cycle? - poll-interval - when is it recommended to use memcached caching? How do you decide to pick what values to use for cache-prefix, cache-local-mb and cache-delta-size-limit? - Under what situations would you recommend changing the commit-lock-timeout from its default of 30 seconds? From the MySQL Adapter options section: - When is/isn't gzip compression a good idea? What is actually being compressed by this options? - Should I worry about init_command, read_default_file, read_default_group and client_flag or just ignore them? - What does load_infile actually do? Should it be used or avoided? Sorry for all the questions, hope these aren't too much of a pain to answer! cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] One more RelStorage question
Hi Shane, I forgot to ask... Is RelStorage suitable for a temporary storage? If so, what would be the recommended options for the following storage: # Temporary storage database (for sessions) name temporary storage for sessioning mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer ...and could one database containing such a temporary storage be used by more than one client, and would that be a good idea? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
On 14/10/2010 17:09, Jim Fulton wrote: >> What does this error message imply? > > That two processes were trying to access the cache. > >> The above did not, however, occur if I stopped and then started the app >> server... >> >> Race condition? > > Hopefully, you screwed up somehow and started the process while it was > already running. :) I don't think so. This is using zc.recipe.rhrc to drive zope2instance Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
