Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 8:12 PM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Dec 10, 2009 at 02:07:06PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 1:50 PM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas mar...@gedmin.as wrote: In 3.9 for FileStorage, if you give a starting tid that is toward the end of the file, it will scan backward, saving a lot of time. Ah, but how can I get a valid tid that is toward the end of the file? I assume that if I give a tid that doesn't exist, I'll get an error -- which is the behavior oldstate() exhibits -- rather than the next existing transaction after that timestamp, or I could fudge by getting the current timestamp and subtracting 24 hours. No, the iterator starts at the first transactions who's tid is = the start tid. So, guess a time and use repr(ZODB.TimeStamp.TimeStamp(...)) Cool! Thank you, Marius Gedminas -- another way is to use the fstools.prev_txn starting at the end of the file to grab the newest transactions, the script i send to the list last week uses it to poke a large zodb to just get the contents of the last 20 transactions without iterating through the whole thing or guessing a timestamp. https://mail.zope.org/pipermail/zodb-dev/attachments/20091207/531fc1e4/attachment.obj cheers, kapil ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
Marius Gedminas wrote: * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 it's gone. Solved by peeking into the method signature with inspect.getargspec() and supplying a version only if it's needed. I believe this may be fixed with 3.9.4 is released. ...but it sounds like your code isn't ending up using the Z303 adapter, which is why you're running into the problem. How are you mixing versions? 3.8 on client and 3.9 on server or? (sorry, I lost the previous thread and saw this as a new one when Jim changed the subject line...) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 11:08:25AM +, Chris Withers wrote: Marius Gedminas wrote: * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 it's gone. Solved by peeking into the method signature with inspect.getargspec() and supplying a version only if it's needed. I believe this may be fixed with 3.9.4 is released. ...but it sounds like your code isn't ending up using the Z303 adapter, which is why you're running into the problem. What's Z303? How are you mixing versions? 3.8 on client and 3.9 on server or? I'm not mixing versions at runtime; my intent is to write code that works with either ZODB 3.8.x or 3.9.x. You can see it here in all its glory: http://bazaar.launchpad.net/~zodbbrowser-dev/zodbbrowser/trunk/annotate/head:/src/zodbbrowser/history.py#L30 http://bazaar.launchpad.net/~zodbbrowser-dev/zodbbrowser/trunk/annotate/head:/src/zodbbrowser/state.py#L115 (sorry, I lost the previous thread and saw this as a new one when Jim changed the subject line...) This is a new topic, completely unrelated to the previous thread. Marius Gedminas -- I'm a shareware signature! Send $2 if you use me, $10 for a manual. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 5:58 AM, Marius Gedminas mar...@gedmin.as wrote: On Wed, Dec 09, 2009 at 01:04:03PM -0500, Jim Fulton wrote: On Wed, Dec 9, 2009 at 12:06 PM, Marius Gedminas mar...@gedmin.as wrote: ... (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very ugly hacks I managed.) This sounds like something that needs to be fixed. Can you share some of the issues you ran into? (Or maybe file bugs reports.) Two issues only: * In ZODB 3.8 PersistentDict and PersistentMapping are unrelated classes, so if I want to have adapters for both, I have to register them separately in ZCML. In ZODB 3.9, PersistentDict is an alias for PersistentMapping, so I get a ZCML configuration conflict error if I have adapter directives for both. OK, it's impractical to do anything about this. Solved by defining a decoy class for one of the adapters so they don't conflict. Hm, I'm not sure I follow this. You could have used a different approach of providing the second adapter registration as an override. * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 it's gone. It's mandatory in ClientStorage and optional in FileStorage. (MappingStorage and DemoStorage don't have a history method in 3.8. I suggest we treat this as a bug in 3.8 and make the version argument to ClientStorage's history method optional in a 3.8 bug fix release. Solved by peeking into the method signature with inspect.getargspec() and supplying a version only if it's needed. Ick. :) In particular, I think it should be a goal that it isn't too hard to write code that works with both ZODB 3.8 and 3.9. Normal code doesn't usually play with storage.history, I'd say, so I'm not too peeved by having to do various contortions. Zope 2 had (has?) a standard view for looking at an object's history that I found very useful. I wish there was one for Zope3. History can be very useful in my experience. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 10:08:36AM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 5:58 AM, Marius Gedminas mar...@gedmin.as wrote: On Wed, Dec 09, 2009 at 01:04:03PM -0500, Jim Fulton wrote: On Wed, Dec 9, 2009 at 12:06 PM, Marius Gedminas mar...@gedmin.as wrote: ... (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very ugly hacks I managed.) This sounds like something that needs to be fixed. Can you share some of the issues you ran into? (Or maybe file bugs reports.) Two issues only: * In ZODB 3.8 PersistentDict and PersistentMapping are unrelated classes, so if I want to have adapters for both, I have to register them separately in ZCML. In ZODB 3.9, PersistentDict is an alias for PersistentMapping, so I get a ZCML configuration conflict error if I have adapter directives for both. OK, it's impractical to do anything about this. Solved by defining a decoy class for one of the adapters so they don't conflict. Hm, I'm not sure I follow this. Code: if PersistentMapping is PersistentDict: # ...snip large comment explaining this... class DecoyPersistentDict(PersistentMapping): Decoy to avoid ZCML errors while supporting both ZODB 3.8 and 3.9. class PersistentDictState(PersistentMappingState): Decoy to avoid ZCML errors while supporting both ZODB 3.8 and 3.9. adapts(DecoyPersistentDict, dict, None) else: class PersistentDictState(PersistentMappingState): Convenient access to a persistent dict's items. adapts(PersistentDict, dict, None) ZCML: adapter factory=.state.PersistentDictState / adapter factory=.state.PersistentMappingState / You could have used a different approach of providing the second adapter registration as an override. Oooh, neat hack. But I'm not sure I like it better than the one I implemented. * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 it's gone. I went back and corrected my statement (originally it referred to FileStorage.history, and I was thinking about ClientStorage in my parenthetical statement), and then I of course forgot to remove the now-obsolete parenthetical statement itself. It's mandatory in ClientStorage and optional in FileStorage. (MappingStorage and DemoStorage don't have a history method in 3.8. I suggest we treat this as a bug in 3.8 and make the version argument to ClientStorage's history method optional in a 3.8 bug fix release. +1 Solved by peeking into the method signature with inspect.getargspec() and supplying a version only if it's needed. Ick. :) :) In particular, I think it should be a goal that it isn't too hard to write code that works with both ZODB 3.8 and 3.9. Normal code doesn't usually play with storage.history, I'd say, so I'm not too peeved by having to do various contortions. Zope 2 had (has?) a standard view for looking at an object's history that I found very useful. I wish there was one for Zope3. Well, there is IStorage, and I think all storages (except for FileStorage) implemented it faithfully (oid, version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated the interface by renaming size=1 to length=1, but it's fixed in 3.9. I personally would like to be able to have size=None imply unlimited, give me as much as you've got), but works just as well in practice. What I miss _a lot_ is a way to look at the last several transactions. There's the iterator API, but it goes from oldest-to-latest, and when you've got a large DB, that takes a while (85 seconds for a Data.fs with 922737 transactions and 8 GB of data, with 100 MB/s read throughput). History can be very useful in my experience. Oh yes. This is why I stopped packing my production Data.fs'es and first started using storage.history() with a couple of nice wrappers from debugzope shell, and, later, convinced a coworker to write ZODBBrowser. Marius Gedminas -- I used to think I was indecisive, but now I'm not so sure. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas mar...@gedmin.as wrote: ... Well, there is IStorage, and I think all storages (except for FileStorage) implemented it faithfully I guess that depends on what you mean by faithfully. MappingStorage and Demostorage in 3.8 inherited a null implementation from BaseStorage and it's arguable how faithful that implementation is. (oid, version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated the interface by renaming size=1 to length=1, but it's fixed in 3.9. and in 3.8. ... What I miss _a lot_ is a way to look at the last several transactions. There's the iterator API, but it goes from oldest-to-latest, and when you've got a large DB, that takes a while (85 seconds for a Data.fs with 922737 transactions and 8 GB of data, with 100 MB/s read throughput). In 3.9 for FileStorage, if you give a starting tid that is toward the end of the file, it will scan backward, saving a lot of time. I've attached an analysis script that illustrates iterating backward. Someday, someone should get around to adding an option to the file-storage iterator to go backward. Jim -- Jim Fulton transaction_rates.py Description: Binary data ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas mar...@gedmin.as wrote: ... Well, there is IStorage, and I think all storages (except for FileStorage) implemented it faithfully I guess that depends on what you mean by faithfully. MappingStorage and Demostorage in 3.8 inherited a null implementation from BaseStorage and it's arguable how faithful that implementation is. (oid, version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated the interface by renaming size=1 to length=1, but it's fixed in 3.9. and in 3.8. *checks* FileStorage in 3.8.1 violated the interface, to be precise. ZODB 3.8.1 is mandated by http://download.zope.org/zope3.4/3.4.0/versions.cfg, which our production app currently uses. (Kinda offtopic: Is there a newer 3.4 KGS release? http://download.zope.org/ tells me Welcome and nothing more. http://download.zope.org/zope3.4/ tells me in great big joyful letters at the top of the page zope 3.4.0 Has Been Released!) What I miss _a lot_ is a way to look at the last several transactions. There's the iterator API, but it goes from oldest-to-latest, and when you've got a large DB, that takes a while (85 seconds for a Data.fs with 922737 transactions and 8 GB of data, with 100 MB/s read throughput). In 3.9 for FileStorage, if you give a starting tid that is toward the end of the file, it will scan backward, saving a lot of time. Ah, but how can I get a valid tid that is toward the end of the file? I assume that if I give a tid that doesn't exist, I'll get an error -- which is the behavior oldstate() exhibits -- rather than the next existing transaction after that timestamp, or I could fudge by getting the current timestamp and subtracting 24 hours. I've attached an analysis script that illustrates iterating backward. Thank you! (Mutt is inconvenient: there's no way to look at attachments while you're replying to a message.) Someday, someone should get around to adding an option to the file-storage iterator to go backward. Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 1:50 PM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas mar...@gedmin.as wrote: In 3.9 for FileStorage, if you give a starting tid that is toward the end of the file, it will scan backward, saving a lot of time. Ah, but how can I get a valid tid that is toward the end of the file? I assume that if I give a tid that doesn't exist, I'll get an error -- which is the behavior oldstate() exhibits -- rather than the next existing transaction after that timestamp, or I could fudge by getting the current timestamp and subtracting 24 hours. No, the iterator starts at the first transactions who's tid is = the start tid. So, guess a time and use repr(ZODB.TimeStamp.TimeStamp(...)) Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 02:07:06PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 1:50 PM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas mar...@gedmin.as wrote: In 3.9 for FileStorage, if you give a starting tid that is toward the end of the file, it will scan backward, saving a lot of time. Ah, but how can I get a valid tid that is toward the end of the file? I assume that if I give a tid that doesn't exist, I'll get an error -- which is the behavior oldstate() exhibits -- rather than the next existing transaction after that timestamp, or I could fudge by getting the current timestamp and subtracting 24 hours. No, the iterator starts at the first transactions who's tid is = the start tid. So, guess a time and use repr(ZODB.TimeStamp.TimeStamp(...)) Cool! Thank you, Marius Gedminas -- This message has been ROT-13 encrypted twice for higher security. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Wed, Dec 9, 2009 at 12:06 PM, Marius Gedminas mar...@gedmin.as wrote: ... (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very ugly hacks I managed.) This sounds like something that needs to be fixed. Can you share some of the issues you ran into? (Or maybe file bugs reports.) In particular, I think it should be a goal that it isn't too hard to write code that works with both ZODB 3.8 and 3.9. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
