Re: [ZODB-Dev] [fsrefs] KeyError: 'n'
Hi Andreas, Your error looks quite similar to: http://www.mail-archive.com/zodb-dev@zope.org/msg02207.html Maybe it's the same issue? -- Sidnei ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Python 2.6 and Windows
On Tue, Oct 7, 2008 at 12:34 PM, Jim Fulton <[EMAIL PROTECTED]> wrote: > > Python 2.6 requires a different version of the MS C compiler than > Python 2.4 or Python 2.5. That would be Visual C++ 2008. The good news is that the Express edition which is freely available works just fine for this. > I don't have this compiler. Also, I don't > know if I can have 2 versions of the compiler available in the same > machine. Yes. In fact, I have 3 versions in our build box, VC6, 2003 and 2008. distutils is usually smart enough to figure out the right compiler to use, and when it doesn't, you just have to run 'vsvars32.bat' from the appropriate installed version. > Does anyone want to volunteer to run final tests and create distros > for ZODB on Windows and Python 2.6? As long as you provide instructions, I would be willing to do this. > I can keep testing and generating releases for ZODB and Python 2.4/2.5 > on Windows. I imagine that most or all windows bugs will be > independent of Python version. Yes, I don't see a reason for this to be any different. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Backing up Data.fs and blob directory
I mean, you have to install cygwin On Mon, Sep 8, 2008 at 3:30 PM, Chris Withers <[EMAIL PROTECTED]> wrote: > Sidnei da Silva wrote: >> >> Keep in mind rsync is not erm, trivial to get going on Windows. > > Really? I've never had problems with cygwin... > > cheers, > > Chris > > -- > Simplistix - Content Management, Zope & Python Consulting > - http://www.simplistix.co.uk > -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Backing up Data.fs and blob directory
Keep in mind rsync is not erm, trivial to get going on Windows. On Thu, Sep 4, 2008 at 1:54 PM, Tres Seaver <[EMAIL PROTECTED]> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Christian Theune wrote: >> Hi Laurence, >> >> On Wed, 2008-09-03 at 08:06 -0700, Laurence Rowe wrote: >>> Backing up a ZODB has always been fairly easy in the past, but with the >>> introduction of blobs things have got a little more complex. >>> >>> How should I create a consistent backup of my Data.fs and blob directory? >>> >>> My inital guess would be to take a copy of the Data.fs, then take a copy of >>> the blob directory to ensure I have all blobs referenced in the Data.fs. >>> Would I be able to restore from such a backup safely? (it may contain blobs >>> from transactions that were newer than the backed up Data.fs). >>> >>> This should be safe because committed blobs are immutable and any dangling >>> blobfiles would not interfere with the creation of blobs from new >>> transactions in the restored zodb, as transaction ids would not overlap. >>> >>> I would be greatful if anyone could point out holes in my reasoning or has >>> experience of this. >> >> Snapshotting a blob directory after taking a copy of your Data.fs should >> be safe, as long as you don't pack in between. >> >> Note that at the design stage we imagined that blob directories might >> become really large making backups unfeasable. For those situations we >> handwaved a "very reliable storage" for this directory, like a >> self-contained SAN/NAS solution that keeps your data safe. > > Assuming we can avoid the race condition induced by packing during the > backup (see below), it should be possible to write a script which > combines of 'repozo' and 'rsync' in such a way as to get a "pure" copy > of the blob directory which corresponds to the repozo dataset. > Somethiing like: > > #!/bin/sh > touch .repozo_start > /path/to/repozo -B -f var/Data.fs -r backups/ > find /path/to/blobs ! -newer .repozo_start |\ > rsync -av --include-from=- /path/to/blobs/ backups/blobs/ > rm .repozo_start > > Packing *during* backup creates problems for repozo + blobs, because a > pack may cause "old" blob files to be unlined. Furthermore, it is not a > use case I think we should support. However, I don't think that > FileStorage supports the idea of a "pack lock" which would be acquirable > by repozo, so I don't know how to prevent the race. > > > > Tres. > - -- > === > Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] > Palladion Software "Excellence by Design"http://palladion.com > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.6 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iD8DBQFIwATJ+gerLs4ltQ4RAm0WAKC/jDUPqBnMTpwkDpBX0mKidGTxvwCghjnM > QYnz9dRolzOdvZX2t9fxM3k= > =/h4K > -END PGP SIGNATURE- > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree pickle size
On Fri, Aug 22, 2008 at 9:49 AM, Roché Compaan <[EMAIL PROTECTED]> wrot> Transaction detail for txn #00099 (first document): > > Txn id,Classname,Object count,Size (bytes) > #00099,BTrees._IIBTree.IIBTree,3,286 > #00099,OFS.Folder.Folder,1,55 > #00099,BTrees._IOBTree.IOBucket,9,4572 > #00099,BTrees._OIBTree.OIBucket,5,2964 > #00099,BTrees._IOBTree.IOBTree,39,17552 > #00099,BTrees.Length.Length,27,768 > #00099,Persistence.mapping.PersistentMapping,2,846 > #00099,Products.ATContentTypes.content.document.ATDocument,1,1544 > #00099,BTrees._OOBTree.OOBTree,20,3986 > #00099,BTrees._IIBTree.IISet,3,184 > #00099,BTrees._OIBTree.OIBTree,9,1404 > #00099,Products.Archetypes.BaseUnit.BaseUnit,3,767 > #00099,BTrees._OOBTree.OOBucket,2,3286 > #00099,BTrees._IIBTree.IITreeSet,55,3905 > > ?Transaction detail for txn #10099 (last document): > > Txn id,Classname,Object count,Size (bytes) > #10099,BTrees._IIBTree.IIBTree,8,2517 > #10099,OFS.Folder.Folder,1,55 > #10099,BTrees._IOBTree.IOBucket,57,81564 > #10099,BTrees._OIBTree.OIBucket,13,9872 > #10099,BTrees._IIBTree.IIBucket,29,20024 > #10099,BTrees._IOBTree.IOBTree,1,85 > #10099,Persistence.mapping.PersistentMapping,2,846 > #10099,BTrees.Length.Length,22,655 > #10099,Products.ATContentTypes.content.document.ATDocument,1,1544 > #10099,BTrees._OOBTree.OOBTree,6,30455 > #10099,BTrees._IIBTree.IISet,65,182708 > #10099,Products.Archetypes.BaseUnit.BaseUnit,3,767 > #10099,BTrees._OOBTree.OOBucket,16,8088 > #10099,BTrees._IIBTree.IITreeSet,2,122 It's pretty clear that the difference here is the IISet(65 vs 3) and the IOBucket(57 vs 9). The rest looks pretty much stable. Now, if I understand correctly that means the last document caused 57 IOBuckets to be modified, but not necessarily created. I wonder if you used QueueCatalog, and if don't, what would it look like if you did. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: become a ZODB mentor in the Google Summer of Code!
On Tue, Mar 4, 2008 at 2:59 PM, Martijn Faassen <[EMAIL PROTECTED]> wrote: > Hey, > > Martijn Faassen wrote: > > It would be great if we put our community's secret gem, the ZODB, into > > the limelight more, and the Google Summer of Code would be a great > > opportunity. We need mentors, and fast, so if you want to mentor > > someone, please sign up in this wiki page here: > > > > http://wiki.zope.org/gsoc/SummerOfCode2008 > > No takers? I'm on the list already, but just want to let you guys know that I've found a victim^H^H^H^H^H volunteer for working on a ZODB project. I entered the five proposals from the ZODB blueprints that he was interested on (obviously he will only work on one!). I guess its to early to start casting votes? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] data loss bug with mounted storages, cut'n'paste and pack
On Feb 12, 2008 4:35 PM, Chris Withers <[EMAIL PROTECTED]> wrote: > Ideas as to how to prune the resulting POSKeyError greatfully received. Jim has acknowledge that this situation was *possible*, but I didn't follow up further with a reproducible test case when I reported it. There are two solutions: 1. Not packing mounted databases :) 2. Prodding Jim into fixing the packing code to take multi-databases into account -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
> Despite this change there are still a huge amount > of unexplained calls to the 'persistent_id' method of the ObjectWriter > in serialize.py. Why 'unexplained'? 'persistent_id' is called from the Pickler instance being used in ObjectWriter._dump(). It is called for each and every single object reachable from the main object, due to the way Pickler works (I believe). Maybe persistent_id can be analysed and optimized for the most common cases? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
I think someone proposed to have something just like a WAL in ZODB. That could be an interesting optimization. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage Interfaces
On 4/27/07, Jim Fulton <[EMAIL PROTECTED]> wrote: On Apr 27, 2007, at 10:21 AM, Stephan Richter wrote: ... > In the next month or so I will > probably need to implement a storage that encrypts its data, so I > will get a > taste of the new API then. I'll make sure to use the trunk. :-) I suspect that Stephan is talking about encrypted storage, and Jim is talking about encrypted transport? Of course you probably need both. :) -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Missing loader for multidatabase refs?
On 3/30/07, Jim Fulton <[EMAIL PROTECTED]> wrote: > What can be done to avoid that? By "that" I assume you mean dangling referenced. I really meant avoiding the PosKeyError. ;) You could create extra references in the source database. For example, if you make a cross-database reference, you could also add a corresponding reference in the source database. Right. And removing this corresponding difference when the object in the target database goes away. > Not packing the databases at all? That will work too. :) Not an option! > Would it be possible to make the loading of a cross-database reference > that is gone return some sort of 'BrokenObject' instead of a > PosKeyError, to work around the problem temporarily? Sure. I suppose the same could and perhaps should be done whenever you would get a POSKeyError. That sounds like a candidate for a ZODB proposal. > Specially if the databases happen to be on different ZEO servers > *wink*. That is a complication, yes. You would need some sort of distributed GC protocol. Ouch. :) >> - A non-GC pack that got rid of old records but didn't bother with >> GC. >>This would be advantagious for lots of folks independent of cross- >> database reference issues. > > Sounds like this would be the easiest way to solve the above issue? It is a fairly straightforward way, depending on the application. The application is named Zope 2. You might have heard about it. :) Unfortunately, it's probably non-trivial, as the FileStorage packing code is fairly intense. :) I've been wanting to redo this code for some time to reduce the amount of disk I/O. That would certainly be an opportunity to make GC optional. But maybe it wouldn't be too hard to disable GC in the current implementation -- I don't know. Let's see. GC right now is done when the object has 0 references pointing to it. A quick workaround would be making sure the code that computes the references (referencesf?) always returns at least one reference pointing to it? >> Cross database references are pretty transparent and automatic. >> >> Maybe there should be an option to make them less so. > > Yes, such an option would be great. Maybe. It depends on what semantics you want. In fact, I don't know how Zope 2 is setting up multi-databases. If you don't want cross- database references at all, you could change the startup code to open separate databases without making them part of a multi-database. Hm, brainstorming: I suppose there could be could be an option that says you can't have a cross reference unless the source database has a root key equal to the object id of the source object with the value being the source object. Then, an intentional cross reference would be made like this:: # Make a reference to foo, which is in another DB: foo._p_jar.root()[foo._p_oid] = foo self.x = foo The code that creates cross-database references would, if this option is available, do a check something like: if self.explicit_cross_database_references: if other._p_jar.root().get(other._p_oid) is not other: raise InvalidCrosDatabaseReference(...) That looks good. If not for solving the problem completely, at least for catching where in Zope implicit references might be created, and possibly fixing those places. I suspect you know what the patch is at this point -- at least to get the pack done. Perhaps you can try it and let us know how it goes. Packing worked fine, and packing the other database didn't generate a PosKeyError... yet. That makes me really nervous though. I will try to come up with a patch that creates a BrokenObject where a PosKeyError would occur in the case of multi databases. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Missing loader for multidatabase refs?
On 3/30/07, Jim Fulton <[EMAIL PROTECTED]> wrote: > Say there's two databases, and the > object in database '1' is the only one referencing some object in > database '2'. Since those references would be treated like weak > references, wouldn't the object in database '2' go away in a 'pack', > thus causing a PosKeyError when loading the object from database '1' > that still points to it? Yes. That is a limitation of cross-database references. What can be done to avoid that? Not packing the databases at all? Would it be possible to make the loading of a cross-database reference that is gone return some sort of 'BrokenObject' instead of a PosKeyError, to work around the problem temporarily? Note that treating the references differently in a pack can't help this as the references are in the referencing database, not the referenced database. It would be cool to have: - A multi-database pack that took multiple databases into account. Such a feature is doable, but obviously, non-trivial. Specially if the databases happen to be on different ZEO servers *wink*. - A non-GC pack that got rid of old records but didn't bother with GC. This would be advantagious for lots of folks independent of cross- database reference issues. Sounds like this would be the easiest way to solve the above issue? > Right. I'm wondering how I did end up with a cross-database reference. > Seems like a copy/paste through the ZMI caused it. My question is if > that's expected/correct. Cross database references are pretty transparent and automatic. Maybe there should be an option to make them less so. Yes, such an option would be great. >> What version of ZODB are you using? > > The one included with Zope 2.9.6. Hm, are you using ZEO? I'm wondering what sort of release would be needed to help you out. Yes, I'm using ZEO. I don't need any sort of release particularly, a patch would be fine. As long as there's a Zope 2.9.x release sometime in the future for the people that can't live with patches, it should be ok. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Missing loader for multidatabase refs?
On 3/30/07, Jim Fulton <[EMAIL PROTECTED]> wrote: > Now my question is, should 'referencesf' load those multi-database > references, or should they be treated just like the 'weak references', > by returning None, or should something else happen? They should be treated as weak references, at least for now. Would that cause any problem? Say there's two databases, and the object in database '1' is the only one referencing some object in database '2'. Since those references would be treated like weak references, wouldn't the object in database '2' go away in a 'pack', thus causing a PosKeyError when loading the object from database '1' that still points to it? > Or maybe that > reference shouldn't be there in the first place? Where? There references should certainly be in the data records, otherwise, cross-database references wouldn't work at all. Right. I'm wondering how I did end up with a cross-database reference. Seems like a copy/paste through the ZMI caused it. My question is if that's expected/correct. > This problem is preventing us from packing a database. It just falls > dead with the KeyError 'n' when packing. What version of ZODB are you using? The one included with Zope 2.9.6. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Missing loader for multidatabase refs?
While doing a 'pack' on a Zope instance that is using multiple databases, a KeyError 'n' happened. Upon investigation, we found out that the 'oid_loaders' dictionary used by 'referencesf' (in ZODB.serialize), which is in turn used by pack (presumably to find the oids referenced by an object), doesn't have a loader registered for 'n'. A reference_type of 'n' means: 'n' Multi-database simple object reference. The arguments consist of a databaase name, and an object id. The ObjectReader class defined a few lines above 'referencesf' *does* know how to load a multi-database simple object reference. Now my question is, should 'referencesf' load those multi-database references, or should they be treated just like the 'weak references', by returning None, or should something else happen? Or maybe that reference shouldn't be there in the first place? This problem is preventing us from packing a database. It just falls dead with the KeyError 'n' when packing. You can find the traceback below. 2007-03-29T19:41:38 ERROR ZEO.zrpc (2822) Error raised in delayed method Traceback (most recent call last): File "/usr/local/zope/fabricio/zserver296/lib/python/ZEO/StorageServer.py", line 1009, in run result = self._method(*self._args) File "/usr/local/zope/fabricio/zserver296/lib/python/ZEO/StorageServer.py", line 345, in _pack_impl self.storage.pack(time, referencesf) File "/usr/local/zope/fabricio/zserver296/lib/python/ZODB/FileStorage/FileStorage.py", line 1348, in pack opos = p.pack() File "/usr/local/zope/fabricio/zserver296/lib/python/ZODB/FileStorage/fspack.py", line 482, in pack self.gc.findReachable() File "/usr/local/zope/fabricio/zserver296/lib/python/ZODB/FileStorage/fspack.py", line 228, in findReachable self.findReachableAtPacktime([z64]) File "/usr/local/zope/fabricio/zserver296/lib/python/ZODB/FileStorage/fspack.py", line 304, in findReachableAtPacktime todo.extend(self.findrefs(pos)) File "/usr/local/zope/fabricio/zserver296/lib/python/ZODB/FileStorage/fspack.py", line 377, in findrefs return referencesf(self._file.read(dh.plen)) File "/usr/local/zope/fabricio/zserver296/lib/python/ZODB/serialize.py", line 647, in referencesf oid = oid_loaders[reference_type](*args) -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: PyPI-Entry
Not sure if there's anyone in charge of building Windows binaries. I could do that if you give me rights to upload to pypi. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables
| Feedback is welcome -- either in the Wiki or here. I've seen an informal version of this by Dieter on a mailing list somewhere (might even have been this list) ages ago and even thought that it was already implemented. I really like the proposal, +1 on it. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] What makes the ZODB slow?
On Mon, Jun 26, 2006 at 05:28:51PM -0400, Tim Peters wrote: | AFAIK, nobody anywhere has used this yet, outside of Python's test | suite. It was intended to be a simple, cheap approach to cutting | pickle bloat for apps motivated enough to set up the registry. You'll | note that half the one-byte codes are reserved for Zope :-) | | >You mean an optimization to make the pickle size for some | >new style classes smaller. That's not yet used because it could | >make the storage exchange between different Python versions impossible | >(the older Python versions would not understand the new pickle protocol). | | The need for protocol 2 is also why the extension registry (above) | can't be used so long as older Pythons are in the mix. Does that mean that if someone didn't care about older python's in the mix and were willing to register those shorter byte code extensions for it's own Zope that person would likely see great improvements in pickle size reduction, and that it would even improve ZEO transport by reducing the size of the data that is transferred? Maybe we should propose such changes to Zope 2.10/2.11 since it already requires Python 2.4 (since Zope 2.9). -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] tempstorage to ZEO?
On Wed, May 31, 2006 at 09:59:36AM -0400, Chris McDonough wrote: | | On May 31, 2006, at 9:55 AM, Sidnei da Silva wrote: | | >On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | >| I've done this (at least with FileStorage) and it's | >| sllo. Might be OK for low-traffic sites, but better | >| to implement a custom session data container that stores stuff in | >| SQL. I have the beginnings of one of these if you want to see it. | > | >Slow for using sessions? Why didn't you use tempstorage then? | | I think I thought it wouldn't have mattered. The difference between | using sessions against a local filestorage and one on a ZEO server | was something like 20X. So, since I couldn't believe the 20X figure, I wrote a very dumb test [1] that shows [2] tempstorage to be only slightly slower than filestorage (roughly 10%) and that adding zeo to the mix makes both of them about 4X slower, which would be pretty acceptable by my standards. I'm looking forward for testing tres' memcached stuff tomorrow. In the meantime, it would be great if someone can run the script on different boxes and platforms (I've ran it on Windows on a Intel Dual Core 3.0) to see if there's any difference. [1] http://awkly.org/files/zeo-bench/bench.py [2] http://awkly.org/files/zeo-bench/bench_results.txt -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] tempstorage to ZEO?
On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | I've done this (at least with FileStorage) and it's | sllo. Might be OK for low-traffic sites, but better | to implement a custom session data container that stores stuff in | SQL. I have the beginnings of one of these if you want to see it. Slow for using sessions? Why didn't you use tempstorage then? Or you are just asserting that FileStorage over ZEO is noticeably slower than directly? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] tempstorage to ZEO?
On Wed, May 31, 2006 at 09:40:01AM -0400, Chris McDonough wrote: | Should be OK, although if you're doing it because you want a "fast" | storage (or one that "produces fewer conflicts"), ZEO takes away much | hope of that intrinsically. My goal was actually sharing sessions between zeo clients, without resorting to zsqlsessions or similar. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] tempstorage to ZEO?
On Wed, May 31, 2006 at 09:53:14AM +0200, Tino Wildenhain wrote: | Sidnei da Silva schrieb: | >I was looking, and trying to understand why 'tempstorage' is not | >included in the ZODB. | > | >Since some people might want to share sessions between ZEO clients, | >and that 'tempstorage' is what is mounted as | >'/temp_folder/session_data', why not make that easy to mount via ZEO? | >Does anyone see a problem with this? | | actually it is easy mountable via ZEO. I'm doing this already... | Do you mean including the schema for zeo.conf per default? Yeah, that too. I was actually thinking about the standalone ZODB package. Or does that not exist anymore? I was also wondering if this is the right thing to do (mounting tempstorage via ZEO) since I don't recall seeing any tutorial that recommended it. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] tempstorage to ZEO?
I was looking, and trying to understand why 'tempstorage' is not included in the ZODB. Since some people might want to share sessions between ZEO clients, and that 'tempstorage' is what is mounted as '/temp_folder/session_data', why not make that easy to mount via ZEO? Does anyone see a problem with this? What I'm proposing is to move 'tempstorage' to 'ZODB', since it has no dependency on anything other than 'zLOG'. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Resource Managers not getting collected?
On Sat, May 27, 2006 at 07:56:30PM +0200, Dieter Maurer wrote: | Apparently, our resource manager instances are released. | At least, we have not observed any serious database | connection leaks... | | Sorry, for not being able to write as you had expected... No problem. Worth a try anyway. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Resource Managers not getting collected?
On Thu, May 25, 2006 at 02:41:32AM -0300, Sidnei da Silva wrote: | Anyone experienced this? Any clues about what to do to get more | information? /me still expecting Dieter Maurer's "I've fixed this in my private copy of Zope by doing X, Y, Z" :) -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Resource Managers not getting collected?
I'm working with Mark Hammond on an experimental Database Adapter which just like all the other Database Adapters for Zope 2 does subclass the infamous 'Shared.DC.ZRDB.TM'. An instance of the said subclass gets registered with the transaction by means of a MultiObjectResourceAdapter. What Mark noticed, and what puzzled me, was that instances of the said class were not being garbage collected, which means that if they hold a reference to a rdb connection the rdb connection never gets closed. Now, this sounds pretty serious to me, and after digging as much as we could we couldn't find any obvious reference cycles that needed explicit breaking or anything. Anyone experienced this? Any clues about what to do to get more information? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Really want to refactor ZEO's networking archiecture
On Fri, May 05, 2006 at 10:32:12AM -0400, Jim Fulton wrote: | Sidnei da Silva wrote: | ... | >Oh, if I'm allowed to speak *wink*, it would be great if in this | >refactoring 'zrpc' could be made to work with less dependencies. | > | >I'm using 'zrpc' on a project and it was a pain to implement a | >minimally working application because it depended on a few convoluted | >steps to setup, some of which could be trivially removed. | > | >One way or another, I think Jim's refactoring would simplify this, | >even if it's not a explicitly stated goal. | | I'm a bit curious why you are using zrpc rather than one of the many | Python RPC mechanisms. I never intended having something that was used | outside of ZEO and I'm not interested in satisfying non-ZEO use cases. | That's not to say I want to prevent you from using it, I'm just curious | why you are. ZEO uses it because it is so small, although maybe it's | not so small that ZEO shouldn't use something else. | | If something isn't an explicitly stated goal, it's not likely | to happen. :) I just wanted a simple, transport-efficient, 'binary' (as opposed to text), transparent RPC mechanism. Since zrpc uses pickle for transport, that did fit the job at the time. Not to say that I didn't want to use twisted or some other RPC mechanism, only that zrpc turned out to be the one I was more familiar with, and the one that was quickiest for me to put into production. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Really want to refactor ZEO's networking archiecture
On Fri, May 05, 2006 at 07:25:50AM -0400, Jim Fulton wrote: | >IMHO there is no downside. People would be "heavily dependent" if they | >e.g. had programs directly interacting with ZEO internals, or | >subclassing ZEO components. I would bet the number of people who are in | >that position can be counted on two hands. | | Or maybe flippers. ;) | | > What everyone does is *use* | >ZEO, and maybe script it a little bit. | | And the scripting is external use too. | | > If the internals change it | >doesn't matter much, as long as the package as a whole fulfills the | >same need that the current ZEO software does. | | It shouldn't matter at all, except to provide new opportunities for the | future. Oh, if I'm allowed to speak *wink*, it would be great if in this refactoring 'zrpc' could be made to work with less dependencies. I'm using 'zrpc' on a project and it was a pain to implement a minimally working application because it depended on a few convoluted steps to setup, some of which could be trivially removed. One way or another, I think Jim's refactoring would simplify this, even if it's not a explicitly stated goal. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Corrupted Data Error?
On Tue, Apr 25, 2006 at 10:22:25AM -0300, Sidnei da Silva wrote: | Any hint to what can be done to recover from this situation? It's a bad disk block situation. No blame on Zope. Getting a new disk. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Phone +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Corrupted Data Error?
Got the following exception while doing some work on a Zope instance here. It's the first time I see such error. * Module ZODB.Connection, line 704, in setstate * Module ZODB.Connection, line 760, in _setstate * Module ZODB.serialize, line 495, in setGhostState * Module ZODB.serialize, line 488, in getState * Module ZODB.serialize, line 436, in _persistent_load * Module ZODB.Connection, line 207, in get * Module ZEO.ClientStorage, line 746, in load * Module ZEO.ClientStorage, line 769, in loadEx * Module ZEO.ServerStub, line 192, in loadEx * Module ZEO.zrpc.connection, line 536, in call CorruptedDataError: Error reading oid 0x0178c2. Found '\x00\x00\x00\x00\x00\x01x\xc2\x03c\xfbq\xd4y\xf3\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xa0\x0c\xcf\x00\x00\x00\x00\x00\x00\x00\x00' at 94379992 Any hint to what can be done to recover from this situation? -- Sidnei da Silva Enfold Systems, Inc. http://enfoldsystems.com Phone +1 713 942 2377 Ext 214 Fax +1 832 201 8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Extenuated memory
On Fri, Mar 24, 2006 at 02:20:06PM +0100, Antonio Beamud Montero wrote: | > | I think I will remove ZODB from my server and use a SQL Object mapper | > | like SQLObject to see if I can get an stable server, or the dbapi | > | directly.. my experience with ZODB and threads has been a nightmare :( | > | | > | A lot of thanks for your help, but this server is in a production/stable | > | system, and use unstable systems give me fear... (what's the next | > | problem?) | > | > Note that SQLObject has it's internal 'cache' of objects, and if you | > don't handle that carefully you might end up with memory bloat just as well! | > | | Ok, first I'll make stress tests to check that, and with the raw | dbapi... if this problems continues I'll change to other | language... :-( (I've been using python for last 6 years, but with this | problems I cannot continue using it...) Don't let yourself be turned down so quickly. All languages and all their tools have their 'problems'. If you don't understand how they work or what they do you will never be able to write a system that performs well and has good memory management. No language or tool will 'magically' give you that, no matter what others say *wink*. -- Sidnei da Silva Enfold Systems, Inc. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Extenuated memory
On Fri, Mar 24, 2006 at 11:12:19AM +0100, Antonio Beamud Montero wrote: | El jue, 23-03-2006 a las 15:02 -0500, Tim Peters escribió: | > [Antonio Beamud Montero] | > > But my server doesn't free any memory, now it uses 55Mb of resident | > > size... How I can debug this "references"? | > | > Many layers are involved in memory management, from ZODB through | > Python down to your platform C's malloc/free implementation and ending | > in your OS kernel Unless you become knowledgable about all of them | > (and vital details vary wildly across specific platform C and OS | > versions), chances are decent you'll never find a correct answer. | | I think I will remove ZODB from my server and use a SQL Object mapper | like SQLObject to see if I can get an stable server, or the dbapi | directly.. my experience with ZODB and threads has been a nightmare :( | | A lot of thanks for your help, but this server is in a production/stable | system, and use unstable systems give me fear... (what's the next | problem?) Note that SQLObject has it's internal 'cache' of objects, and if you don't handle that carefully you might end up with memory bloat just as well! -- Sidnei da Silva Enfold Systems, Inc. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Fri, Mar 03, 2006 at 10:20:13AM -0500, Chris McDonough wrote: | On Mar 3, 2006, at 10:13 AM, Sidnei da Silva wrote: | | >| Well, yes. I've already done that. But it's a hack. The runtime | >| dependencies aren't a problem, it's the build-time dependencies. I | >| don't know of a way to say "use this header file from this egg to | >| build this other egg" in an egg setup script. | > | >Maybe it's a missing feature. I would ask Phillip Eby. | | Maybe it is, but in this case, is it worth it? Does anyone want to | use BTrees outside of a persistent hierarchy? IMO, you have to ask the opposite question. Does anyone want to use persistent without using BTrees? Then the answer is yes. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Fri, Mar 03, 2006 at 09:52:43AM -0500, Chris McDonough wrote: | | On Mar 3, 2006, at 8:08 AM, Sidnei da Silva wrote: | | >On Thu, Mar 02, 2006 at 10:03:48PM -0500, Chris McDonough wrote: | >| I'm taking a stab at packaging the various pieces of ZODB as eggs. | >| One of the things I'd like to do is to separate the packages of ZODB | >| that are currently shipped together (BTrees, ZODB/ZEO, persistent | >| transaction, ZConfig) into separate packages, because each may | >| arguably be useful outside of what we call "ZODB" now. | >| | >| One of the issues I've run into is that there are compile-time | >| dependencies between packages mainly due to C header files. For | >| example, the BTrees package assumes that it will be able to find the | >| cPersistence.h file, which currently ships with the "persistent" | >| package. I wonder if I should bother breaking the persistent and | >| BTrees packages apart as separate eggs. I suspect not. | > | >Can't you make them separate eggs that depend on the other? | | Well, yes. I've already done that. But it's a hack. The runtime | dependencies aren't a problem, it's the build-time dependencies. I | don't know of a way to say "use this header file from this egg to | build this other egg" in an egg setup script. Maybe it's a missing feature. I would ask Phillip Eby. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Thu, Mar 02, 2006 at 10:03:48PM -0500, Chris McDonough wrote: | I'm taking a stab at packaging the various pieces of ZODB as eggs. | One of the things I'd like to do is to separate the packages of ZODB | that are currently shipped together (BTrees, ZODB/ZEO, persistent | transaction, ZConfig) into separate packages, because each may | arguably be useful outside of what we call "ZODB" now. | | One of the issues I've run into is that there are compile-time | dependencies between packages mainly due to C header files. For | example, the BTrees package assumes that it will be able to find the | cPersistence.h file, which currently ships with the "persistent" | package. I wonder if I should bother breaking the persistent and | BTrees packages apart as separate eggs. I suspect not. Can't you make them separate eggs that depend on the other? -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Conflict Error that won't go away?
On Thu, Mar 02, 2006 at 02:28:50PM -0300, Sidnei da Silva wrote: | Hi there, | | I'm facing an issue with a ConflictError that won't go away even after | restarting. Looks like the ZODB got screwed somehow. Here's the | traceback: False alarm. The zeo cache was just in an unusual place so I didn't spot it. After removing the cache files all systems back to normal. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Conflict Error that won't go away?
On Thu, Mar 02, 2006 at 09:37:02AM -0800, Dennis Allison wrote: | | We've seen this problem occasionally. Usually deleting the ZEO cache | clears the problem. Except it didn't seem to resolve the problem in this case, that's why I'm confused. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Conflict Error that won't go away?
Hi there, I'm facing an issue with a ConflictError that won't go away even after restarting. Looks like the ZODB got screwed somehow. Here's the traceback: * Module Zope.App.startup, line 163, in zpublisher_exception_hook * Module ZPublisher.Publish, line 107, in publish * Module Zope.App.startup, line 222, in commit * Module ZODB.Transaction, line 252, in commit * Module ZODB.Connection, line 738, in tpc_vote * Module ZEO.ClientStorage, line 849, in tpc_vote * Module ZEO.ClientStorage, line 833, in _check_serials ConflictError: database conflict error (oid 0xd40acd, serial this txn started with 0x0363a06b42bc2499 2006-02-24 15:39:15.641000, serial currently committed 0x0363b6ce866bdc88 2006-02-28 15:10:31.505000) The system where this is happening is running a ZEO Server and a single ZEO Client. At some point, near the date mentioned in the traceback, the system ran out of disk space during a large transaction. Could this have affected the database? What are the steps to recover from it? Maybe truncate the database? What can be done to avoid it in the future? -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persistent ZEO Cache corruption?
On Thu, Jan 12, 2006 at 10:17:54AM -0500, Tim Peters wrote:
| [Sidnei da Silva]
| >> Every now and then I face a corruption of the persistent zeo cache, but
| >> this is the first time I get this variant.
|
| What other variants do you see?
Can't remember right now, it was quite some time ago and involved
making changes to one zeo client while the other one was down using
'zopectl debug'. Seen it about 6 times in different environments, so
should be reproduceable.
| >> The cause is very likely to be a forced shutdown of the box this zope
| >> instance was running on, but I thought it would be nice to report the
| >> issue.
|
| Yes it is! Thank you. It would be better to open a bug report ;-).
Sure will.
| >> Here's the traceback::
| >>
| >> File "/home/sidnei/src/zope/28five/lib/python/ZEO/ClientStorage.py", line
| 314, in __init__
| >> self._cache.open()
| >> File "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line 112, in
| open
| >>self.fc.scan(self.install) File
| >> "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line 835, in scan
| >>install(self.f, ent) File
| >> "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line 121, in
| install
| >> o = Object.fromFile(f, ent.key, skip_data=True)
| >> File "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line 630, in
| fromFile
| >> raise ValueError("corrupted record, oid")
| >> ValueError: corrupted record, oid
| >>
| >> I have a copy of the zeo cache file if anyone is interested.
|
| Attaching a compressed copy to the bug report would be best (if it's too big
| for that, or it's proprietary, let me know how to get it and I'll put it on
| an internal ZC machine). Can't tell in advance whether that will reveal
| something useful, though (see below).
Don't think there might be anything sensitive in there, maybe my blog
password in the worst case *wink*. Here's the files (zeo1-1.zec is
probably the one you're after):
http://awkly.org/files/zeo-cache.tar.bz2
| > It seems as though persistent caches haven't been a very sucessful
| > feature. Perhaps we should abandon them.
|
| They do seem to be implicated in more than their share of problems, both
| before and after MVCC.
|
| The post-MVCC ZEO persistent cache _intends_ to call flush() after each file
| change. If it's missing one of those, and depending on what "forced
| shutdown" means exactly, that could be a systematic cause for corruption.
| It doesn't call fsync() unless it's explicitly closed cleanly, but it's
| unclear what good fsync() actually does across platforms when flush() is
| called routinely and the power stays on.
Oh, I really meant to say "accidental shutdown", though I wasn't around
when the box restarted it looks like it was a power failure.
--
Sidnei da Silva
Enfold Systems, LLC.
http://enfoldsystems.com
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Persistent ZEO Cache corruption?
Every now and then I face a corruption of the persistent zeo cache,
but this is the first time I get this variant.
The cause is very likely to be a forced shutdown of the box this zope
instance was running on, but I thought it would be nice to report the
issue.
Here's the traceback::
File "/home/sidnei/src/zope/28five/lib/python/ZEO/ClientStorage.py",
line 314, in __init__
self._cache.open()
File "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line
112, in openself.fc.scan(self.install)
File "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line
835, in scaninstall(self.f, ent)
File "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line
121, in install
o = Object.fromFile(f, ent.key, skip_data=True)
File "/home/sidnei/src/zope/28five/lib/python/ZEO/cache.py", line
630, in fromFile
raise ValueError("corrupted record, oid")
ValueError: corrupted record, oid
I have a copy of the zeo cache file if anyone is interested.
What is bad about this problem is that it prevented Zope from starting
and there is no obvious clue that removing the persistent zeo cache
would cure it, though that's what anyone that has a clue about what
he's doing would do *wink*.
--
Sidnei da Silva
Enfold Systems, LLC.
http://enfoldsystems.com
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Savepoints and Shared.DC.ZRDB.TM
On Tue, Dec 20, 2005 at 11:51:16PM -0500, Tim Peters wrote: | [Tim Peters] | >> It's the traceback that's needed here. You're working too hard ;-) | | [Sidnei da Silva] | > Indeed. | > | > File "C:\Arquivos de programas\Enfold | Server\Products\ATContentTypes\tests\atcttestcase.py", line 258, in | test_migrationKeepsPermissions | > transaction.savepoint() # subtransaction | | Bingo! That's the root cause of it all. This isn't Zope code, so I can't | know what it's trying to accomplish. _Why_ is this code making a savepoint? | | Since it throws away the result of the savepoint() call, it's not possible | to roll back to this savepoint's state later. So the most likely reason is | that it's just "trying to save memory". In that case, change the code to | | transaction.savepoint(optimistic=True) | | and your original problem will go away. I do understand that. But that doesn't answer the original question, which was: - Should Shared.DC.ZRDB.TM support (non-optimistic) savepoints? >From what I understand, if the answer is yes then that class should be updated to use the new datamanager API instead of the old 'register' call, otherwise it just won't support non-optimistic savepoints at all. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Savepoints and Shared.DC.ZRDB.TM
On Tue, Dec 20, 2005 at 09:28:35PM -0500, Tim Peters wrote:
| It's the traceback that's needed here. You're working too hard ;-)
Indeed.
File "C:\Arquivos de programas\Enfold
Server\Products\ATContentTypes\tests\atcttestcase.py", line 258, in
test_migrationKeepsPermissions
transaction.savepoint() # subtransaction
File "C:\Arquivos de programas\Enfold
Server\Zope\lib\python\transaction\_manager.py", line 90, in savepoint
return self.get().savepoint(optimistic)
File "C:\Arquivos de programas\Enfold
Server\Zope\lib\python\transaction\_transaction.py", line 297, in savepoint
self._saveCommitishError() # reraises!
File "C:\Arquivos de programas\Enfold
Server\Zope\lib\python\transaction\_transaction.py", line 294, in savepoint
savepoint = Savepoint(self, optimistic, *self._resources)
File "C:\Arquivos de programas\Enfold
Server\Zope\lib\python\transaction\_transaction.py", line 654, in __init__
raise TypeError("Savepoints unsupported", datamanager)
TypeError: ('Savepoints unsupported', at 62064976>)
--
Sidnei da Silva
Enfold Systems, LLC.
http://enfoldsystems.com
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Savepoints and Shared.DC.ZRDB.TM
On Tue, Dec 20, 2005 at 09:01:08PM -0500, Tim Peters wrote:
| [Sidnei da Silva]
| > I have a object that subclasses from Shared.DC.ZRDB.TM and it's bombing
| > with a 'TypeError: Savepoints unsupported' with Zope 2.8.
|
| Please find a full traceback and share it.
|
| > Should the Shared.DC.ZRDB.TM implement savepoints as a 'noop' maybe?
|
| No -- "lying" would make matters worse.
|
| > Note that some (all?) RDB adapters might depend on this class still.
|
| Whether that matters largely depends on the traceback we haven't seen yet
| ;-)
Happens on line 632 of a current Zope 2.8 checkout:
if not optimistic:
raise TypeError("Savepoints unsupported", datamanager)
'datamanager' is a MultiObjectResourceAdapter for a subclass of
Shared.DC.ZRDB.TM that was registered using the 'register' method of
the Transaction class.
--
Sidnei da Silva
Enfold Systems, LLC.
http://enfoldsystems.com
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Savepoints and Shared.DC.ZRDB.TM
I have a object that subclasses from Shared.DC.ZRDB.TM and it's bombing with a 'TypeError: Savepoints unsupported' with Zope 2.8. Should the Shared.DC.ZRDB.TM implement savepoints as a 'noop' maybe? Note that some (all?) RDB adapters might depend on this class still. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [Zope-dev] Re: [ZODB-Dev] Test Failures
On Fri, Dec 16, 2005 at 03:01:59PM -0200, Sidnei da Silva wrote: | But, why only the 2.8 tests would fail then? I'll try a 'make clean' | before running the tests and see if it helps. That did indeed help, sorry for the noise. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Test Failures
| No idea why it failed for you. The only thing that rings a bell here is | that this test was added in ZODB 3.4.2b1, corresponding to this ZODB news | entry: | | """ | BTrees | -- | | - (3.4.2b1) Collector 1873. It wasn't possible to construct a BTree | or Bucket from, or apply their update() methods to, a PersistentMapping | or PersistentDict. This works now. | """ | | So my only guesses are that you have some older (than 3.4.2) version of ZODB | on your PYTHONPATH, or that your checkout is screwed up. Here are the | externals you _should_ have: But, why only the 2.8 tests would fail then? I'll try a 'make clean' before running the tests and see if it helps. | [EMAIL PROTECTED]:~/Zope2.8$ svn propget svn:externals lib/python | zope | svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.1-Zope-2.8/src/zope | ZConfigsvn://svn.zope.org/repos/main/ZConfig/tags/ZConfig-2.3 | BTrees svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/BTrees | Persistencesvn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/Persistence | persistent svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/persistent | ThreadedAsync | svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ThreadedAsync | transactionsvn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/transaction | ZEOsvn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ZEO | ZODB svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ZODB | ZopeUndo svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ZopeUndo | zdaemonsvn://svn.zope.org/repos/main/zdaemon/tags/zdaemon-1.1 Looks good to me: [EMAIL PROTECTED]:~/src/zope/2.8$ svn propget svn:externals lib/python zope svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.1-Zope-2.8/src/zope ZConfigsvn://svn.zope.org/repos/main/ZConfig/tags/ZConfig-2.3 BTrees svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/BTrees Persistencesvn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/Persistence persistent svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/persistent ThreadedAsync svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ThreadedAsync transactionsvn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/transaction ZEOsvn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ZEO ZODB svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ZODB ZopeUndo svn://svn.zope.org/repos/main/ZODB/tags/3.4.2/src/ZopeUndo zdaemonsvn://svn.zope.org/repos/main/zdaemon/tags/zdaemon-1.1 | | > Additionally, while running the tests for Zope 2.8, 2.9 and trunk (all | > at the same time *wink*), the following tests failed. I suspect the | > failures have to do with heavy load on the box while running the tests? | | Almost certainly, yes. The Transience tests are part of Zope (not part of | ZODB), and it's in the nature of transient objects that they "go away by | magic" as time passes. If "too much" wall-clock time elapses while a | Transience test is running, failure is expected. Running all the tests takes 1864 seconds, so I can see it taking too long :) -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Test Failures
I've seen the following tests fail today, after updating Zope 2.8 branch for all variants of BTrees: == ERROR: testUpdateFromPersistentMapping (BTrees.tests.testBTrees.IIBucketTest) -- Traceback (most recent call last): File "/usr/lib/python2.4/unittest.py", line 260, in run testMethod() File "/home/sidnei/src/zope/2.8/lib/python/BTrees/tests/testBTrees.py", line 353, in testUpdateFromPersistentMapping self.t.update(pm) TypeError: Sequence must contain 2-item tuples This is on a Powerbook running Ubuntu Breezy PPC. Python 2.4.2 (#2, Nov 20 2005, 17:20:59) [GCC 4.0.3 20051023 (prerelease) (Ubuntu 4.0.2-3ubuntu1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Additionally, while running the tests for Zope 2.8, 2.9 and trunk (all at the same time *wink*), the following tests failed. I suspect the failures have to do with heavy load on the box while running the tests? Zope 2.8: == FAIL: testPathologicalLeftBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObjectContainer) -- == FAIL: testPathologicalRightBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObjectContainer) -- Zope 2.9: Error in test testPathologicalLeftBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObjectContainer) Failure in test testPathologicalRightBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObjectContainer) Failure in test testRandomNonOverlappingInserts (Products.Transience.tests.testTransientObjectContainer.TestTransientObjectContainer) Zope trunk: Failure in test testPathologicalLeftBranching (Products.Transience.tests.testTransientObjectContainer.TestTransientObjectContainer) -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Close callback failure
On Fri, Nov 25, 2005 at 01:50:51PM +, Chris Withers wrote: | Sidnei da Silva wrote: | >On Fri, Nov 25, 2005 at 01:22:31PM +, Chris Withers wrote: | >| Sidnei da Silva wrote: | >| >| Interesting, I thought we had killed off PTS, since we don't use it, | >but | >| it may still be there, so I'll take a look... | >| > | >| >FWIW, that bug has been fixed. | >| | >| In what version of Plone? | > | >http://plone.org/products/pts/releases/1.2.4 | | Is that in any released verion of Plone? I think Plone 2.1.1 had 1.2.3, which had a small problem with IOError being raised if a translation was gone or something but it already had the ConnectionStateError fixed according to the change log. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Close callback failure
On Fri, Nov 25, 2005 at 01:22:31PM +, Chris Withers wrote: | Sidnei da Silva wrote: | >| Interesting, I thought we had killed off PTS, since we don't use it, but | >| it may still be there, so I'll take a look... | > | >FWIW, that bug has been fixed. | | In what version of Plone? http://plone.org/products/pts/releases/1.2.4 -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Close callback failure
On Fri, Nov 25, 2005 at 10:05:31AM +, Chris Withers wrote: | Dieter Maurer wrote: | >Seb Bacon wrote at 2005-11-23 17:33 +: | > | >>2005-11-21T16:38:07 ERROR(200) ZODB Shouldn't load state for 0x4723d3 | >>when the connection is closed | > | >A persistency bug. | > | > I know that (e.g.) PlacelessTranslationService | > can cause such exceptions. It caches a persistent object (the service) | > at module level and accesses it from all workers. | > However, a persistent object must only be accessed from a thread | > that has opened its ZODB connection and not by other threads. | > When this restriction is violated and the object has been deactivated, | > the exception occurs. | | Interesting, I thought we had killed off PTS, since we don't use it, but | it may still be there, so I'll take a look... FWIW, that bug has been fixed. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Zope 2.8.1 w Zeo 3.4.1 fails where standalone Zope 2.8.1works
On Thu, Sep 22, 2005 at 09:22:01AM +0200, Erik Lundh wrote: | Thank you for your comments! | It is good to know that I am not wasting my time hunting ghosts in my setup. | I must have some flaw in my new OS environment or way of setting up Zope. | | The only packages that gets installed with the script in the original post | is Plone 2.1 and ZMySQLDA 2.08 | | I usually get poskeyerrors just by visiting the ZMI control panel. If I try | to add a Plone 2.1 instance, that's an instant transation failure | | The same operations works fine when I switch zope.conf to standalone Zope | configuration and restart. | | I never get far enough to install my own products in the zeo setup. | | I'll continue to dig... Hint: If you are using persistent (disk-based) ZEO cache, removing the cache file and/or disabling persistent caching. I've had several issues with that, but never been able to get a consistent case to reproduce. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zrpc marshall hidden issue
On Thu, Sep 15, 2005 at 09:15:16PM -0400, Tim Peters wrote: | [Sidnei da Silva] | > We've hit an error that seems to be related to some code that probably | > has no tests coverage. | > | > When a marshalling error occurs, we get the following traceback, which | > seems to be trying to reference a '.errors' attribute of a Marshaller | > instance that doesn't seem to exist: | | If this is the same as Tres's Thursday bug report: | | http://www.zope.org/Collectors/Zope/1900 | | please put follow-ups there. Else, if this isn't related to Tres's report, | say more, because I'm assuming it is ;-) It is. Tres beat me to reporting it. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zrpc marshall hidden issue
Hello, We've hit an error that seems to be related to some code that probably has no tests coverage. When a marshalling error occurs, we get the following traceback, which seems to be trying to reference a '.errors' attribute of a Marshaller instance that doesn't seem to exist: Traceback (most recent call last): File "/usr/lib/python2.3/asyncore.py", line 69, in read obj.handle_read_event() File "/usr/lib/python2.3/asyncore.py", line 390, in handle_read_event self.handle_read() File "/home/runyaga/work/projects/entransit/src/ZEO/zrpc/smac.py", line 219, in handle_read self.message_input(msg) File "/home/runyaga/work/projects/entransit/src/ZEO/zrpc/connection.py", line 396, in message_input self.handle_request(msgid, flags, name, args) File "/home/runyaga/work/projects/entransit/src/ZEO/zrpc/connection.py", line 430, in handle_request return self.return_error(msgid, flags, *error) File "/home/runyaga/work/projects/entransit/src/ZEO/zrpc/connection.py", line 485, in return_error except self.marshal.errors: AttributeError: Marshaller instance has no attribute 'errors' We are using: ZEOsvn://svn.zope.org/repos/main/ZODB/tags/3.4.1/src/ZEO ZODB svn://svn.zope.org/repos/main/ZODB/tags/3.4.1/src/ZODB -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On Fri, Sep 02, 2005 at 08:55:28PM +0200, Dieter Maurer wrote: | Sidnei da Silva wrote at 2005-9-1 17:13 -0300: | > ... | >Yes, he's the one in charge of registering the subscribers so they are | >called in the order he wants them to. | > | >So: | > | > event(IBeforeSubsystem, SubSystem) | > execute SubSystem | > event(IAfterSubsystem, Subsystem) | > | >Back to my example, 'B' would subscribe to 'event(IAfterSubsystem, | >A)', and 'AB' could subscribe to 'event(IBeforeSubsystem, B)'. | | You mean the integrator would subscribe "B" to "event(after, A)" | and "AB" to "event(before, B)"? | | That is near what I am looking for (the individual systems do not | know about each other -- only the integrator does). Yes. | However, I find it more natural to use numbers on a global | scale to coordinate the subsystems than explicit tighing systems | on one another. Can you give a simple example? -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On Thu, Sep 01, 2005 at 08:34:11PM +0200, Dieter Maurer wrote: | Sidnei da Silva wrote at 2005-8-31 23:42 -0300: | > ... | >I would instead fire 'before' and 'after' events for each of those | >systems, and a system that wants to explicitly run after/before | >another system would do so as a subscriber to the explicit event | >fired. | | But the "systems" do not know about one another... That's right, and they shouldn't need to. In my example, 'A' doesn't need to know about 'B', and none of them need to know about 'AB'. | Only the integrator knows what "systems" are in its application. Yes, he's the one in charge of registering the subscribers so they are called in the order he wants them to. So: event(IBeforeSubsystem, SubSystem) execute SubSystem event(IAfterSubsystem, Subsystem) Back to my example, 'B' would subscribe to 'event(IAfterSubsystem, A)', and 'AB' could subscribe to 'event(IBeforeSubsystem, B)'. That would for sure satisfy any need *I* can think of. | All a single system can say: I want some callbacks | registered -- in the following partial order (among my callbacks). | | There may be ordering restrictions with the callbacks of | other (unknown) "systems". That depends on what my and the other callback do. Don't quite get what you meant to say here, but I have a feeling it's not much different from what Jim proposes, and would imply that in my example SubSystem 'A' would need to know about SubSystem 'B'. Is that what you meant? -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On Wed, Aug 31, 2005 at 06:02:21PM -0400, Jim Fulton wrote: | You might be right, but the only credible use case I've seen | is to go last. The example Julien has something going first | and something going last. Looking at the code, I don't see | why the thing he wanted to go first needed to do so. | The example you gave above had subscribers that wanted to go last. | What evidence to I have of other orderings? Julien did give | a "future need" to order two specific systems. IMO, having | one system implement that requirement explicitly by invoking | the other would be much clearer than having the | requirement implemented implicitly through integers orders. I would instead fire 'before' and 'after' events for each of those systems, and a system that wants to explicitly run after/before another system would do so as a subscriber to the explicit event fired. Otherwise, if system 'A' is hardcoded to call system 'B' and you want to add a system 'AB' that runs between 'A' and 'B' them you have to resort to code modification or monkey patching, which brings us back to memories of Zope 2. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Reloading product in 2.8
On Fri, Jul 15, 2005 at 07:49:10PM +0200, Dieter Maurer wrote: | But, it does not need to fail -- provided that with a product | all its dependent products are refreshed as well. | | At least, I never saw it failing when I had met this condition... | | And I lost time only when I saw the not very helpful problem messages | for the first few times (when I did not met this condition). | | But otherwise, refresh saved me *lots* of time... Same here. If you know the gotchas and you program properly it can work flawleslly. Hell, I've even been able to refresh Archetypes products, what can be more prone to fail than that? -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Reloading product in 2.8
On Thu, Jul 14, 2005 at 06:01:26PM -0400, Jim Fulton wrote: | When it fails, it does so in subtle ways that cause people | to lose lots of time. Usually when it fails I just restart. I wouldn't consider that losing lots of time comparing restarting now and then to restarting constantly. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Try and fail connect strategy for zeo/zrpc
| Does't the *already included* zeoup.py do exactly what you are trying | to do? See the bin directory of your zope home. Unfortunately no. It uses ClientStorage, which goes through the 1000's of lines of the connection dance using connect threads and large timeouts. Effectively, ClientStorage will retry a connection 'forever'. I just want to see if the port is listening and it smells like ZEO with the minimal reasonable timeout. -- Sidnei da Silva Enfold Systems, LLC. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Try and fail connect strategy for zeo/zrpc
On Mon, Jun 27, 2005 at 02:38:48PM -0400, Tim Peters wrote:
| > Any suggestions about how to best implement this,
|
| Maybe just create your own socket, attempt to connect, and if that attempt
| fails produce the kind of error message you want to see? Of course if it
| succeeds, there's some chance it will fail if you immediately close the
| socket and do a "normal" connection attempt.
That is good enough for me yes. Attached the script I've came up with,
which is far from perfect but has just the amount of functionality I
needed. Maybe we should include something similar to that with ZEO
itself?
--
Sidnei da Silva
Enfold Systems, LLC.
#!/usr/bin/python
#$Id: zeo_ping.py 2338 2005-07-02 00:39:31Z sidnei $
# Licensed under the Enfold Enterprise Server License Agreement
# Copyright(c), 2004-5, Enfold Systems, LLC, ALL RIGHTS RESERVED
import re
import socket
SERVER_PROTOCOL = re.compile('Z\d{3}$')
def zeo_ping(host, port, timeout=5, reraise=False,
handshake=False):
"""Ping a supposed ZEO server and see if it's up and running.
host
hostname or ip to connect to
port
port to use for connection
timeout
maximum time in seconds to wait for a connection
reraise
if a connection error occurs, re-raise the exception or not
handshake
try a zeo handshake
"""
port = int(port)
timeout = float(timeout)
reraise = bool(reraise)
handshake = bool(handshake)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(timeout)
try:
try:
s.connect((host, port))
except socket.error:
if reraise:
raise
return False
else:
if not handshake:
return True
# See if it smells like ZEO
try:
data = s.recv(1024)
except socket.error:
if reraise:
raise
return False
else:
return bool(SERVER_PROTOCOL.search(data))
finally:
s.close()
return True
if __name__ == '__main__':
import sys
print zeo_ping(*sys.argv[1:])
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Try and fail connect strategy for zeo/zrpc
We have a couple custom applications that use of zeo or just zrpc that would need a different connect strategy than what's provided as default. To be more specific, we need a way to try a connection and either connect or fail immediately with a reasonably short timeout and if possible a useful error message. I have tried to understand the woes involved in establishing a zeo/zrpc connection and what comes closer to what we want to implement is providing the 'wait' argument to ClientStorage, but even then it would at least fire off a thread and retry a number of times (forever?) before giving up the connection attempt. Any suggestions about how to best implement this, and would such code be accepted for inclusion in a latter version of ZEO? -- Sidnei da Silva Enfold Systems, LLC. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] TmpStore missing loadBefore
On Sun, Jun 26, 2005 at 09:43:43PM -0400, Tim Peters wrote: | I'll make an internal 3.4.1a1 release and stitch that into Zope 2.8 branch | and Zope trunk tomorrow. "Internal" means I skip most of the steps in a | full-blown release (will not create a ZODB tarball or Windows installers, | will not register it with PyPI, will not send a release announcement, ...). | I won't make any serious move toward releasing 3.4.1 before I detect serious | work elsewhere toward releasing Zope 2.8.1 or Zope 2.9 (ZODB releases are | driven by Zope releases). I would expect a 2.8.1 soon, as in no longer than 3 weeks from now tops. There's this bug which I would consider 'serious' and another issue with the last-minute inclusion of BTreeFolder2 breaking CMF 1.5. IANRM though. | > Good, I had came to the same result with test and all *wink*. Wish I had | > looked at checkins before. | | Indeed, and you're even one of the few people subscribed to zodb-checkins | . BTW, Jim & I conspired last Wednesday to stop the flow of spam | coming from that list, so if anyone unsubscribed in a fit of Spam Hatred, | feel free to sign up again. And indeed I got the checking email. And, Oh! I know how I missed that one. I was in the u.s. consulate getting a visa and it went unnoticed as I cleaned the barrage of emails when I got back home. Doh! | > Humm... I don't see where the assumption would be except for a import | > dependency. | | Yes, I understand that. Maybe I've overestimate the value of this. | > My fear is that TmpStore does basically what FileStorage does | | Ah, maybe that's where we're getting off track. Not really; TmpStore and | FileStorage both write pickles to files, and both keep a kind of index | structure, but those are shallow similarities. TmpStore is about 60 lines | of code total, while FileStorage is almost 2100 in FileStorage.py alone (not | counting support modules only FileStorage uses); because TmpStore _must_ | defer to the underlying storage for most operations, it couldn't even | usefully take more from FileStorage. But, but, but... even for those 60 lines, would it make sense to factor out the code from FileStorage so that they could share just this small bit of common functionality? | > but get way less attention and ends up bitrotting. This is not the first | > time I've hit a (shallow) bug in TmpStore that would just be triggered by | > a fairly big or long subtransaction/savepoint. | | More tests would be appreciated! Since nobody looks at most of this code | most of the time, tests are the only reliable defense we have. Jim | "confessed" last week that he deliberately didn't write TmpStore tests | originally because TmpStore is "an internal implementation detail". You'll | note that the new test I added didn't mention TmpStore either, except in | comments. For that matter, the new test didn't use FileStorage either (it | used MinimalMemoryStorage, which is a bare-bones storage used only in the | test suite). Tests specifically against TmpStore would be welcome. /me adds to bottom of todo list. -- Sidnei da Silva Enfold Systems, LLC. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] TmpStore missing loadBefore
On Sun, Jun 26, 2005 at 12:41:40PM -0400, Tim Peters wrote: | [Sidnei da Silva] | > There's a collector issue open against ZODB 3.3 [Issue 5166], but the | > bug is still present in 3.4 and trunk. | | Can you be more specific, then? As I pointed out last time, rev 30825 | attempted to delegate loadBefore, and the new test for it passes (but failed | before the change TmpStore): | | http://svn.zope.org/ZODB/branches/3.4/?rev=30825&view=rev | | If you're still seeing the problem on the ZODB 3.4 branch or ZODB trunk, | then it must require more to provoke it than has been explained so far. Sorry, I really meant the ZODB version included in Zope 2.8 branch and trunk. See my other email. -- Sidnei da Silva Enfold Systems, LLC. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] TmpStore missing loadBefore
On Sun, Jun 26, 2005 at 12:27:57PM -0400, Tim Peters wrote: | This was fixed a few weeks, in rev 30825: | | http://svn.zope.org/ZODB/branches/3.4/?rev=30825&view=rev Ah, there we go. So that was fixed after 3.4.0 was tagged, and Zope 2.8 branch still points to the 3.4.0 tag. Guess we need a new tag. | If you don't care too much about the new test, it's just a matter of adding | the string "loadBefore" to this list in TmpStore.__init__: | | for method in ( | 'getName', 'new_oid', 'modifiedInVersion', 'getSize', | 'undoLog', 'versionEmpty', 'sortKey', 'loadBefore', | ): | setattr(self, method, getattr(storage, method)) Good, I had came to the same result with test and all *wink*. Wish I had looked at checkins before. | > Wouldn't it be simpler to make TmpStore subclass from FileStorage | | No, because TmpStore is used by Connection (to support savepoints), and | Connection can't make assumptions about the _kind_ of storage in use. Humm... I don't see where the assumption would be except for a import dependency. My fear is that TmpStore does basically what FileStorage does but get way less attention and ends up bitrotting. This is not the first time I've hit a (shallow) bug in TmpStore that would just be triggered by a fairly big or long subtransaction/savepoint. Anyway, you're probably in better position to judge this than I. -- Sidnei da Silva Enfold Systems, LLC. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] TmpStore missing loadBefore
On Sun, Jun 26, 2005 at 01:09:54PM -0300, Sidnei da Silva wrote: | We've hit a case where loadBefore would get called on TmpStore during | plone.org migration. | | Unfortunately TmpStore doesn't implement loadBefore. There seems to be | no tests for this. I tried to write one and provide a fix but just | occurred me that TmpStore tries to be a 'simpler FileStorage that | mixes with the current storage'. | | Wouldn't it be simpler to make TmpStore subclass from FileStorage and | just delegate to the current storage where appropriate? There's a collector issue open against ZODB 3.3, but the bug is still present in 3.4 and trunk. http://mail.zope.org/pipermail/zope-collector-monitor/2005-February/005166.html -- Sidnei da Silva Enfold Systems, LLC. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] TmpStore missing loadBefore
We've hit a case where loadBefore would get called on TmpStore during plone.org migration. Unfortunately TmpStore doesn't implement loadBefore. There seems to be no tests for this. I tried to write one and provide a fix but just occurred me that TmpStore tries to be a 'simpler FileStorage that mixes with the current storage'. Wouldn't it be simpler to make TmpStore subclass from FileStorage and just delegate to the current storage where appropriate? -- Sidnei da Silva Enfold Systems, LLC. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Running the test suite
On Tue, Apr 05, 2005 at 03:15:33PM +0200, Florent Guillaume wrote: | As it is I do a setup.py build every time before I run the tests but I'd | like to do everything in place. Like python setup.py build_ext -i ? -- Sidnei da Silva <[EMAIL PROTECTED]> http://awkly.org - dreamcatching :: making your dreams come true http://www.enfoldsystems.com http://plone.org/about/team#dreamcatcher want to write my requirements for me? Sure! "show a dancing monkey in the about box" ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
