[Zope] Finding unused products
Quick question before I reinvent. Is there some established way of finding products which have no instances in zodb, and can be safely removed? Regards Gaute Amundsen ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Finding unused products
On Nov 29, 2009, at 11:59 , Gaute Amundsen wrote: > > Quick question before I reinvent. > > Is there some established way of finding products which have no > instances in zodb, and can be safely removed? No there isn't. You may have to invest some time writing a script that visits all objects to find out what products they belong to. It may not be worth the effort. jens ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
Am 25.11.09 17:37, schrieb Jaroslav Lukesh: > At first, try to eliminate error outside of the Zope itself. Try to > install it all into plain whole new (and reliable!) machine. Do not use > restore of any backups! > > - Original Message - From: "Andreas Krasa" > > >> A week ago we switched to a new layout (for corporate reasons) and now >> we're experiencing frequent crashes of the Zope servers. Fortunately Hi Jaroslav, we're right in the process of tracking down the error outside of ZOPE. We have completely installed a new server from scratch with RHEL 5.4 and have re-installed python 2.4.6 and the latest versions of libxml2 and libxslt there. We double checked the LD config, and made sure that te correct shared objects get loaded (via lsof). We also reinstalled a few other modules that contain C-code (such as python-ldap) which we need for being able to do authenitcation. Unfortunately that didn't really help much. We still experience crashes. Are there any known issues with Zope 2.11.2, LibXML2 and/or LibXSLT that could cause these problems? The only thing we re-used is the Data.fs, which we have to, because we're talking about a production system here. Also note, that we have used excatly the same setup for a long time now, even on the same hardware, without any of these troubles. The problems only started when we switched over to a new (and probably more resource-intensive layout). We're unfortunately still not able to reproduce these crashes. Kind regards, Andreas ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
Try to cache page segments selectively and then you should identify problematic piece of page. If You does not use page segments, then divide page to some pieces and get them together. - Original Message - From: "Andreas Krasa" > > The only thing we re-used is the Data.fs, which we have to, because > we're talking about a production system here. > > Also note, that we have used excatly the same setup for a long time now, > even on the same hardware, without any of these troubles. The problems > only started when we switched over to a new (and probably more > resource-intensive layout). ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Finding unused products
On Sun, 2009-11-29 at 12:32 +0100, Jens Vagelpohl wrote: > On Nov 29, 2009, at 11:59 , Gaute Amundsen wrote: > > > > > Quick question before I reinvent. > > > > Is there some established way of finding products which have no > > instances in zodb, and can be safely removed? > > No there isn't. You may have to invest some time writing a script that > visits all objects to find out what products they belong to. It may > not be worth the effort. > > jens Unfortunately it has "not been worth the effort" for a number of years, but now it has become unavoidable. Walking the tree it is then. Gaute ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andreas Krasa wrote: > Am 25.11.09 17:37, schrieb Jaroslav Lukesh: >> At first, try to eliminate error outside of the Zope itself. Try to >> install it all into plain whole new (and reliable!) machine. Do not use >> restore of any backups! >> >> - Original Message - From: "Andreas Krasa" >> >> >>> A week ago we switched to a new layout (for corporate reasons) and now >>> we're experiencing frequent crashes of the Zope servers. Fortunately > > Hi Jaroslav, > > we're right in the process of tracking down the error outside of ZOPE. > > We have completely installed a new server from scratch with RHEL 5.4 and > have re-installed python 2.4.6 and the latest versions of libxml2 and > libxslt there. We double checked the LD config, and made sure that te > correct shared objects get loaded (via lsof). > > We also reinstalled a few other modules that contain C-code (such as > python-ldap) which we need for being able to do authenitcation. > > Unfortunately that didn't really help much. We still experience crashes. > > Are there any known issues with Zope 2.11.2, LibXML2 and/or LibXSLT that > could cause these problems? > > The only thing we re-used is the Data.fs, which we have to, because > we're talking about a production system here. > > Also note, that we have used excatly the same setup for a long time now, > even on the same hardware, without any of these troubles. The problems > only started when we switched over to a new (and probably more > resource-intensive layout). > > We're unfortunately still not able to reproduce these crashes. Can you set 'ulimit -c' to get a core file, which might at least help point to the extension which is to blame (although it may just show the "downstream" victim of a heap munge). What versions of libxml2 / libxslt are you using? How about lxml? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksS4FEACgkQ+gerLs4ltQ4d+ACeJom2eaL9cKTU5qq0zYd+4wjT syMAoI+2rPjBOJtz/HlQnbQgvDEghfGP =gOdT -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Finding unused products
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gaute Amundsen wrote: > On Sun, 2009-11-29 at 12:32 +0100, Jens Vagelpohl wrote: >> On Nov 29, 2009, at 11:59 , Gaute Amundsen wrote: >> >>> Quick question before I reinvent. >>> >>> Is there some established way of finding products which have no >>> instances in zodb, and can be safely removed? >> No there isn't. You may have to invest some time writing a script that >> visits all objects to find out what products they belong to. It may >> not be worth the effort. >> >> jens > > Unfortunately it has "not been worth the effort" for a number of years, > but now it has become unavoidable. > Walking the tree it is then. You could probably work directly with the pickles way faster than activating every object in a large site: look at the 'fsdump' script for clues. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksS4f0ACgkQ+gerLs4ltQ5s0wCfdk8Ru7uHUI7Fj7OGjDfcyAla 724An0ZOhZ2qorejVAwUzK2OdhFdPfYv =quZ0 -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Frequent ZOPE crashes
Hi Tres, thank you very much for your reply! Am 29.11.09 21:57, schrieb Tres Seaver: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > >>> - Original Message - From: "Andreas Krasa" >>> >>> >> we're right in the process of tracking down the error outside of ZOPE. >> >> We have completely installed a new server from scratch with RHEL 5.4 and >> have re-installed python 2.4.6 and the latest versions of libxml2 and >> libxslt there. We double checked the LD config, and made sure that te >> correct shared objects get loaded (via lsof). >> >> We also reinstalled a few other modules that contain C-code (such as >> python-ldap) which we need for being able to do authenitcation. >> >> Unfortunately that didn't really help much. We still experience crashes. >> >> Are there any known issues with Zope 2.11.2, LibXML2 and/or LibXSLT that >> could cause these problems? >> >> The only thing we re-used is the Data.fs, which we have to, because >> we're talking about a production system here. >> >> Also note, that we have used excatly the same setup for a long time now, >> even on the same hardware, without any of these troubles. The problems >> only started when we switched over to a new (and probably more >> resource-intensive layout). >> >> We're unfortunately still not able to reproduce these crashes. > > Can you set 'ulimit -c' to get a core file, which might at least help > point to the extension which is to blame (although it may just show the > "downstream" victim of a heap munge). > > What versions of libxml2 / libxslt are you using? How about lxml? Yes, we did set the ulimit and were indeed able to produce a coredump for each crash happening (each having something between 300 and 700 MB). We tried to debug using "gdb" but unfortunaley they only reveal two cases when the crashes occur: 1) During garbage collection where the gc tries to clean up damaged python objects 2) During some "ceval" process, also related to accessing damaged python objects Unfortunately it doesn't reveal what exactly trashes the objects. To us it seems that this could happen some time earlier before either of the two processes mentioned above tries to access the objects and crashes ZOPE. For now, we don't really see a reproduceable pattern as it seems to be a somewhat more complex user behavior which leads to this. We were able to extract a few URLs out of the coredumps but directly accessing those does nothing. Also the last logged access in the Z2.log before the coredump triggers nothing, when directly accessing it. We're running ZOPE-2.11.2 with an eggified version of ZODB3-3.8.4 plus libxml2-2.7.6, libxslt-1.1.26 and lxml-2.2.4 now, the crashes still happen. Previously we've been running with ZOPE-2.11.2, libxml2-2.7.3, libxslt-1.1.24 and lxml-2.1.5. That also crashed ZOPE occasionally. This only happened since we switched to a new layout (probably in combination with a few minor Silva updates). We have been using the same system software (RHEL5), hardware, python version and libxml2/libxslt/lxml versions with our old old layout, where everything worked fine for years. I would be happy to paste any particular gdb outputs if that is of any help...? Kind regards, Andreas ___ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
