Re: [Zope-dev] Strange bug(?) accessing File objects
Bjorn Stabell wrote: When accessing File objects that are not accessible to Anonymous (HTTP and WebDAV View permissions not given), the Basic HTTP Auth window pops up repeatedly even after the user has logged in using the cookie crumbler method, and the user has permissions to view the file. Clicking cancel actually lets the user view the file, but that's not an acceptable solution, of course. Hmmm, you absolutely positive that's a basic auth box? Do you have MS Office installed on the machines where this happens? What type of files does this occur with? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.7.0-b2 startup error on windows
Michael Long wrote:
I have recently installed Zope 2.7.0-b2 on windows XP and am getting the
same error message as 2.7.0-b1 when starting zope. Is this truly an
error message or just informational?
2003-09-05T11:37:23 BLATHER(-100) ZODB Commiting subtransaction of size 5382
Traceback (most recent call last):
File C:\Program Files\Zope-2.7.0-b2\lib\python\Zope\App\startup.py,
line 52, in startup
m=imp.find_module('custom_zodb',[getConfiguration().instancehome])
ImportError: No module named custom_zodb
Traceback (most recent call last):
File C:\Program Files\Zope-2.7.0-b2\lib\python\Zope\App\startup.py,
line 52, in startup
m=imp.find_module('custom_zodb',[getConfiguration().instancehome])
ImportError: No module named custom_zodb
Try adding objects to your Zope, if that works and they stick around even after
you restart Zope, then this isn't so much of a problem.
However, it's nasty, and I suspect you find you may not be able to store objects
to disk, so this does need fixing :-(
Please file on http://collector.zope.org
cheers,
Chris
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Catalog performance
Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr ... With about 10.000 documents this Python script takes 10 minutes and more than 500MB of memory, after that I had to restart Zope. I am running Zope 2.6.1 + Plone 1.0 on Windows 2000, Xeon P4 with 1GB RAM. What's wrong with this code? Any suggestion is appreciated. Nguyen Quan Son. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Catalog performance
Nguyen Quan Son wrote: Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr ... With about 10.000 documents this Python script takes 10 minutes and more than 500MB of memory, after that I had to restart Zope. I am running Zope 2.6.1 + Plone 1.0 on Windows 2000, Xeon P4 with 1GB RAM. What's wrong with this code? Any suggestion is appreciated. Nguyen Quan Son. it's not the catalog that's slow: t1=time.time() docs = container.portal_catalog(meta_type='Document', ...) t2=time.time() for doc in docs: obj = doc.getObject() value = obj.attr ... t3=time.time() print out the times and you'll see that the the finding is fast. the problem is that you are inflating each and every document one after another, and that takes time. Romain Slootmaekers. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Caching prob with AHCM and headers
Accelerated HTTP Caching Manager doesn't work out-of-the-box as this thread reported, but the thread had no conclusion: http://mail.zope.org/pipermail/zope/2003-April/134800.html http://mail.zope.org/pipermail/zope/2003-April/135059.html http://mail.zope.org/pipermail/zope/2003-April/135101.html I'm running into the same problem now, also with Apache as the caching reverse proxy, so I was wondering if there was any progress? Here are some of my thoughts; It seems reasonable that without a Last-Modified or Etag header in the response, that a client (and surrogate/ http accelerator) will not have any validators (e.g., If-Modified-Since or If-Match corresponding to last modified and etag headers, respectively) in the request headers. It's probably right that dynamic code doesn't return a Last-Modified header; instead, the RFC includes an Etag header, which is hash of the content, which is much easier to use for dynamic code. The Etag header returned by Zope, however, looks very suspicous; it is empty. This could perhaps be fooling the caching machinery? Perhaps the Etag header should just always be a hash of the content? (Another system worth looking at: http://www.jpcache.com/) Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: PythonLibraries Product
Evan Simpson wrote: I'm thinking seriously about writing a Product to provide collections of Python functions defined by a single source text -- PythonLibraries. This would *not* be the same as Zope 3's persistent modules, although it would provide some of the same benefits. Here's the README.txt: Python Libraries The Python Libraries Product provides support for collections of restricted Python code. A Python Library is similar to a Folder full of Python-based Scripts, except that the functions in the Library are more like ordinary Python functions than Scripts, and a single persistent global variable namespace is shared among the functions in the Library. So AFAICT it's a convenience which allows you (a) to keep related functions together; and (b) to store local variables in a convenient place. It definitely sounds useful but also a lot of work for something it's possible to manage without quite easily at the moment..? Could you provide a brief summary of why this is better than a folder of python scripts? Perhaps a use case which illustrates the problems of the current way of doing things? seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Catalog performance
Nguyen Quan Son wrote: Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr ... With about 10.000 documents this Python script takes 10 minutes and more than 500MB of memory, after that I had to restart Zope. I am running Zope 2.6.1 + Plone 1.0 on Windows 2000, Xeon P4 with 1GB RAM. What's wrong with this code? Any suggestion is appreciated. With getObject(), you're loading entire objects into memory in order to grab a single attribute. This is very wasteful. Try putting the attribute into the metadata for the catalog and grabbing it from there. Then you can do: for doc in docs: value = doc.attr seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] AW: Caching prob with AHCM and headers
Well, sadly I didn't find the time to pursue the issue any further yet. We're kneedeep in work to get the system up and running and caching isn't top priority so far. I don't even know if someone patched the AHCM yet, simply haven't looked. But as someone suggested somewhere and you can see in http://mail.zope.org/pipermail/zope/2003-April/135101.html it works if you set the headers yourself. So if you need caching _now_ and AHCM is not fixed yet, change the object used so that it sets the headers correctly itself. I know that this solution is ikky but I don't know when I will have time to tackle the issue of a comfortable cache system. Though I know I will have to solve it sometime :) /Carsten -Ursprüngliche Nachricht- Von: Bjorn Stabell [mailto:[EMAIL PROTECTED] Gesendet: Mittwoch, 10. September 2003 13:07 An: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Betreff: Re: Caching prob with AHCM and headers Accelerated HTTP Caching Manager doesn't work out-of-the-box as this thread reported, but the thread had no conclusion: http://mail.zope.org/pipermail/zope/2003-April/134800.html http://mail.zope.org/pipermail/zope/2003-April/135059.html http://mail.zope.org/pipermail/zope/2003-April/135101.html I'm running into the same problem now, also with Apache as the caching reverse proxy, so I was wondering if there was any progress? Here are some of my thoughts; It seems reasonable that without a Last-Modified or Etag header in the response, that a client (and surrogate/ http accelerator) will not have any validators (e.g., If-Modified-Since or If-Match corresponding to last modified and etag headers, respectively) in the request headers. It's probably right that dynamic code doesn't return a Last-Modified header; instead, the RFC includes an Etag header, which is hash of the content, which is much easier to use for dynamic code. The Etag header returned by Zope, however, looks very suspicous; it is empty. This could perhaps be fooling the caching machinery? Perhaps the Etag header should just always be a hash of the content? (Another system worth looking at: http://www.jpcache.com/) Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] i18n:attributes DeprecationWarning on Zope head? Space separated attributes in i18n:attributes are deprecated (i18n:attributes=value title)
The message makes sense, what should they be replaced with? [EMAIL PROTECTED] wrote: == Python Version:2.2.3 (#1, Jun 10 2003, 13:52:48) [GCC 2.95.2 2220 (Debian GNU/Linux)] Modules included: Zope (HEAD) Running unit tests from /stuff/chrisw/ZopeTests/sandbox/Zope /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes=value title). Please use semicolon to separate attributes (i18n:attributes=value; title). File None at row, column (5, 3) Attributes alt , DeprecationWarning) /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes=value title). Please use semicolon to separate attributes (i18n:attributes=value; title). File None at row, column (1, 0) Attributes name , DeprecationWarning) /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:862: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes=value title). Please use semicolon to separate attributes (i18n:attributes=value; title). File None at row, column (14, 0) Attributes name title , DeprecationWarning) /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:862: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes=value title). Please use semicolon to separate attributes (i18n:attributes=value; title). File None at row, column (17, 0) Attributes name title , DeprecationWarning) /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes=value title). Please use semicolon to separate attributes (i18n:attributes=value; title). File None at row, column (1, 0) Attributes value , DeprecationWarning) /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes=value title). Please use semicolon to separate attributes (i18n:attributes=value; title). File None at row, column (1, 0) Attributes alt , DeprecationWarning) -- Ran 2288 tests in 1155.309s OK ___ Zope-Coders mailing list [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-coders ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Catalog performance
Max M wrote: Nguyen Quan Son wrote: Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr ... With about 10.000 documents this Python script takes 10 minutes and more than 500MB of memory, after that I had to restart Zope. I am running Zope 2.6.1 + Plone 1.0 on Windows 2000, Xeon P4 with 1GB RAM. What's wrong with this code? Any suggestion is appreciated. Nguyen Quan Son. Most likely you are filling the memory of your server so that you are swapping to disk. Try cutting the query into smaller pieces so that the memory doesn't get filled up. If you can't use catalog metadata as Seb suggests (eg. you are actually accessing many attributes, large values, etc.) and if indeeed memory is the problem (which seems likely) then you can ghostify the objects that were ghosts to begin with, and it will save memory (unless all those objects are already in cache). The problem with this strategy though is that doc.getObject() method used in your code activates the object and hence you won't know if it was a ghost already or not. To get around this you can shortcut this method and do something like : docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.aq_parent.unrestrictedTraverse(doc.getPath()) was_ghost = obj._p_changed is None value = obj.attr if was_ghost:obj._p_deactivate() You can test this by running your code on a freshly restarted server, and check the number of objects in cache. The number shouldn't change much after running the above method, but will increase dramatically if you just used 'obj = doc.getObject()' instead, or didn't do the deactivating of the objects. The lower number of objects in your cache should in turn keep your memory usage down, and prevent your computer paging through the request, and hence speed things up considerably! Another option would be to reduce the size of your cache so that the amount of memory your zope instance consumes doesn't cause your computer to swap, though doing the above code changes will also help keep your cache with the 'right' objects in it as well, which in turn will further help with the performance of subsequent requests. Cheers, JB. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.7.0-b2 startup error on windows
Try adding objects to your Zope, if that works and they stick around even after you restart Zope, then this isn't so much of a problem. However, it's nasty, and I suspect you find you may not be able to store objects to disk, so this does need fixing :-( Please file on http://collector.zope.org I love the ease of adding zope instances in 2.7!! After some further investigation I have found that a product that I am developing is causing the error to appear. The product appears to work fine in zope. I can add it, create objects with it and the objects are persistent. Can someone give me some advice on how to go about debugging my product to identify the offending code? The error appears in the following environments: SuSE 8.2 -- Zope 2.6.1 WinXP Pro -- Zope 2.7.x Thanks, Mike ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.7.0-b2 startup error on windows -- SOLVED
I have found the offending code...he said sheepishly. In the __init__.py file I had the following: def initialize(context): ... import traceback; traceback.print_exc() instead of: def initialize(context): try: ... except: import traceback; traceback.print_exc() My thanks goes out to all who helped. Later, Mike After some further investigation I have found that a product that I am developing is causing the error to appear. The product appears to work fine in zope. I can add it, create objects with it and the objects are persistent. Can someone give me some advice on how to go about debugging my product to identify the offending code? The error appears in the following environments: SuSE 8.2 -- Zope 2.6.1 WinXP Pro -- Zope 2.7.x Thanks, Mike ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: PythonLibraries Product
- Original Message - From: Evan Simpson [EMAIL PROTECTED] Both of these solutions suffer from the two or more objects to accomplish one task management problem. I usually combine the objects in a dedicated Folder, which is still a bit awkward, and doesn't solve everything. One drawback is that the individual components are all published, despite the fact that I only intend one of them to be web-addressable. ...znip... On the other hand, Libraries don't support 'traverse_subpath', require an extra path step to address, and don't allow permission and proxy settings at individual function granularity. hm... I have made similar observations, and not only with Scripts (Python)... one idea I had, though I do not know if it is possible to implement, is to create a new kind of container that only exposes/publishes objects with a certain permission. Is it possible w/o patching the way zpublisher works? /dario - Dario Lopez-Kästen, IT Systems Services Chalmers University of Tech. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Catalog performance
John Barratt wrote: the problem (which seems likely) then you can ghostify the objects that were ghosts to begin with, and it will save memory (unless all those objects are already in cache). This is rather interesting, but I don't quite follow what's happening. If you can say a little more, or suggest a doc reference, I'm all ears. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PythonLibraries Product
Evan Simpson wrote: Seb Bacon wrote: Could you provide a brief summary of why this is better than a folder of python scripts? Perhaps a use case which illustrates the problems of the current way of doing things? It isn't *better* than a Folder of Scripts, it's *different* :-) I have two weak use-cases, a longstanding I'm going to write that some day itch, and some aesthetic arguments. One of the more frequently stated complaints about ZPTs is that to properly remove non-presentation logic and ugly little Python expressions from a template, you either need a pile of little Scripts, or a single controller Script that precalculates everything and passes it to the template. I've settled, fairly happily, on the second solution. Both of these solutions suffer from the two or more objects to accomplish one task management problem. I usually combine the objects in a dedicated Folder, which is still a bit awkward, and doesn't solve everything. One drawback is that the individual components are all published, despite the fact that I only intend one of them to be web-addressable. It would be very handy to attach the Scripts directly to the template, but the straightforward way of doing that means making the template a container, and opens a whole can of worms. Instead, what if templates had a Library tab, and it worked with External Editor? It would actually be easier to *just* write this, and not have independent Library objects, but generalizing this capability feels right to me Actually, the restricted case is the one which has the real win; the free-floating library is pretty, but not semanticaally needed. An added argument: a ZPT with its own private library becomes, in effect, a Zope3 view component; adopting such beasts will ease migration to Zope3. For the filesystem representation: what if we just have two files for templates with libraries: 'foo.html' and 'foo.html.py'? Tools will like that better than either of the approaches which try to preserve the twins as a single file: - Embedding the Python code into an HTML/XML comment at the head or foot of the template in the .pt file will not let editors help with syntax highlighting. - In addition to messing up syntax highlighting, embedding the template as a triple-quoted string literal in the .py file feels *really* nasty (almost as nasty as Zope3's script tag). And there is yet another argument for the bound library: it shows how unnecessary, as well has how evil, that script tag hack is. Refactoring a one-off template-cum-library will be simple, too: once the refactored code is moved to a real module (even a persistent module in Zope3!), the library can just contain 'from realmodule import usefulFunction', and the template won't need to change. I would vote for having the bound library's names inserted into the 'template' TALES namespace, BTW. Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] AW: Caching prob with AHCM and headers
Carsten wrote: Well, sadly I didn't find the time to pursue the issue any further yet. We're kneedeep in work to get the system up and running and caching isn't top priority so far. I don't even know if someone patched the AHCM yet, simply haven't looked. But as someone suggested somewhere and you can see in http://mail.zope.org/pipermail/zope/2003-April/135101.html it works if you set the headers yourself. So if you need caching _now_ and AHCM is not fixed yet, change the object used so that it sets the headers correctly itself. I know that this solution is ikky but I don't know when I will have time to tackle the issue of a comfortable cache system. Though I know I will have to solve it sometime :) I know, I was just trying to figure out how this was supposed to work. Currently setting Last-Modified seems to be the only way to get something cached in Apache; Expires alone is no good, and adding an Etag header doesn't seem to have any effect either. Maybe this should be classified as an Apache problem? Shouldn't Apache cache pages that have Expires? Setting Last-Modified to the current time isn't always the right thing; some objects, .e.g., Files, Images, Documents, have last modification times, and using those would be better. Not sure if the accelerator can be smart enough to know about these, though. Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Catalog performance
Simon Michael wrote: John Barratt wrote: the problem (which seems likely) then you can ghostify the objects that were ghosts to begin with, and it will save memory (unless all those objects are already in cache). This is rather interesting, but I don't quite follow what's happening. If you can say a little more, or suggest a doc reference, I'm all ears. In general when an object is first loaded from the ZODB it is in a 'ghost' state, and is only a shell, it has no attributes etc. When you access (almost) any attribute on that object (eg. do : value = ob.attr), it gets activated (the contents are loaded automatically, and then the value returned). This is when the real memory usage takes place. So if you get an object from the ZODB and don't access any attributes, it will remain in a ghosted state. Some core python attributes *don't* cause it to activate such as accessing __dict__, and also clearly the reserved persistent _p_* attributes. If you look at the Cache Paramaters tab of your Database in the Control Panel (at least with Zope 2.6.2, perhaps 2.6.1) you can see how many objects are in memory, and how many are just 'ghosts'. I think ghosts are only 'removed' after a restart, and essentially just contain a _p_oid that references the object in the ZODB, ready for re-activation. A general reference for the ZODB can be found here that explains more : http://www.python.org/workshops/2000-01/proceedings/papers/fulton/zodb3.html An example use (and good discussion) that is similar, can be found at the link below. I found this after having problems with objects not de-ghostifying properly when just accessing __dict__ : http://aspn.activestate.com/ASPN/Mail/Message/zodb-dev/913762 Also a grep through the zope source code some products will also find many examples of 'deactivating' objects after a 'walk' : eg. From OFS.ObjectManager : def manage_afterAdd(self, item, container): for object in self.objectValues(): try: s=object._p_changed except: s=0 if hasattr(aq_base(object), 'manage_afterAdd'): object.manage_afterAdd(item, container) if s is None: object._p_deactivate() A change to my example code that would be advisable is the wrapping of the _p_changed test in a try/except incase the object is None, or for some reason isn't persistent, and hence doesn't have a _p_changed. I hope this helps makes sense! Cheers, JB. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Catalog performance - SOLVED
I've added catalog metadata as Seb suggested and it works fine. Thank you very much. Nguyen Quan Son Nguyen Quan Son wrote: Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr ... With about 10.000 documents this Python script takes 10 minutes and more than 500MB of memory, after that I had to restart Zope. I am running Zope 2.6.1 + Plone 1.0 on Windows 2000, Xeon P4 with 1GB RAM. What's wrong with this code? Any suggestion is appreciated. From: John Barratt [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, September 10, 2003 6:41 PM Subject: Re: [Zope-dev] Catalog performance If you can't use catalog metadata as Seb suggests (eg. you are actually accessing many attributes, large values, etc.) and if indeeed memory is the problem (which seems likely) then you can ghostify the objects that were ghosts to begin with, and it will save memory (unless all those objects are already in cache). The problem with this strategy though is that doc.getObject() method used in your code activates the object and hence you won't know if it was a ghost already or not. To get around this you can shortcut this method and do something like : docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.aq_parent.unrestrictedTraverse(doc.getPath()) was_ghost = obj._p_changed is None value = obj.attr if was_ghost:obj._p_deactivate() You can test this by running your code on a freshly restarted server, and check the number of objects in cache. The number shouldn't change much after running the above method, but will increase dramatically if you just used 'obj = doc.getObject()' instead, or didn't do the deactivating of the objects. The lower number of objects in your cache should in turn keep your memory usage down, and prevent your computer paging through the request, and hence speed things up considerably! Another option would be to reduce the size of your cache so that the amount of memory your zope instance consumes doesn't cause your computer to swap, though doing the above code changes will also help keep your cache with the 'right' objects in it as well, which in turn will further help with the performance of subsequent requests. Cheers, JB. From: Seb Bacon [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, September 10, 2003 6:18 PM Subject: [Zope-dev] Re: Catalog performance With getObject(), you're loading entire objects into memory in order to grab a single attribute. This is very wasteful. Try putting the attribute into the metadata for the catalog and grabbing it from there. Then you can do: for doc in docs: value = doc.attr seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] AW: Caching prob with AHCM and headers
I know, I was just trying to figure out how this was supposed to work. Currently setting Last-Modified seems to be the only way to get something cached in Apache; Expires alone is no good, and adding an Etag header doesn't seem to have any effect either. Maybe this should be classified as an Apache problem? Shouldn't Apache cache pages that have Expires? Please keep in mind that **Apache is not a cache server**, it has caching attached more like an afterthought to the mod_proxy. If you want predictable caching you will need to use real caching software such as Squid. I personally have moved away from Apache even for small websites since there are some other issues with Apache caching, such as zero-length responses in some cases. jens ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Catalog performance
John - your post and the links helped a lot. Thanks! ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
