[Zope-dev] Lazy expressions appear to cause memory leaks
In Zope 2.8+ there is a little known but very useful TALES feature: you can have expressions be lazily evaluated. For example, span tal:define=foo lazy:python:someExpensiveMethod() / The lazy: prefix causes the python expression to not be evaluated until foo is used somewhere. There appears to be a fairly big problem with this setup. The lazy: prefix wraps the expression in a LazyExpr which stores the expression and its context. The LazyExpr in turn generates a LazyWrapper (that holds similar information) which ends up getting held in the TALInterpreter's global/local variable list. The expression context for TAL expressions in a page template includes things like the template itself, the context object (a fully wrapped object), the request (chock full of complicated stuff), and so on. It appears that storing the expression context in the lazy wrapper creates some kind of circular reference or something similar that is preventing garbage collection of these lazy wrappers. The result is a nasty memory leak. The problem appears to be fixable via some cleaning up in PageTemplate.pt_render after the interpreter does its thing. The code snippet below is probably overkill, but something like this appears to be what is needed: context = getEngine().getContext(c) TALInterpreter(self._v_program, self._v_macros, context, output, tal=not source, strictinsert=0)() # clean up - try to eliminate circular references - this may be overkill context._compiler = None context.contexts = None context.repeat_vars = None from Products.PageTemplates.DeferExpr import LazyWrapper for k,v in context.global_vars.items(): if isinstance(v, LazyWrapper): v._expr = None v._econtext = None v._result = None if context.vars: while len(context.vars): context.vars._pop() context.global_vars.clear() context.global_vars = None context.local_vars.clear() context.local_vars = None context.vars = None context._scope_stack = None ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: running unit tests for Zope 2.8
Balazs, That almost did the trick. In the end I found that having both --libdir=/opt/Zope-2.8/lib/python/Products and --dir=/opt/Zope-2.8/lib/python/Products/SiteAccess/tests enabled the tests to run. Thanks! Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] running unit tests for Zope 2.8
Hi -- I have a fix for http://www.zope.org/Collectors/Zope/2037 and would like to write a unit test for it. However, I don't seem to be able to run the existing unit tests for SiteAccess with zopectl test. I created a new instance, then in my instance run bin/zopectl test --dir=/opt/Zope-2.8/lib/python/Products/SiteAccess/tests/ which yields the following: Running tests via: /usr/local/bin/python /opt/Zope-2.8/bin/test.py -v --config-file /home/zope/zopefix/etc/zope.conf --libdir Products --dir=/opt/Zope-2.8/lib/python/Products/SiteAccess/tests/ Running unit tests at level 1 Running unit tests from /opt/Zope-2.8/lib/python/Products/SiteAccess/tests Parsing /home/zope/zopefix/etc/zope.conf -- Ran 0 tests in 0.000s Any idea as to why no tests run? Am I missing some special incantation to get zopectl test to run? Or should I be running the Zope tests some other way? Thanks Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Principles
I am very glad to see that Jim's efforts to better articulate a vision for Zope have generated so much interest. I am not so sure that the discussion has been an entirely productive one. I think that we as a community would benefit by working on our social engineering as much as our software engineering. There are some straightforward and incredibly effective techniques for helping to ensure that the kinds of discussions we have been having are more productive. The book _Getting To Yes_ ( http://www.amazon.com/gp/product/0140157352 ) does a great job of describing them -- it's required reading in lots of university classes, particularly in MBA programs. The book is nominally about negotiations, but the ideas apply to a huge range of interactions. Here's a summary: http://www.colorado.edu/conflict/peace/example/fish7513.htm One of the things that GTY recommends is to establish a set of agreed upon principles for evaluating proposals. I think that having such a set of principles would help us better focus our current discussion. Let's take a step back from the particulars of the various proposals floating around and see if we can nail down some principles. Here is a very rough, very incomplete start: 1. Zope should have a clear message about where we are going. I'm sure we all agree on this, but this is so broad that it is not very useful. Here's a stab at refining it: 1.1 We should have a clear message about where Zope 2 is going. The message should give existing and prospective Zope 2 users an idea of how long their code will continue to work on releases in the Zope 2 path and what kind of upgrade process they will face as the Zope 2 line evolves. 1.2 Ditto for Zope 3. 2. Zope should try to expand its developer base. Again, I am sure we all agree, but this is too broad to be useful. 2.1 Zope should leverage the work of others by moving toward an architecture that allows one to easily use code from outside Zope. This effectively increases the developer base by letting us leverage the work of others outside the immediate Zope community. I assume that this (and integration) are the primary motivations driving the CA. 2.2 Zope should be useful for developers not using the full application server stack. Again, this serves to increase our developer base by drawing in people outside our traditional core audience. We probably need some principles about the Zope brand, and so on, but I think this should serve as a useful starting point. Let's see if we can expand and refine these principles before going down the vision road. I think that we will find that once we have articulated a set of core principles, defining a vision that adheres to them will be much easier. Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Re: Two visions
On Thu, 02 Mar 2006 09:43:03 -0330, Rocky Burt wrote: Anyway, this still keeps things very confusing from a naming perspective (mostly for new adopters). So having said all of that, I am actually +1 on Jim's proposal #2. What I see from that (someone correct me if I'm wrong) is the following: 1) rename zope 3 the framework as Z or zopelib or Zed or something sensical that doesn't confuse the early adopter's conquest of trying to figure out which zope to start with 2) Make zope 2 the application server acquire the name zope once again and be the only app server. This could only work (from a new adopter's perspective) if either the application server is given a new name or given a version number higher than 3. I agree with Rocky's assessment. +1 on Jim's suggestion #2. However, if I am understanding things correctly, it doesn't really sound like door #2 entails a huge deviation from from our current course of bringing Zope 2 and Zope 3 together gradually. I don't really care what the converged product is called, Zope 2.250 or Zope 3.99 or Zope 5. My take is that Jim is not really proposing anything all that different from what Martijn wants -- a gradual convergence of Zope 2 and 3. Rather, it sounds like the biggest changes in Jim's proposal #2 entail 1) a change in how we _talk_ about what we are doing, and 2) an explicit attempt to factor out some of the Zope 3 goodness into a more generic, less-monolithic-app-server framework, Zed. (am I right here, Jim?) I think that the idea of giving Zed its own, distinct identity is great. Zope 3 is a _huge_ overhaul and it needs to be obvious to the world that it is dramatically better than crufty old Zope 2. Zope 3 then becomes the Zed application server; Zope 2 is getting Zed retrofits via Five, and the two will eventually converge into Zope 5. A distinct Zed distribution could bring in developers who are just interested in using the component architecture but not necessarily a big app server stack. It would be cool to see Zed popping up in random python products or perhaps in TurboGears / Django internals. And more than just cool -- the more people we can get using Zed, the more code we will be able to mix in easily to Zope (2/3/5) applications. Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Two visions?
+1 on Jim's suggestion #2. However, if I am understanding things correctly, it doesn't really sound like door #2 entails a huge deviation from from our current course of bringing Zope 2 and Zope 3 together gradually. I don't really care what the converged product is called, be it Zope 2.250 or Zope 3.99 or Zope 5. My take is that Jim is not really proposing anything all that different from what Martijn wants -- a gradual convergence of Zope 2 and 3. Rather, it sounds like the biggest changes in Jim's proposal #2 entail: 1) a change in how we _talk_ about what we are doing, and 2) an explicit attempt to factor out some of the Zope 3 goodness into a more generic, less-monolithic-app-server framework, Zed (or Z or ZomethingElse). Am I right here, Jim? I think that the idea of giving Zed its own, distinct identity is great. Zope 3 is a _huge_ overhaul and it needs to be obvious to the world that it is dramatically better than crufty old Zope 2. Zope 3 then becomes the Zed application server; Zope 2 is getting Zed retrofits via Five, and the two will eventually converge into Zope 5 (or Zope 2.27 or whatever). A distinct Zed distribution could bring in developers who are just interested in using the component architecture but not necessarily a big app server stack. It would be cool to see Zed popping up in random python products or perhaps even in TurboGears / Django internals. And more than just cool -- the more people we can get using Zed, the more code we will be able to mix in easily to Zope (2/3/5) applications. Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Two visions?
On Thu, 02 Mar 2006 10:38:03 -0500, Jim Fulton wrote: I think that the idea of giving Zed its own, distinct identity is great. Zope 3 is a _huge_ overhaul and it needs to be obvious to the world that it is dramatically better than crufty old Zope 2. Zope 3 then becomes the Zed application server; Zope 2 is getting Zed retrofits via Five, and the two will eventually converge into Zope 5 (or Zope 2.27 or whatever). Ooops. OK I guess I was clear as mud. :) My idea for Z, pronounced zed or whatever the naming gods decide is that it was *not* an app server. It is an un-app-server. :) A collection of technologies that are useful by themselves, to support an app server and useful to build non-app-server applications, web or otherwise. No, I think I understood you. I was being sloppy in my use of language. I should have said something more like Zope 3 then becomes an application server built around the Zed library. I think that Z3 is better than Z2 in a lot of ways. I also think that Z2 is more mature and complete. I really want us to combine those efforts. I think we've achieved enough and learned enough with Zope 3 that we can now bring that to bear and make Zope 2 better, refactoring the cruft away and applying the lessons we've learned with Zope 3. (Note that Zope 3 is not crust free.) I don't really care what this thing ends up being called, except that it *must* be called Zope. Yes, I agree. Zope is the app server. I think that is consistent with the past use of the brand. This paragraph makes me think I was clear. Yes, we need to follow Ian Bicking's advice and release our technology in bite-sized chunks. I'm hopeful that the packaging efforts underway will lead to more of that. Yes, and the use of the new name Z or Zed is a way to emphasize that the Zed library is NOT a big, monolithic app server; rather, it's something new and cool. Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] CopySupport bug or feature?
I have been developing some classes that need to know when they are copied or moved. I have been achieving this by adding my own versions of _notifyOfCopyTo, _postCopy, and manage_afterClone. Everything works as expected when you copy / paste an individual object. However, I have recently been made aware of a problem with this approach: if you copy not the object itself, but rather a folder containing the object, only manage_afterClone is called. In looking through OFS/CopySupport.py and OFS/ObjectManager.py, I discovered the source of the problem: manage_afterClone is called recursively on ObjectManagers, but _notifyOfCopyTo and _postCopy are not. Is this intentional? If not, I'd like to check in a fix. Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: CopySupport bug or feature?
I've been talking to people on #zope about this. They're scared to touch the methods for backward compatibility reasons. Maybe the way to go is to create some new methods that are called recursively on objects when they are copied or moved. How about: def _beforeMove(self, destination_container, destination_path, recurse=True): # destination_container may be None (since for subobjects, the #destination will not have been created yet) # destination_path is a tuple containing the physical path of #the new container (which may not yet exist) # call subobjects recursively if recurse flag is True # # default implementation does nothing and returns self def _afterMove(self, source_container, recurse=True) # source_container may be None (since for subobjects, the #source may have been moved elsewhere) # source_path is a tuple containing the physical path of #the old container (which may no longer exist) # call subobjects recursively if recurse flag is True # # default implementation does nothing and returns self def _beforeCopy(self, destination_container, recurse=True): # destination_container may be None (since for subobjects, the #destination will not have been created yet) # destination_path is a tuple containing the physical path of #the new container (which may not yet exist) # call subobjects recursively if recurse flag is True # # default implementation does nothing and returns self def _afterCopy(self, source_container, original_object, recurse=True): # source_path is a tuple containing the physical path of #the new container (which should still exist) # call subobjects recursively if recurse flag is True # # default implementation does nothing and returns self The calls would look something like: ob = ob._beforeMove(destination, destination.getPhysicalPath(), True) ... moved_ob = moved_ob._afterMove(source, source.getPhysicalPath(), True) etc Once we have a decent event system in place, these methods could be the place where copy / move events are generated. Thoughts? Geoff On Mon, 08 Aug 2005 11:26:55 -0400, Chris McDonough wrote: I've been able to get around this by creating a _notifyOfCopyTo on the folder type that does the recursion. There's a delicate dance in some of my code that does some work in _notifyOfCopyTo and other work in _getCopy. This could be improved I bet, but touching the code means that you surely own it forever. ;-) On Mon, 2005-08-08 at 11:05 -0400, Geoff Davis wrote: I have been developing some classes that need to know when they are copied or moved. I have been achieving this by adding my own versions of _notifyOfCopyTo, _postCopy, and manage_afterClone. Everything works as expected when you copy / paste an individual object. However, I have recently been made aware of a problem with this approach: if you copy not the object itself, but rather a folder containing the object, only manage_afterClone is called. In looking through OFS/CopySupport.py and OFS/ObjectManager.py, I discovered the source of the problem: manage_afterClone is called recursively on ObjectManagers, but _notifyOfCopyTo and _postCopy are not. Is this intentional? If not, I'd like to check in a fix. Geoff ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: CopySupport, hooks, events
Florent, thanks for a thoughtful reply. Maybe we should start the larger discussion in moving to an event-driven OFS in Zope 2.8/2.9 using Five? Does Five support Z3's events (does Z3 have events yet? I have ordered Phillip's Z3 book but it has yet to arrive...) I'm cross posting to the Plone folks to see what they think. Geoff On Mon, 08 Aug 2005 20:29:38 +0200, Florent Guillaume wrote: I'd prefer names less potentially prone to clashes with existing code, namely _beforeItemMove, _afterItemCopy, etc. Also I tried to make a point on IRC about the recursion, and how we can switch this model to an event system later, but I probably wasn't clear enough. I'll try to explain better, but please bear with me :) First and foremost, I want to be able to switch from the current system where each class has to implement (or inherit) _beforeItemMove so that recursion applies, which would conceptually mean a base class or a mixin, to a system driven by events. Very importantly, this means that it's not _beforeItemMove's job to do the recursion. Why? Because suppose I inherit from a base class that has something to do before the move, and I want to add behavior to that. I'll have to call the base class, let it do its recursion, and then do my behavior on the current object. But suppose I inherit from two base classes that both had a _beforeItemMove that did recursion ? Then the recursion has to be defined and done only once, or you'd have double recursions (which turns into an exponential complexity). Maybe in some kind of unique base class? That's a total mess, I don't want to have to add artificial base classes to my code when there's no need to. I want the recursing behavior to be defined in a component (à la Zope 3), not in a base class. So the first consequence of having a clean approach is that: It's not the hook's job to do the recursion. So the recursion has to be done by external code, a component, that calls all the relevant objects. Now, in the current Zope 2 system having a class with a manage_beforeDelete (for instance) that just does pass means that the recursion is stopped. We want to keep that flexibility; this is easily done by having the hook return True if it wants to recurse (or to stop recursion -- we have to find what's best). Here's how an event system would work (using afterItemCopy as an example): 1. something decides to copy an object. A beforeCopy event is sent. Then the copy is done. The an afterCopy event is sent. 2. some code having the need to tidy up things after the copy has a subscriber subscribed to the afterCopy event. For instance, there could be a subscriber in CMFCatalogAware.py that indexes the new objects, calling indexObject on each. A subscriber implements a policy, in this case it would be copied objects have to be indexed in their new place. Notice, it's not indexObject's role to do any recursion. 3. the subscriber may or may not want the recursion to apply. How it does that is up to its implementation. I'm stressing point 3 here because if I copy 100.000 objects I don't want the default system to send 100.000 events. If I have an efficient way of doing my bookkeeping, I don't want the event system do screw things up behind my back. Note for instance that in CMF reindexObjectSecurity does not do recursion like manage_afterAdd does, because it has optimized ways to do its job without recursing using objectValues(). So I strongly believe the default event system should send a simple event on copy (and I think I'll have to fight to get my way in Zope 3...). Now, if something wants to recurse, it's free to do so, and maybe send more events. But I want the system to be able to work without that. Using this system, there could be a default subscriber in OFS/CopySupport.py that calls _afterItemCopy on the toplevel object and then does the recursion. The *subscriber* does the recursion, not _afterItemCopy. But we're not there yet (no events in Zope 2), so instead of sending an event that's caught by a subscriber that does a recursion, this can be done for now by calling directly the code equivalent to the subscriber. Later we'll move to the event system. (And it is my hope that we can deprecate manage_beforeDelete al. too.) So I propose: def _beforeItemMove(dest_container, dest_path): Called before an item is moved. dest_container may be None (since for subobjects, the destination will not have been created yet). dest_path is a tuple containing the physical path of the new container (which may not yet exist). Returns True if recursion should stop. def _afterItemMove(source_container, source_path): Called afer an item has been moved. source_container may be None (since for subobjects, the source may have been moved elsewhere). source_path
