Re: [Zope-dev] 64-bit BTrees
Fred Drake wrote: (possibly named LLBTree, LOBTree, and OLBTree). For my half-penny's worth, this is the way I'd like to see it go. Explicit is better than implicit and all that. If you need more than 32-bits, you can explicitly use them. The implicit change to make them all 64-bits which results in some unknown slowdown for all I BTree users seems a bit too scary to bite off... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Traversal issue which affects Five
Hi Alec, Five traversal is definitely very touchy, and the interactions with Zope numerous. So I'm sure the problem you observed is real and that a solution must be found. The unit tests you propose would go a long way to having a fix committed asap, so yes please, provide one. Florent Alec Mitchell wrote: It seems that the way OFS.Traversable.restrictedTraverse() handles security checking on objects with __bobo_traverse__ methods is considerably different from the way it normally checks security. The result is that traversal cannot obtain attributes using acquisition from objects that are marked five:traversable. In the normal case, security is checked using guarded_getattr, which gets an attribute and checks the permissions on the retrieved object in its original context. For __bobo_traverse__methods which return simple properties (say strings), it is impossible to determine the container from which the returned attribute originates, so unless the attribute was not acquired, an Unauthorized error will always be raised. Objects that are Five Traversable always have __bobo_traverse__ methods which attempt to mimic normal traversal, which effectively means that the security checks end up preventing acquisition of simple properties using traversal from ever working on these objects (say using a TAL path expression 'context/attribute' which you expect to be acquired). Unfortunately, because Five has no control over the security checks done during traversal, this cannot be fixed directly in Five. However, IMHO fixing this makes sense for Zope itself, provided there aren't any undesirable consequences. I propose that if the validation of a __bobo_traverse result raises Unauthorized, that we make one last check to see if the result o 'guarded_getattr(obj, name)' is identical to the result of the __bobo_traverse__ call and allow it if that's the case. Here is my proposed patch against Zope 2.9 trunk: --- Traversable.py (revision 66323) +++ Traversable.py (working copy) @@ -201,9 +201,18 @@ else: # Can't determine container container = _none -if not securityManager.validate( -obj, container, name, next): -raise Unauthorized, name +try: +if not securityManager.validate( +obj, container, name, next): +raise Unauthorized, name +except Unauthorized: +# If next is a simple unwrapped property, it's +# parentage is indeterminate, but it may have been +# acquired safely. In this case validate will raise +# an error, and we can check that our value was +# acquired safely. +if guarded_getattr(obj, name, marker) is not next: +raise Unauthorized, name else: if restricted: next = guarded_getattr(obj, name, marker) At the moment Plone 2.5 is really struggling with this issue, and it would be wonderful if a fix for this could go into Zope 2.8 and 2.9 soon. I'm happy to write tests for this, I just want to make sure that I'm not proposing something really wrong/inappropriate here. Generally, the validate() call should return a True or False value, so this change should have little performance impact except in the case where 'container == _none' and validate would otherwise raise a very unhelpful unauthorized error. Your feedback is much appreciated. Alec Mitchell ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cvs.zope.org/Products - svn.zope.org?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 17 Apr 2006, at 14:15, Andreas Jung wrote: Hi, almost all big Zope projects have changed to Subversion fortunately. Using svn:externals is a nice way to build bundles and helps a lot deploying software. However projects still have some dependencies from cvs.zope.org some older material (possibly not updated material). So the current solution is to write a shell script that pulls the stuff manually from cvs.zope.org. Wouldn't it make sense to migrate the complete Products hierarchy from cvs.zope.org to svn.zope.org? - -1 on migrating everything. IMHO it would be better to identify those dependencies and selectively migrate products as needed. I'd volunteer to do the migrations, I have a well-working test harness and migration scripts for that purpose on the cvs.zope.org box. Let's get a consensus on what should be migrated and where (meaning, what's the SVN directory naming fashion of the week), I'll handle it after that. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEQ6VKRAx5nvEhZLIRAqC5AJsHO1Oz/Ku/1cZDEPLnecg+E6DVOgCfX3Si vsxrmktoYeORi05zbo0wdog= =3n3Q -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Traversal issue which affects Five
On 4/17/06, Florent Guillaume [EMAIL PROTECTED] wrote: Hi Alec, Five traversal is definitely very touchy, and the interactions with Zope numerous. So I'm sure the problem you observed is real and that a solution must be found. The unit tests you propose would go a long way to having a fix committed asap, so yes please, provide one. ... Alec Mitchell wrote: It seems that the way OFS.Traversable.restrictedTraverse() handles security checking on objects with __bobo_traverse__ methods is considerably different from the way it normally checks security. The result is that traversal cannot obtain attributes using acquisition from objects that are marked five:traversable. In the normal case, security is checked using guarded_getattr, which gets an attribute and checks the permissions on the retrieved object in its original context. For __bobo_traverse__methods which return simple properties (say strings), it is impossible to determine the container from which the returned attribute originates, so unless the attribute was not acquired, an Unauthorized error will always be raised. Objects that are Five Traversable always have __bobo_traverse__ methods which attempt to mimic normal traversal, which effectively means that the security checks end up preventing acquisition of simple properties using traversal from ever working on these objects (say using a TAL path expression 'context/attribute' which you expect to be acquired). Unfortunately, because Five has no control over the security checks done during traversal, this cannot be fixed directly in Five. However, IMHO fixing this makes sense for Zope itself, provided there aren't any undesirable consequences. I propose that if the validation of a __bobo_traverse result raises Unauthorized, that we make one last check to see if the result o 'guarded_getattr(obj, name)' is identical to the result of the __bobo_traverse__ call and allow it if that's the case. Here is my proposed patch against Zope 2.9 trunk: snip old patch At the moment Plone 2.5 is really struggling with this issue, and it would be wonderful if a fix for this could go into Zope 2.8 and 2.9 soon. I'm happy to write tests for this, I just want to make sure that I'm not proposing something really wrong/inappropriate here. Generally, the validate() call should return a True or False value, so this change should have little performance impact except in the case where 'container == _none' and validate would otherwise raise a very unhelpful unauthorized error. Your feedback is much appreciated. Ok, I've attached a refined patch with tests. Only one of these will fail with the original version of Traversable.py (testBoboTraverseToAcquiredAttribute), the other three just make sure that expected security restrictions and behavior are preserved. I'll reiterate that fixing this issue is quite essential for Plone, and likely for any other reasonably complex Zope 2 project which wishes to use Five/Zope3 extensively. I'll file a collector issue later today. Alec bobo_traverse_security.diff Description: Binary data ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Traversal issue which affects Five
Alec Mitchell wrote at 2006-4-16 12:28 -0700: ... It seems that the way OFS.Traversable.restrictedTraverse() handles security checking on objects with __bobo_traverse__ methods is considerably different from the way it normally checks security. The result is that traversal cannot obtain attributes using acquisition from objects that are marked five:traversable. In the normal case, security is checked using guarded_getattr, which gets an attribute and checks the permissions on the retrieved object in its original context. For __bobo_traverse__methods which return simple properties (say strings), it is impossible to determine the container from which the returned attribute originates, so unless the attribute was not acquired, an Unauthorized error will always be raised. ... However, IMHO fixing this makes sense for Zope itself, provided there aren't any undesirable consequences. I propose that if the validation of a __bobo_traverse result raises Unauthorized, that we make one last check to see if the result o 'guarded_getattr(obj, name)' is identical to the result of the __bobo_traverse__ call and allow it if that's the case. Here is my proposed patch against Zope 2.9 trunk: In our local Zope version, I implemented a solution that is (in my opinion) superior: Define an exception UseTraversalDefault that can be used by __bobo_traverse__ to tell the traversal process (either URL traversal in the publisher or restricted/unrestricted/CMF traversal) to use the default lookup. One big advantage is that e.g. Archetypes.BaseObject.BaseObject.__bobo_traverse__ no longer need this horrible dance to decide whether it should or must not create a NullResource. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: 64-bit BTrees
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fred Drake wrote: I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). I think coming up with new types is the only reasonable thing to do, given the prevalence of persistent BTrees out in the wild. Changing the runtime behavior (footprint, performance) of those objects is probably not something which most users are going to want, at least not without carefully considering the implications. It really depends on what the impact is. It would be nice to get a feel for whether this really impacts memory or performance for real applications. This adds 4-bytes per key or value. That isn't much, especially in a typical Zope application. Similarly, it's hard to say what the difference in C integer operations will be. I can easily imagine it being negligible (or being significant :). OTOH, adding a new type could be a huge PITA. We'd like to use these with existing catalog and index code, all of which uses IIBTrees. If the performance impacts are modest, I'd much rather declare IIBTrees to use 64-bit rather than 32-bit integers. I suppose an alternative would be to add a mechanism to configure IIBTrees to use either 32-bit or 64-bit integers at run-time. Who uses IOBTree / OIBTree / IIBTree? - Catalogs map RIDs to UIDs as IOBTrees (one record per indexed object) - Most indexes (those derived from Unindex) map RID to indexed value as an IOBTree (one record per object with a value meaningful to that index) and map values to RIDs as OOBTrees (where the second O is usually an IITreeSet). - ZCTextIndex uses IIBTrees to map word IDs to RIDs, in various ways, and make use of IOBTrees as wel.. - Relationship indexes (typically not stored within catalogs) usually have an IIBTree which is the mapping of the edges as pairs of internal node IDs (one per explicit relationship), with OIBTrees to map the user-supplied node value to a node ID. I would guess that if you could do a census of all the OIDs in all the Datas.fs in the world, a significant majority of them would be instances of classes declared in IOBTree / IIBTree (certainly the bulk of *transaction* records are going to be tied up with them). OK. I think we are misscommunicating. Using 64 bits for IIBTrees types would not in any way invalidate existing pickles. 64-bit IIBTrees types can be unpickled from existing data. Of course, someone who created 64-bit BTrees type instances that had values outside the 32-bit range would have trouble reading these values with 32-bit IIBTrees, The fact that IIBTrees is so widely used is exatly the reason I want to use 64-bits for the existing types rather than having to introduce a new type. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: [Zope-dev] Re: 64-bit BTrees
On 4/17/06, Jim Fulton [EMAIL PROTECTED] wrote: The fact that IIBTrees is so widely used is exatly the reason I want to use 64-bits for the existing types rather than having to introduce a new type. Oops, I was checking in the separated version of 64-bit BTrees while this was landing in my inbox. ;-/ Once we determine which approach we're going with, I should make an alpha release of ZODB 3.7 and knit that into the Zope 3 trunk so we can get more testing in context. -Fred -- Fred L. Drake, Jr.fdrake at gmail.com Don't let schooling interfere with your education. -- Mark Twain ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: 64-bit BTrees
Fred Drake wrote: I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). -1 to the L*BTree flavours. The 'long' type is disappearing from Python anyways, why should the implementation detail worry us regarding BTrees? +1 to making the I*BTree flavours 64bit-aware. If Jim is right, existing pickles will be read back just fine and as long as the integers stay under 32 bit, they won't even be bigger. The only real implication therefore is the pushing around twice as many bits in memory. Dieter estimates 20% to 35% slowdown for the C algorithms (whatever that means), Tim seems to think it won't have such a big effect. I guess we'll only know after some benchmarks. Philipp ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Traversal issue which affects Five
On 4/17/06, Dieter Maurer [EMAIL PROTECTED] wrote: Alec Mitchell wrote at 2006-4-16 12:28 -0700: ... It seems that the way OFS.Traversable.restrictedTraverse() handles security checking on objects with __bobo_traverse__ methods is considerably different from the way it normally checks security. The result is that traversal cannot obtain attributes using acquisition from objects that are marked five:traversable. In the normal case, security is checked using guarded_getattr, which gets an attribute and checks the permissions on the retrieved object in its original context. For __bobo_traverse__methods which return simple properties (say strings), it is impossible to determine the container from which the returned attribute originates, so unless the attribute was not acquired, an Unauthorized error will always be raised. ... However, IMHO fixing this makes sense for Zope itself, provided there aren't any undesirable consequences. I propose that if the validation of a __bobo_traverse result raises Unauthorized, that we make one last check to see if the result o 'guarded_getattr(obj, name)' is identical to the result of the __bobo_traverse__ call and allow it if that's the case. Here is my proposed patch against Zope 2.9 trunk: In our local Zope version, I implemented a solution that is (in my opinion) superior: Define an exception UseTraversalDefault that can be used by __bobo_traverse__ to tell the traversal process (either URL traversal in the publisher or restricted/unrestricted/CMF traversal) to use the default lookup. One big advantage is that e.g. Archetypes.BaseObject.BaseObject.__bobo_traverse__ no longer need this horrible dance to decide whether it should or must not create a NullResource. Yes, it does sound like a better solution. However, the issue I see with it is that it is essentially adding new functionality, rather than fixing a problem with the existing behavior. That would seem to make it a much less likely candidate for getting into zope 2.8.7 (which is important IMHO), despite the fact that it is a bit more elegant. In fact, I would say that even with this clever exception handling, there may still be cases where one wants to return an acquired attribute directly from a __bobo_traverse__, without simply falling back on the default behavior (say if one wants to limit the acquirable names or some other silly thing). So the two solutions can coexist peacefully; though the exception based solution is likely to allow a lot of nice simplifications to some of the hacky __bobo_traverses__ out there. Alec ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
