Re: [Zope-dev] Improvements for Zope2's security
I think it's great that you did this... nice job! I have some
specific disagreements (while I think it's a reasonable constraint,
and I think something should enforce it, I don't believe it's the job
of something that we call a *security policy* to enforce whether a
method is called, e.g. via POST rather than GET), I think you did a
wonderful job analyzing this.. +1 on the developer-helper tools
portion of this that finds declarations that have no corresponding
method.
Thanks!
- C
On Sep 18, 2006, at 11:20 AM, Christian Heimes wrote:
Hey guys!
In the past few months I fiddled around with Zope2's security and
access
control code. I analysied my own code and code from other
developers to
search for common errors. Also I tried to think of ways to make the
security system easier and more verbose on coding errors
I have not yet implemented all my ideas but Zope 2.10 is on the door
steps. Here is my first set of improvements.
Issue 1
===
Zope's security declarations have to be called with a method *name* AS
STRING. Developers are human beeings and human beeings tend to make
small errors like typos. Or they forget to change the security
declaration when they rename a method. Zope doesn't raise an error
when
a developer adds a security declaration for a non existing method.
Have a look at the following example. It contains a tiny but
devastating
typo::
security.declarePrivate('chooseProtocol')
def chooseProtocols(self, request):
...
These kinds or errors are extremly hard to find and may lead to big
security holes. By the way this example was taken from a well known
and
well tested Zope addon!
Solution
The solution was very easy to implement. I created a small helper
function checkClassHasMethod() and called it in the apply() method of
AccessControl.SecurityInfo.ClassSecurityInfo. The apply() method is
called at startup. The code doesn't slow down requests.
Issue 2
===
Another way to introduce security breaches is to forget the
InitializeClass() call. The call is responsable for applying the
informations from a ClassSecurityInfo. Without the call all security
settings made through security.declare* are useless.
The good news is: Even if a developer forgets to call the method
explictly in his code the function is called implictly.
The implicit call was hard to find for me at first. I struggled
with the
code in OFS.Image because there is an import of InitializeClass but it
is never called. Never the less the security informations are somehow
merged automagically. After some digging I found the code that is
responsible for the magic. It's ExtensionClass' __class_init__
magic and
OFS.ObjectManager.ObjectManager.__class_init__
Now the bad news: The magic doesn't work under some conditions. For
example if you class doesn't subclass ObjectManager or if you monkey
patch a class with security info object you have to call
InitializeClass
explictly.
Solution
Not yet finished. I've created a function checkObjectHasSecurityInfo()
and added a call to ZPublisher.BaseRequest.DefaultPublishTraverse
but it
is untested.
Issue 3
===
Developers are lazy and they like to make typos. No one likes to type
security.declarePrivate('chooseProtocol') so we are using copy & paste
which may cause even more typos. Wouldn't it be cool to get rid of
security. and typing the name of the method twice? Let's use
decorators!
Here is the doc test example from my patch:
Solution
Security decorators are an alternative syntax to define security
declarations on classes.
from ExtensionClass import Base
from AccessControl import ClassSecurityInfo
from AccessControl.decorator import declarePublic
from AccessControl.decorator import declarePrivate
from AccessControl.decorator import declareProtected
from AccessControl.Permissions import view as View
from Globals import InitializeClass
class DecoratorExample(Base):
... '''decorator example'''
...
... security = ClassSecurityInfo()
...
... @declarePublic
... def publicMethod(self):
... "public method"
...
... @declarePrivate
... def privateMethod(self):
... "private method"
...
... @declareProtected(View)
... def protectedByView(self):
... "method protected by View"
...
InitializeClass(DecoratorExample)
With the new syntax you have to type only 15 letters instead of 41!
Issue 4
===
Some methods shouldn't be callable from certain types of request
methods. For example there is no need that the webdav DELETE and PROP*
methods should be callable from an ordinary HTTP GET request. I don't
want to go into details. Some people know why :)
Solution
Only a small subset is implemented. There are two methods in my
patch to
either whitelist or blacklist a request method. An older version of my
patch contained even code to distinguish between request types (ftp,
http, ftp, xml-rpc) but Jim told me in a private mail
Re: [Zope-dev] Serving large files
Oops.. should have read further on there... On Sep 18, 2006, at 4:44 PM, Chris McDonough wrote: Could you return an iterator to the server that knows enough to open its own database connection? Provide it with a callback that opens the connection and finds and chunks the relevant object? - C On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. d) If you just return the file as a string you can potentially run out of memory if the file is too big. I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
On Mon, Sep 18, 2006 at 04:44:17PM -0400, Chris McDonough wrote: | Could you return an iterator to the server that knows enough to open | its own database connection? Provide it with a callback that opens | the connection and finds and chunks the relevant object? I could certainly do that if I wanted to. What I'm trying to figure out is how/if Zope should do this for me. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Could you return an iterator to the server that knows enough to open its own database connection? Provide it with a callback that opens the connection and finds and chunks the relevant object? - C On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. d) If you just return the file as a string you can potentially run out of memory if the file is too big. I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Improvements for Zope2's security
You have many good points in your list of troubles. Many of them are resolved by using security declarations through ZCML instead. It would be interesting to here your views on this. //Lennart ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Sidnei da Silva wrote at 2006-9-17 16:09 -0300: > ... >c) If you use RESPONSE.write() you can break other > applications. ExternalEditor comes to mind. If you know the size and set it in the "Content-Length" response header, I cannot see a reason why it should break some applications. > ... >I would like to be able to return an iterator that can read from the >ZODB. Give the iterator a new ZODB (read only) connection; close it when the iterator comes to its end. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] buildbot failure in Zope branches 2.9 2.4 Linux zc-buildbot
The Buildbot has detected a failed build of Zope branches 2.9 2.4 Linux zc-buildbot. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 7709 Blamelist: adamg,andreasjung,baijum,benji_york,chrism,ctheune,dobe,fdrake,flox,jim,jukart,mgedmin,philikon,poster,rogerineichen,shh,srichter,tseaver,yusei BUILD FAILED: failed test sincerely, -The Buildbot ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] buildbot failure in Zope branches 2.9 2.4 Windows 2000 zc-bbwin2
The Buildbot has detected a failed build of Zope branches 2.9 2.4 Windows 2000 zc-bbwin2. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 7709 Blamelist: adamg,andreasjung,baijum,benji_york,chrism,ctheune,dobe,fdrake,flox,jim,jukart,mgedmin,philikon,poster,rogerineichen,shh,srichter,tseaver,yusei BUILD FAILED: failed compile sincerely, -The Buildbot ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: SVN: Zope/branches/2.9/lib/python/zope/ "Backport" testrunner from 2.10 branch.
Stefan H. Holek wrote: Log message for revision 70218: "Backport" testrunner from 2.10 branch. Changed: _U Zope/branches/2.9/lib/python/zope/ -=- Property changes on: Zope/branches/2.9/lib/python/zope ___ Name: svn:externals - cachedescriptors svn://svn.zope.org/repos/main/Zope3/tags/Zope-3.2.1/src/zope/cachedescriptors ... testing -r 41141 svn://svn.zope.org/repos/main/zope.testing/trunk/src/zope/testing contentprovider svn://svn.zope.org/repos/main/Zope3/tags/Zope-3.2.1/src/zope/contentprovider viewlet svn://svn.zope.org/repos/main/Zope3/tags/Zope-3.2.1/src/zope/viewlet + cachedescriptors svn://svn.zope.org/repos/main/Zope3/tags/Zope-3.2.1/src/zope/cachedescriptors ... testing svn://svn.zope.org/repos/main/zope.testing/trunk/src/zope/testing contentprovider svn://svn.zope.org/repos/main/Zope3/tags/Zope-3.2.1/src/zope/contentprovider viewlet svn://svn.zope.org/repos/main/Zope3/tags/Zope-3.2.1/src/zope/viewlet Stefan, we usually specify a specific revision or a tag so that the included package is not pulled in an arbitrary development version (which is the case now). ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Improvements for Zope2's security
Hey guys!
In the past few months I fiddled around with Zope2's security and access
control code. I analysied my own code and code from other developers to
search for common errors. Also I tried to think of ways to make the
security system easier and more verbose on coding errors
I have not yet implemented all my ideas but Zope 2.10 is on the door
steps. Here is my first set of improvements.
Issue 1
===
Zope's security declarations have to be called with a method *name* AS
STRING. Developers are human beeings and human beeings tend to make
small errors like typos. Or they forget to change the security
declaration when they rename a method. Zope doesn't raise an error when
a developer adds a security declaration for a non existing method.
Have a look at the following example. It contains a tiny but devastating
typo::
security.declarePrivate('chooseProtocol')
def chooseProtocols(self, request):
...
These kinds or errors are extremly hard to find and may lead to big
security holes. By the way this example was taken from a well known and
well tested Zope addon!
Solution
The solution was very easy to implement. I created a small helper
function checkClassHasMethod() and called it in the apply() method of
AccessControl.SecurityInfo.ClassSecurityInfo. The apply() method is
called at startup. The code doesn't slow down requests.
Issue 2
===
Another way to introduce security breaches is to forget the
InitializeClass() call. The call is responsable for applying the
informations from a ClassSecurityInfo. Without the call all security
settings made through security.declare* are useless.
The good news is: Even if a developer forgets to call the method
explictly in his code the function is called implictly.
The implicit call was hard to find for me at first. I struggled with the
code in OFS.Image because there is an import of InitializeClass but it
is never called. Never the less the security informations are somehow
merged automagically. After some digging I found the code that is
responsible for the magic. It's ExtensionClass' __class_init__ magic and
OFS.ObjectManager.ObjectManager.__class_init__
Now the bad news: The magic doesn't work under some conditions. For
example if you class doesn't subclass ObjectManager or if you monkey
patch a class with security info object you have to call InitializeClass
explictly.
Solution
Not yet finished. I've created a function checkObjectHasSecurityInfo()
and added a call to ZPublisher.BaseRequest.DefaultPublishTraverse but it
is untested.
Issue 3
===
Developers are lazy and they like to make typos. No one likes to type
security.declarePrivate('chooseProtocol') so we are using copy & paste
which may cause even more typos. Wouldn't it be cool to get rid of
security. and typing the name of the method twice? Let's use decorators!
Here is the doc test example from my patch:
Solution
Security decorators are an alternative syntax to define security
declarations on classes.
>>> from ExtensionClass import Base
>>> from AccessControl import ClassSecurityInfo
>>> from AccessControl.decorator import declarePublic
>>> from AccessControl.decorator import declarePrivate
>>> from AccessControl.decorator import declareProtected
>>> from AccessControl.Permissions import view as View
>>> from Globals import InitializeClass
>>> class DecoratorExample(Base):
... '''decorator example'''
...
... security = ClassSecurityInfo()
...
... @declarePublic
... def publicMethod(self):
... "public method"
...
... @declarePrivate
... def privateMethod(self):
... "private method"
...
... @declareProtected(View)
... def protectedByView(self):
... "method protected by View"
...
>>> InitializeClass(DecoratorExample)
With the new syntax you have to type only 15 letters instead of 41!
Issue 4
===
Some methods shouldn't be callable from certain types of request
methods. For example there is no need that the webdav DELETE and PROP*
methods should be callable from an ordinary HTTP GET request. I don't
want to go into details. Some people know why :)
Solution
Only a small subset is implemented. There are two methods in my patch to
either whitelist or blacklist a request method. An older version of my
patch contained even code to distinguish between request types (ftp,
http, ftp, xml-rpc) but Jim told me in a private mail it's kind of YAGNI.
At the moment blacklistRequestMethod() and whitelistRequestMethod() have
to be called explictly inside a method. There is no way to protect
methods via security.blacklistRequestMethod() or @blacklistRequestMethod.
I have two ideas to implement such security declarations but both ways
are complicated and hard to implement. An ordinary decorator doesn't
work because it messes up with ZPublisher.mapply.
The following code does NOT work with POST requests because mapply is
using introspection to get the names and default values of a method
Re: [Zope-dev] Serving large files
On Sep 18, 2006, at 8:16 AM, Christian Theune wrote: Jim Fulton wrote: On Sep 17, 2006, at 3:24 PM, Christian Theune wrote: I wonder whether this statement is true: Any connection handed out after the connection breaks the ACID compliance of the request. I wonder what "Any connection handed out after the connection" means. Argh. Obviously this was a broken sentence. :) I wonder again: Would the ACID compliance break if we hand out any ZODB connection after the worker thread is completed. And perhaps, implicitly, after the worker thread has closed the connection. It is likely that all sorts of bad things would happen, as the connection might already have been haded to another thread, Sidnei already mentioned to re-create a connection for the iterator to be passed back with the tid that represents the right state as of the end of the transaction in the worker thread. Of course, ot would be a different instance of the iterator, Something like that could certainly work. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
FileCacheManager does the copy-to-file-and-serve-as-iterator dance for you. http://www.dataflake.org/software/filecachemanager Stefan On 17. Sep 2006, at 21:09, Sidnei da Silva wrote: One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues. b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. -- It doesn't necessarily do it in chronological order, though. --Douglas Adams ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
Jim Fulton wrote: > > On Sep 17, 2006, at 3:24 PM, Christian Theune wrote: > >> I wonder whether this statement is true: >> >>Any connection handed out after the connection breaks the ACID >>compliance of the request. > > I wonder what "Any connection handed out after the connection" > means. Argh. Obviously this was a broken sentence. :) I wonder again: Would the ACID compliance break if we hand out any ZODB connection after the worker thread is completed. Sidnei already mentioned to re-create a connection for the iterator to be passed back with the tid that represents the right state as of the end of the transaction in the worker thread. -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: OpenPGP digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
On Sep 17, 2006, at 3:24 PM, Christian Theune wrote: I wonder whether this statement is true: Any connection handed out after the connection breaks the ACID compliance of the request. I wonder what "Any connection handed out after the connection" means. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Serving large files
On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote: I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool. One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support). Why would you ignore that? ... b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time. OK, so? c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind. I wasn't aware that it would break RESPONSE.write, but, in any case, this will cause a tempirary file to be created. ... I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this? You could create an entirely new server with different thread-management semantics. You could probably do this with the normal Zope publisher using one of the standard threading WSGI servers, Or you could use a temporary file or wait for blobs. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Serving large files
On Sun, 2006-17-09 at 16:09 -0300, Sidnei da Silva wrote: > b) If you dump the data to a temp file and return that as a file >iterator, it can potentially take twice the time. Potentially, yes... but from a practical standpoint I believe this will never happen. Writing to a tempfile over any sort of local storage (even a local network) will be *much* faster than downloading the file itself. If this is indeed a problem perhaps it's a deployment issue where you should ensure the temp dir is on a fast local volume to make sure this never happens. As it stands right now this seems to be the only sensible approach to returning a large file. - Rocky -- Rocky Burt ServerZen Software -- http://www.serverzen.com News About The Server (blog) -- http://www.serverzen.net signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: WebDAV PUT factory tests failing on 2.10 branch
Yes, this is a blocker IMO as it breaks FTP for a whole class of virtual hosting scenarios. No, I don't know how to fix, maybe Lennart? Stefan On 17. Sep 2006, at 12:52, Andreas Jung wrote: --On 17. September 2006 12:38:11 +0200 Philipp von Weitershausen <[EMAIL PROTECTED]> wrote: AFAIK, Stefan Hollek checked it in to demonstrate an incompatibility of Zope 2.10 vs. 2.9. See http://www.zope.org/Collectors/Zope/2187. Ahh..I remember. @Stefan: is this a blocker? Any chance to get this fixed? Andreas -- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
