Re: [Zope-dev] Re: More resilient indexes
On Tue, 2007-09-18 at 20:01 +0200, Dieter Maurer wrote: > Roché Compaan wrote at 2007-9-18 08:55 +0200: > > ... > >Sorry if I was unclear but what I'm really asking is if it is possible > >to improve the conflict handling of the current indexes that we have in > >Zope. I am also asking if concurrent indexing in the ZODB is a realistic > >goal. > > I have implemented "Conflict Reduced Indexes". > > They essentially work as follows: > >Standard indexes use for efficiency reasons a complex >dance with a quite high conflict potential: >The document list for a term can have 3 implementation > missing, represented as an integer, represented as an IITreeSet. > >Whenever the implentation type changes, a conflict will occur >when a concurrent request accesses the same document list. > >The conflict reduced indexes use only 2 implementation types: > missing and IITreeSet and once, the list used an IITreeSet, > it remains this way. >This can leverage the conflict resolution build in "OOBTree" >and "IITreeSet" quite well. > > Nevertheless, it turned out that these separate indexes were > not worth the efford (meanwhile, they have been replaced by > "ManagableIndex"). Thanks for your feedback. I refactored things a little bit so that I don't require immediate indexing which makes QueueCatalog a good solution. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] security problem in an monkey-patch
Joachim Schmitz wrote at 2007-9-19 11:54 +0200: >and > >../portal_catalog/getBypassQueue >displays a 1 This looks like a security bug. You should not be able to "call" something via the ZPublisher what you cannot call in a script. Maybe, you file a bug report? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: More resilient indexes
Laurence Rowe wrote at 2007-9-19 10:03 +0100: > ... >Indexes are inherently difficult to perform conflict resolution on. As >Dieter mentioned their implementation is designed for efficient reading, >not efficient writing. You did not mean me? I have implemented the "Conflict Reduced Indexes" because my colleague thought he would need to import mass data in parallel threads -- thus, the reason was to support parallel writing. Fortunately, the Zope based system was much faster than its C++ implemented predecessor. Thus, the need for mass import parallelization was not given to the expected extent We also use the QueueCatalog (in fact a two level queue: one for fast indexes and one for text indexes) and have only the most important workflow indexes updated inline. The workflow indexes, of course, need to be updated inline as otherwise, the workflows would not work reliably. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: security problem in an monkey-patch
Tres Seaver schrieb:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
security.declareProtected(view_management_screens, 'getBypassQueue')
def getBypassQueue(self):
"get _by_pass"
if not hasattr(self,"_bypass"):
self._bypass = False
return self._bypass
I would write this as:
return getattr(self, '_bypass', False)
avoiding both write-on-read and hasattr in one fell swoop.
thanks for the tip.
I get:
Unauthorized: The container has no security assertions. Access to
'getBypassQueue' of (QueueCatalog at /uniben/portal_catalog) denied.
What I am missing here.
You need to supply security assertions for the new method you have adeed
to the class (your security assertions are being "left behind" in the
context where you defined the function).. Likely you can add another
attribute to the class, 'getBypassQueue__roles__', with the value being
a tuple, ('Manager',) (unless you want to figure out how to create a
PermissionRoles object yourself).
I solved it with:
QueueCatalog.getBypassQueue__roles__ = ['Manager', 'Owner',]
thanks for the help.
--
Gruß Joachim
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: security problem in an monkey-patch
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Joachim Schmitz wrote:
> Hi,
>
> I have monkey-patched the QueueCatalog to adopt it to our needs, which
> works fine. I now wanted to introduce a new feature:
>
> The QueueCatalog should be bypassed during mass-import of data.
> So I introduced a new variable "_bypass", and new getBypassQueue() and
> setBypassQueue methods in the monkey-patch:
>
> security.declareProtected(view_management_screens, 'getBypassQueue')
> def getBypassQueue(self):
> "get _by_pass"
> if not hasattr(self,"_bypass"):
> self._bypass = False
> return self._bypass
I would write this as:
return getattr(self, '_bypass', False)
avoiding both write-on-read and hasattr in one fell swoop.
> security.declareProtected(view_management_screens, 'setBypassQueue')
> def setBypassQueue(self, bypass=False):
> "set _bypass"
> self._bypass = bypass
>
> from Products.QueueCatalog.QueueCatalog import QueueCatalog
> QueueCatalog.getBypassQueue = getBypassQueue
> QueueCatalog.setBypassQueue = setBypassQueue
>
>
> I can invoke these methods from the url like:
>
> ../portal_catalog/setBypassQueue?bypass=1
>
> and
>
> ../portal_catalog/getBypassQueue
> displays a 1
>
> But when I do a:
>
> tal:attributes="checked
> here/portal_catalog/getBypassQueue" />
>
> I get:
> Unauthorized: The container has no security assertions. Access to
> 'getBypassQueue' of (QueueCatalog at /uniben/portal_catalog) denied.
>
> What I am missing here.
You need to supply security assertions for the new method you have adeed
to the class (your security assertions are being "left behind" in the
context where you defined the function).. Likely you can add another
attribute to the class, 'getBypassQueue__roles__', with the value being
a tuple, ('Manager',) (unless you want to figure out how to create a
PermissionRoles object yourself).
Tres.
- --
===
Tres Seaver +1 540-429-0999 [EMAIL PROTECTED]
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG8S9F+gerLs4ltQ4RAp8kAJ4xECJyWPwPzvkOdDNiNGA3Vp6zNACg0bI5
41ihaq521kUpdFKgieWa0+A=
=IBzZ
-END PGP SIGNATURE-
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 5 OK
Summary of messages to the zope-tests list. Period Tue Sep 18 12:00:00 2007 UTC to Wed Sep 19 12:00:00 2007 UTC. There were 5 messages: 5 from Zope Unit Tests. Tests passed OK --- Subject: OK : Zope-2.7 Python-2.3.6 : Linux From: Zope Unit Tests Date: Tue Sep 18 20:50:14 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008361.html Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Unit Tests Date: Tue Sep 18 20:51:50 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008362.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Unit Tests Date: Tue Sep 18 20:53:21 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008363.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Unit Tests Date: Tue Sep 18 20:54:51 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008364.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Unit Tests Date: Tue Sep 18 20:56:21 EDT 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-September/008365.html ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] security problem in an monkey-patch
Hi, I have monkey-patched the QueueCatalog to adopt it to our needs, which works fine. I now wanted to introduce a new feature: The QueueCatalog should be bypassed during mass-import of data. So I introduced a new variable "_bypass", and new getBypassQueue() and setBypassQueue methods in the monkey-patch: security.declareProtected(view_management_screens, 'getBypassQueue') def getBypassQueue(self): "get _by_pass" if not hasattr(self,"_bypass"): self._bypass = False return self._bypass security.declareProtected(view_management_screens, 'setBypassQueue') def setBypassQueue(self, bypass=False): "set _bypass" self._bypass = bypass from Products.QueueCatalog.QueueCatalog import QueueCatalog QueueCatalog.getBypassQueue = getBypassQueue QueueCatalog.setBypassQueue = setBypassQueue I can invoke these methods from the url like: ../portal_catalog/setBypassQueue?bypass=1 and ../portal_catalog/getBypassQueue displays a 1 But when I do a: I get: Unauthorized: The container has no security assertions. Access to 'getBypassQueue' of (QueueCatalog at /uniben/portal_catalog) denied. What I am missing here. -- Gruß Joachim ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: More resilient indexes
Roché Compaan wrote: I use QueueCatalog often and I know how it works. But if an application requires immediate indexing then QueueCatalog is not a solution. Sorry if I was unclear but what I'm really asking is if it is possible to improve the conflict handling of the current indexes that we have in Zope. I am also asking if concurrent indexing in the ZODB is a realistic goal. The reason I mentioned QueueCatalog is not because of its batch indexing in one thread, but because it has a lot of conflict handling on the queue itself. Sessions might be another good example of a product that tries hard to handle conflicts. Do these products have strategies that can be made to work for indexes? By definition, if you're not using QueueCatalog then you don't have a queue to perform conflict resolution on. QueueCatalog does not perform conflict resolution as such, but optimises the number of cataloguing operations by applying rules to the queue to remove duplicate operations. Indexes are inherently difficult to perform conflict resolution on. As Dieter mentioned their implementation is designed for efficient reading, not efficient writing. Do you really need to have concurrent indexing? Would concurrently updating the metadata records (there is an option in QueueCatalog for this) be sufficient? Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
