[Zope] long running methods

2010-05-20 Thread Jürgen Herrmann 

hi there!

as our zope2 zodb grows we experience more and more conflicts 
with long running transactions. so i'm going to rewrite these
methods to commit transactions in batches - here my attempt
to tackle this problem. the idea is to get a list of objects
to be modified in this transaction, break it down to batches
and commit after each batch was processed. to avoid zope's 
retry-on-conflict machinery and still have a chance to retry 
per batch transaction commits, the iteration over one batch
is guarded by a try:except ConflictError: and a retry logic.

i wrote some more or less pseudo code to make this clear.

def myLongRunningMethod(self):

  BATCH_SIZE = 10
  MAX_CONFLICTS = 3

  work_items = [some, work, to , do, ...]
  # list of persistent objects to be modified in this method

  idx = 0
  while idx < len(work_items):
conflicts = 0
try:
  my_batch = work_items[idx, idx+BATCH_SIZE]
  for work_item in my_batch:
do_some_work(work_item)
  transaction.commit()
except ConflicError:
  conflicts += 1
  if conflicts > MAX_CONFLICTS:
raise
else:
  idx += BATCH_SIZE

does this sound like a reasonable approach?

jürgen
-- 
>> XLhost.de - eXperts in Linux hosting ® <<

XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany

Geschäftsführer: Volker Geith, Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218

Fon:  +49 (0)800 XLHOSTDE [0800 95467833]
Fax:  +49 (0)800 95467830

WEB:  http://www.XLhost.de
IRC:  #xlh...@irc.quakenet.org
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] PAU + loginpagename = FAIL

2010-05-20 Thread Dr. Volker Jaenisch 
Hi zope list!

Tried to use the PAU feature of the challenge mechanism
of the sessionbasedcredential plugin in bluebream.

The main registration of PAU :

@adapter(INewMLSApplicationEvent)
def createPAU( event ):
sm = event.object.getSiteManager()
pau = PluggableAuthentication()
sm['authentication'] = pau
sm.registerUtility(pau, IAuthentication)
users = PrincipalFolder()
sm['authentication']['Users'] = users
sm.registerUtility(users, IAuthenticatorPlugin, name="Users")
pau.credentialsPlugins = ( "No Challenge if Authenticated", "Session
Credentials" )

# introduce my own login page
for name, plugin in pau.getCredentialsPlugins():
#iterate over all credential plugin tupels
if hasattr(plugin,'loginpagename'):
plugin.loginpagename = '@@login'

This code should bring the functionality that an unauthorized user will jump
to the login page (@@login).

I addressed without a former login the following URL

http://127.0.0.1:8080/test11/@@test

where test11 is my custom site-manager-instance and @@test a empty
test-view just
containing a pdb in __call__.

class testPage(BrowserPagelet):
"""test pagelet"""

def __call__(self):
import pdb; pdb.set_trace()

def update(self):
import pdb; pdb.set_trace()

I end up with
URL: http://127.0.0.1:8080/test11/%40%40test
Module paste.evalexception.middleware:*306* in |respond|

|<<  *try**:*
__traceback_supplement__ *=*
errormiddleware*.*Supplement*,* self*,* environ
app_iter *=* self*.*application*(*environ*,*
detect_start_response*)*
*try**:*
return_iter *=* list*(*app_iter*)*||>>
app_iter *=*
self*.*application*(*environ*,* detect_start_response*)*|
Module paste.translogger:*68* in |__call__|

|<<  
self.write_log(environ, method, req_uri, start, status, bytes)
return start_response(status, headers)
return self.application(environ, replacement_start_response)

def write_log(self, environ, method, req_uri, start, status,
bytes):||>>  *return*
self*.*application*(*environ*,* replacement_start_response*)*|
Module zope.app.wsgi:*59* in |__call__|

|<<  handle_errors *=*
environ*.*get*(*'wsgi.handleErrors'*,* self*.*handleErrors*)*

request *=* publish*(*request*,*
handle_errors*=*handle_errors*)*
response *=* request*.*response
# Get logging info from principal for log use||>>
request *=* publish*(*request*,*
handle_errors*=*handle_errors*)*|
Module zope.publisher.publish:*131* in |publish|

|<<  
obj *=*
publication*.*getApplication*(*request*)*
obj *=* request*.*traverse*(*obj*)*

publication*.*afterTraversal*(*request*,* obj*)*||>>
obj *=* request*.*traverse*(*obj*)*|
Module zope.publisher.browser:*556* in |traverse|

|<<  
nsteps *=* *0*
ob*,* add_steps *=*
publication*.*getDefaultTraversal*(*self*,* ob*)*
*while* add_steps*:*
nsteps *+=* len*(*add_steps*)*||>>
ob*,* add_steps *=*
publication*.*getDefaultTraversal*(*self*,* ob*)*|
Module zope.app.publication.browser:*36* in |getDefaultTraversal|
  
|<<  *if*
IBrowserPublisher*.*providedBy*(*ob*)**:*
# ob is already proxied, so the result of calling a
method will be
*return* ob*.*browserDefault*(*request*)*
*else**:*
adapter *=* queryMultiAdapter*(**(*ob*,* request*)**,*
IBrowserPublisher*)*||>>  *return*
ob*.*browserDefault*(*request*)*|
*Unauthorized: (,
'browserDefault', 'zope.ManageContent')

*This trace is correct in so far that the view has the following
permission settings:
  

  
*
*
1) The plugin "Session Credentials" is registered and works.
I know this because the "extractCredentials"-Routine of the plugin is
called and
it is my instance of this plugin because it has its loginpagename set to
"@@login" which is not the default value.

2) But the "challenge"-routine of the plugin which should bring up the
login page is never called.

3) I traced that back into PAU itself. The "unauthorized"-routine in
zope.pluggableauth-1.0.1-py2.6.egg/zope/pluggableauth/authentication.py
which will in turn call the plug

Re: [Zope] long running methods

2010-05-20 Thread Patrick Gerken 
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann
wrote:

>
> hi there!
>
> as our zope2 zodb grows we experience more and more conflicts
> with long running transactions. so i'm going to rewrite these
> methods to commit transactions in batches - here my attempt
> to tackle this problem. the idea is to get a list of objects
> to be modified in this transaction, break it down to batches
> and commit after each batch was processed. to avoid zope's
> retry-on-conflict machinery and still have a chance to retry
> per batch transaction commits, the iteration over one batch
> is guarded by a try:except ConflictError: and a retry logic.
>

Zope already has its own retry mechanism, maybe you might want to
call your long running method multiple times, each time working on the
next batch.

Viele Grüße,

   Patrick
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] long running methods

2010-05-20 Thread Jürgen Herrmann 

On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken
 wrote:
> On Thu, May 20, 2010 at 14:18, Jürgen Herrmann
> wrote:
> 
>>
>> hi there!
>>
>> as our zope2 zodb grows we experience more and more conflicts
>> with long running transactions. so i'm going to rewrite these
>> methods to commit transactions in batches - here my attempt
>> to tackle this problem. the idea is to get a list of objects
>> to be modified in this transaction, break it down to batches
>> and commit after each batch was processed. to avoid zope's
>> retry-on-conflict machinery and still have a chance to retry
>> per batch transaction commits, the iteration over one batch
>> is guarded by a try:except ConflictError: and a retry logic.
>>
> 
> Zope already has its own retry mechanism, maybe you might want to
> call your long running method multiple times, each time working on the
> next batch.
> 
in fact zope's retry machinery is kicking in too often which is a
bad thing if the retried transaction runs for say 30min or so.

having to gather the list of objects outside of zope, doing the
batching there and calling into zope multiple times - i wanted
to avoid that.

i'd like to have all the logic contained in one zope method
for the sake of simplicity on the caller's side (most of these
are kicked off by cron jobs).

jürgen
-- 
>> XLhost.de - eXperts in Linux hosting ® <<

XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany

Geschäftsführer: Volker Geith, Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218

Fon:  +49 (0)800 XLHOSTDE [0800 95467833]
Fax:  +49 (0)800 95467830

WEB:  http://www.XLhost.de
IRC:  #xlh...@irc.quakenet.org
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] long running methods

2010-05-20 Thread Andreas Jung 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Patrick Gerken wrote:
> On Thu, May 20, 2010 at 14:18, Jürgen Herrmann
> mailto:juergen.herrm...@xlhost.de>> wrote:
> 
> 
> hi there!
> 
> as our zope2 zodb grows we experience more and more conflicts
> with long running transactions. so i'm going to rewrite these
> methods to commit transactions in batches - here my attempt
> to tackle this problem. the idea is to get a list of objects
> to be modified in this transaction, break it down to batches
> and commit after each batch was processed. to avoid zope's
> retry-on-conflict machinery and still have a chance to retry
> per batch transaction commits, the iteration over one batch
> is guarded by a try:except ConflictError: and a retry logic.
> 
> 
> Zope already has its own retry mechanism, maybe you might want to
> call your long running method multiple times, each time working on the
> next batch.
> 

There retry applies only for methods called through the web  - but
not when you use bin/instance run or so.

Andreas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkv1RdkACgkQCJIWIbr9KYzAPACgmuwVdknppy0Ilg0MzC6nzWJA
1UoAn2DW/IVPfIjqfC4x6pG0b1IMObVU
=gHpt
-END PGP SIGNATURE-
<>___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] long running methods

2010-05-20 Thread Maik Derstappen, Derstappen IT 
Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
> On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken
>   wrote:
>
>> On Thu, May 20, 2010 at 14:18, Jürgen Herrmann
>> wrote:
>>
>>
>>> hi there!
>>>
>>> as our zope2 zodb grows we experience more and more conflicts
>>> with long running transactions. so i'm going to rewrite these
>>> methods to commit transactions in batches - here my attempt
>>> to tackle this problem. the idea is to get a list of objects
>>> to be modified in this transaction, break it down to batches
>>> and commit after each batch was processed. to avoid zope's
>>> retry-on-conflict machinery and still have a chance to retry
>>> per batch transaction commits, the iteration over one batch
>>> is guarded by a try:except ConflictError: and a retry logic.
>>>
>>>
>> Zope already has its own retry mechanism, maybe you might want to
>> call your long running method multiple times, each time working on the
>> next batch.
>>
>>
> in fact zope's retry machinery is kicking in too often which is a
> bad thing if the retried transaction runs for say 30min or so.
>
> having to gather the list of objects outside of zope, doing the
> batching there and calling into zope multiple times - i wanted
> to avoid that.
>
> i'd like to have all the logic contained in one zope method
> for the sake of simplicity on the caller's side (most of these
> are kicked off by cron jobs).
>
> jürgen
>
take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a 
solution for you.
You can deligate your jobs out of zope and get it back after some time.
zamqp then fire an event and you subscribe this event to handle thie 
results.

with kind regards
Maik Derstappen

-- 


DerstappenI T  Consulting   Tel: +49 ( 341 )   600  13  0 31
Zope/E-Mail/Backup/Monitoring   Mobil:   +49 ( 178 )   861  2833
M a i k   D e r s t a p p e n   Fax: +49 ( 180 ) 5 021 121 90 56
H e r l o ß s o h n s t r  12   Email:  maik.derstap...@derstappen-it.de
0 4 1 5 5   L e i p z i g   Internet:http://www.derstappen-it.de



___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] long running methods

2010-05-20 Thread robert rottermann 
Am 20.05.2010 19:34, schrieb Maik Derstappen, Derstappen IT:
> Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
>> On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken
>>   wrote:
>>
>>> On Thu, May 20, 2010 at 14:18, Jürgen Herrmann
>>> wrote:
>>>
>>>
 hi there!

 as our zope2 zodb grows we experience more and more conflicts
 with long running transactions. so i'm going to rewrite these
 methods to commit transactions in batches - here my attempt
 to tackle this problem. the idea is to get a list of objects
 to be modified in this transaction, break it down to batches
 and commit after each batch was processed. to avoid zope's
 retry-on-conflict machinery and still have a chance to retry
 per batch transaction commits, the iteration over one batch
 is guarded by a try:except ConflictError: and a retry logic.


>>> Zope already has its own retry mechanism, maybe you might want to
>>> call your long running method multiple times, each time working on the
>>> next batch.
>>>
>>>
>> in fact zope's retry machinery is kicking in too often which is a
>> bad thing if the retried transaction runs for say 30min or so.
>>
>> having to gather the list of objects outside of zope, doing the
>> batching there and calling into zope multiple times - i wanted
>> to avoid that.
>>
>> i'd like to have all the logic contained in one zope method
>> for the sake of simplicity on the caller's side (most of these
>> are kicked off by cron jobs).
>>
>> jürgen
>>
> take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a 
> solution for you.
> You can deligate your jobs out of zope and get it back after some time.
> zamqp then fire an event and you subscribe this event to handle thie 
> results.
> 
> with kind regards
> Maik Derstappen
> 
maik,
this looks interesting.
just out of curiosity: how would one alert the user that a long running job (say
after some 10 minutes) has finished.
does the client have to poll?
or is there some "magic" push functionality?

robert
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] long running methods

2010-05-20 Thread Maik Derstappen, Derstappen IT 
Am 20.05.2010 20:04, schrieb robert rottermann:
> Am 20.05.2010 19:34, schrieb Maik Derstappen, Derstappen IT:
>
>> Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
>>  
>>> On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken
>>>wrote:
>>>
>>>
 On Thu, May 20, 2010 at 14:18, Jürgen Herrmann
 wrote:


  
> hi there!
>
> as our zope2 zodb grows we experience more and more conflicts
> with long running transactions. so i'm going to rewrite these
> methods to commit transactions in batches - here my attempt
> to tackle this problem. the idea is to get a list of objects
> to be modified in this transaction, break it down to batches
> and commit after each batch was processed. to avoid zope's
> retry-on-conflict machinery and still have a chance to retry
> per batch transaction commits, the iteration over one batch
> is guarded by a try:except ConflictError: and a retry logic.
>
>
>
 Zope already has its own retry mechanism, maybe you might want to
 call your long running method multiple times, each time working on the
 next batch.


  
>>> in fact zope's retry machinery is kicking in too often which is a
>>> bad thing if the retried transaction runs for say 30min or so.
>>>
>>> having to gather the list of objects outside of zope, doing the
>>> batching there and calling into zope multiple times - i wanted
>>> to avoid that.
>>>
>>> i'd like to have all the logic contained in one zope method
>>> for the sake of simplicity on the caller's side (most of these
>>> are kicked off by cron jobs).
>>>
>>> jürgen
>>>
>>>
>> take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a
>> solution for you.
>> You can deligate your jobs out of zope and get it back after some time.
>> zamqp then fire an event and you subscribe this event to handle thie
>> results.
>>
>> with kind regards
>> Maik Derstappen
>>
>>  
> maik,
> this looks interesting.
> just out of curiosity: how would one alert the user that a long running job 
> (say
> after some 10 minutes) has finished.
> does the client have to poll?
> or is there some "magic" push functionality?
>
> robert
>
hi robert,

zamqp has an process that continuously receive new message from his 
message queue and call a callback method which fire the event, that you 
can subscribe.

see zamqp and carrot docs for more info ;)

regards maik

-- 


DerstappenI T  Consulting   Tel: +49 ( 341 )   600  13  0 31
Zope/E-Mail/Backup/Monitoring   Mobil:   +49 ( 178 )   861  2833
M a i k   D e r s t a p p e n   Fax: +49 ( 180 ) 5 021 121 90 56
H e r l o ß s o h n s t r  12   Email:  maik.derstap...@derstappen-it.de
0 4 1 5 5   L e i p z i g   Internet:http://www.derstappen-it.de



___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] long running methods

2010-05-20 Thread robert rottermann 
Am 20.05.2010 21:01, schrieb Maik Derstappen, Derstappen IT:
> Am 20.05.2010 20:04, schrieb robert rottermann:
>> Am 20.05.2010 19:34, schrieb Maik Derstappen, Derstappen IT:
>>
>>> Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
>>>  
 On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken
wrote:


> On Thu, May 20, 2010 at 14:18, Jürgen Herrmann
> wrote:
>
>
>  
>> hi there!
>>
>> as our zope2 zodb grows we experience more and more conflicts
>> with long running transactions. so i'm going to rewrite these
>> methods to commit transactions in batches - here my attempt
>> to tackle this problem. the idea is to get a list of objects
>> to be modified in this transaction, break it down to batches
>> and commit after each batch was processed. to avoid zope's
>> retry-on-conflict machinery and still have a chance to retry
>> per batch transaction commits, the iteration over one batch
>> is guarded by a try:except ConflictError: and a retry logic.
>>
>>
>>
> Zope already has its own retry mechanism, maybe you might want to
> call your long running method multiple times, each time working on the
> next batch.
>
>
>  
 in fact zope's retry machinery is kicking in too often which is a
 bad thing if the retried transaction runs for say 30min or so.

 having to gather the list of objects outside of zope, doing the
 batching there and calling into zope multiple times - i wanted
 to avoid that.

 i'd like to have all the logic contained in one zope method
 for the sake of simplicity on the caller's side (most of these
 are kicked off by cron jobs).

 jürgen


>>> take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a
>>> solution for you.
>>> You can deligate your jobs out of zope and get it back after some time.
>>> zamqp then fire an event and you subscribe this event to handle thie
>>> results.
>>>
>>> with kind regards
>>> Maik Derstappen
>>>
>>>  
>> maik,
>> this looks interesting.
>> just out of curiosity: how would one alert the user that a long running job 
>> (say
>> after some 10 minutes) has finished.
>> does the client have to poll?
>> or is there some "magic" push functionality?
>>
>> robert
>>
> hi robert,
> 
> zamqp has an process that continuously receive new message from his 
> message queue and call a callback method which fire the event, that you 
> can subscribe.
> 
> see zamqp and carrot docs for more info ;)
> 
> regards maik
> 
thanks maik,
this I expected.
however (I assume) these are all "things" that happen on the server.
how do I push the "all done" event to the client?  the user will probably have
navigated to some different place.

robert

___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] long running methods

2010-05-20 Thread Shane Hathaway 
On 05/20/2010 06:18 AM, Jürgen Herrmann wrote:
> def myLongRunningMethod(self):
>
>BATCH_SIZE = 10
>MAX_CONFLICTS = 3
>
>work_items = [some, work, to , do, ...]
># list of persistent objects to be modified in this method
>
>idx = 0
>while idx<  len(work_items):
>  conflicts = 0
>  try:
>my_batch = work_items[idx, idx+BATCH_SIZE]
>for work_item in my_batch:
>  do_some_work(work_item)
>transaction.commit()
>  except ConflicError:
>conflicts += 1
>if conflicts>  MAX_CONFLICTS:
>  raise
>  else:
>idx += BATCH_SIZE
>
> does this sound like a reasonable approach?

More than that, it looks nearly finished. ;-)  The transaction machinery 
does all the heavy lifting, so you don't need to do much.  Issues to fix:

- Your code resets the "conflicts" counter in every iteration.  You 
obviously didn't intend that.

- You should start each loop with "transaction.begin()" to avoid any 
side effects of transactions started before the long running method was 
called.

- Writing thorough tests for this kind of code is very important.  If 
you don't, it might be a while before you discover that ConflictError 
was misspelled.

Shane
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )