Re: [Zope-dev] SQLAlchemy integration experiment

2008-06-17 Thread Brian Sutherland
On Mon, Jun 16, 2008 at 08:40:24PM +0200, Martijn Faassen wrote:
 Hi there,

 In some earlier discussions a number of approaches to integrate SQLAlchemy 
 into Zope were discussed. Following up on that, I've tried a particular 
 approach that tries to use ScopedSessions with a custom scope that isn't 
 just per thread, but also per site (application). The benefit of this 
 approach is that it should allow the following to just work:

 from z3c.sa_integration import Session

 ...
 def somewhere_in_a_view(self):
 session = Session()
 return session.query(Test).all()

For some reason this raises a warning bell in my head. I keep on
thinking: this is zope, the session is a classic case for a utility, we
should be getting it in views by an interface.

That may be because I've already become too fixed in the way I think
about things.

However, Laurence's point about database re-connection is relevant here.
If the Session were looked up by an interface, someone could implement
seamless database re-connection (lots of complex code) without polluting
zope.sqlalchemy.

 where 'Session' is a custom SA scoped session.

 I've checked in my (documented) experiment in here:

 svn://svn.zope.org/repos/main/Sandbox/faassen/rdbintegration/trunk

 The interesting bits are here:

 http://svn.zope.org/Sandbox/faassen/rdbintegration/trunk/src/rdbintegration/app.py

This looks very nice. I'm not sure IDatabase.engine is necessary, it
seems to be only used internally to Database. Also, any view code
should really be getting the connection directly from the session.

 See the documentation in that file for more information.

 The one bit I'm not happy about yet is the way engines need to be global 
 utilities now. I'd like it to be possible for users to edit the database 
 connection parameters through the web. I have a sketchy idea about how to 
 solve this by registering engines as non-persistent local utilities, but 
 it's still unclear how they'd get created at the right time. It might also 
 be possible to simply implement the engine method on IDatabase to create 
 the engine when called the first time, and then cache it in a volatile 
 property. I think this is something Laurence Rowe has been talking about. 
 I'm not sure whether it's a good idea to rely on the ZODB cache, but it 
 might be possible to simply cache the engines in a global registry...

 Comments? Suggestions?

As Laurence has suggested, the session persists, so
IDatabase.configuration will only be called once per thread. Not sure if
creating one new engine per thread is bad.


 Regards,

 Martijn

 ___
 Zope-Dev maillist  -  Zope-Dev@zope.org
 http://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martin Aspeli

Brian Sutherland wrote:

On Mon, Jun 16, 2008 at 08:40:24PM +0200, Martijn Faassen wrote:

Hi there,

In some earlier discussions a number of approaches to integrate SQLAlchemy 
into Zope were discussed. Following up on that, I've tried a particular 
approach that tries to use ScopedSessions with a custom scope that isn't 
just per thread, but also per site (application). The benefit of this 
approach is that it should allow the following to just work:


from z3c.sa_integration import Session

...
def somewhere_in_a_view(self):
session = Session()
return session.query(Test).all()


For some reason this raises a warning bell in my head. I keep on
thinking: this is zope, the session is a classic case for a utility, we
should be getting it in views by an interface.


FWIW, I had the same though.

I think there's a trade-off here: we can use patterns that SQLAlchemy 
and Pylons and others use directly (use a global that isn't actually 
global) or we can use patterns that are ubiquitous in Zope (look up 
utilities by interface).


To my mind, the latter is better because it makes us internally 
consistent, and because it promotes one of the formalisms that IMHO 
makes Zope 3 easier to work with.


Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen

Brian Sutherland wrote:

On Mon, Jun 16, 2008 at 08:40:24PM +0200, Martijn Faassen wrote:

[snip]

from z3c.sa_integration import Session

...
def somewhere_in_a_view(self):
session = Session()
return session.query(Test).all()


For some reason this raises a warning bell in my head. I keep on
thinking: this is zope, the session is a classic case for a utility, we
should be getting it in views by an interface.


Well, it's also SQLAlchemy, where people get a session this way. That 
said, integrating the two frameworks produces lots of questions about 
which framework takes precedence. I've chosen to try to make the 
SQLAlchemy examples work in the way I interpret them they should work, 
and try to use SQLAlchemy infrastructure as much as possible, but this 
is certainly not the only choice.



That may be because I've already become too fixed in the way I think
about things.

However, Laurence's point about database re-connection is relevant here.
If the Session were looked up by an interface, someone could implement
seamless database re-connection (lots of complex code) without polluting
zope.sqlalchemy.


And what is seamless database re-connection and why is this a job for 
Zope and not for SQLAlchemy?



where 'Session' is a custom SA scoped session.

I've checked in my (documented) experiment in here:

svn://svn.zope.org/repos/main/Sandbox/faassen/rdbintegration/trunk

The interesting bits are here:

http://svn.zope.org/Sandbox/faassen/rdbintegration/trunk/src/rdbintegration/app.py


This looks very nice. I'm not sure IDatabase.engine is necessary, it
seems to be only used internally to Database. 


Yes, but it is policy someone may want to change. In this case I 
implemented the policy to look up the engine by name, but it might be 
the default policy would be to simply look up IEngine locally or something.



Also, any view code  should really be getting the connection directly from the 
session.


I don't understand that what you mean by 'connection'. Oh, you mean 
which engine is used? That's really just some testing code, I imagine it 
could directly manipulate the session in realer code.


[snip]

As Laurence has suggested, the session persists, so
IDatabase.configuration will only be called once per thread. Not sure if
creating one new engine per thread is bad.


I don't know either. I've assumed that one wouldn't want to recreate the 
engine for each thread, but perhaps that's fine and we don't break any, 
say, connection pooling mechanisms that way? I do not know.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen

Martin Aspeli wrote:

Brian Sutherland wrote:

[snip]

For some reason this raises a warning bell in my head. I keep on
thinking: this is zope, the session is a classic case for a utility, we
should be getting it in views by an interface.


FWIW, I had the same though.

I think there's a trade-off here: we can use patterns that SQLAlchemy 
and Pylons and others use directly (use a global that isn't actually 
global) or we can use patterns that are ubiquitous in Zope (look up 
utilities by interface).


To my mind, the latter is better because it makes us internally 
consistent, and because it promotes one of the formalisms that IMHO 
makes Zope 3 easier to work with.


The former integrates smoothly with SQLAlchemy. It also is closer to the 
SQLAlchemy documentation. It's also quite likely that someone writing a 
larger application that does use the interface lookup pattern will get 
bored and write something like:


def Session():
   return component.getUtility(ISession)

The Zope component architecture is not about seeing explicit calls into 
it everywhere. That's not the point of it. The point of it is about 
making applications more flexible by allowing people to plug in 
components. My approach allows you to do that.


Anyway, the balance can come out somewhere else. People are free to 
write their own integration approaches, it's just that mine is actually 
 about trying to make exactly this pattern work in the first place. 
Then when I succeed people want it changed. :) Anyway, no surprise: I 
knew that some want other patterns, and I'll be curious to see the other 
approaches as well.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Zope Tests: 5 OK

2008-06-17 Thread Zope Tests Summarizer
Summary of messages to the zope-tests list.
Period Mon Jun 16 11:00:00 2008 UTC to Tue Jun 17 11:00:00 2008 UTC.
There were 5 messages: 5 from Zope Tests.


Tests passed OK
---

Subject: OK : Zope-2.8 Python-2.3.6 : Linux
From: Zope Tests
Date: Mon Jun 16 20:55:01 EDT 2008
URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009715.html

Subject: OK : Zope-2.9 Python-2.4.4 : Linux
From: Zope Tests
Date: Mon Jun 16 20:56:31 EDT 2008
URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009716.html

Subject: OK : Zope-2.10 Python-2.4.4 : Linux
From: Zope Tests
Date: Mon Jun 16 20:58:01 EDT 2008
URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009717.html

Subject: OK : Zope-2.11 Python-2.4.4 : Linux
From: Zope Tests
Date: Mon Jun 16 20:59:31 EDT 2008
URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009718.html

Subject: OK : Zope-trunk Python-2.4.4 : Linux
From: Zope Tests
Date: Mon Jun 16 21:01:01 EDT 2008
URL: http://mail.zope.org/pipermail/zope-tests/2008-June/009719.html

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [Checkins] SVN: zope.app.container/trunk/src/zope/app/container/browser/adding.py 1 include per line

2008-06-17 Thread Benji York
On Mon, Jun 16, 2008 at 8:23 AM, Christophe Combelles [EMAIL PROTECTED] wrote:
 Benji York a écrit :

 On Mon, Jun 16, 2008 at 4:42 AM, Christophe Combelles [EMAIL PROTECTED]
 wrote:

 Log message for revision 87419:
  1 include per line

 Why's that?

 Not extremely important, but it's better for diffs and it allows
 easy-sorting of import lines. (and it's pep8)

It's not important (i.e., has little value), and not the status quo,
that's why I brought it up.

It's also not prescribed in PEP-8.  In fact PEP-328
(http://www.python.org/dev/peps/pep-0328/#rationale-for-parentheses) was
introduced -- in part -- to make it easier to import multiple things
from a module.

I don't think the change should be reverted, but I also wouldn't want
this style to be promulgated.
-- 
Benji York
Senior Software Engineer
Zope Corporation
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Brian Sutherland
On Tue, Jun 17, 2008 at 11:52:49AM +0200, Martijn Faassen wrote:
 However, Laurence's point about database re-connection is relevant here.
 If the Session were looked up by an interface, someone could implement
 seamless database re-connection (lots of complex code) without polluting
 zope.sqlalchemy.

 And what is seamless database re-connection and why is this a job for Zope 
 and not for SQLAlchemy?

I mean in the situation where you change the database connection
parameters on an existing connection and don't want to restart zope to
use the new connection parameters.

Why people seem to want the above escapes me, but people do seem to want
it.

I totally agree that Zope should not be responsible for re-connecting on
error.

 where 'Session' is a custom SA scoped session.

 I've checked in my (documented) experiment in here:

 svn://svn.zope.org/repos/main/Sandbox/faassen/rdbintegration/trunk

 The interesting bits are here:

 http://svn.zope.org/Sandbox/faassen/rdbintegration/trunk/src/rdbintegration/app.py

 This looks very nice. I'm not sure IDatabase.engine is necessary, it
 seems to be only used internally to Database. 

 Yes, but it is policy someone may want to change. In this case I 
 implemented the policy to look up the engine by name, but it might be the 
 default policy would be to simply look up IEngine locally or something.

Just commenting that IDatabase.engine is not used by any code external
to the Database object. There's no use case for it to be public.

 Also, any view code  should really be getting the connection directly from 
 the session.

 I don't understand that what you mean by 'connection'. Oh, you mean which 
 engine is used? That's really just some testing code, I imagine it could 
 directly manipulate the session in realer code.

I was more thinking if you wanted to execute raw sql from a view class
you would do that via the engine.  But it seems that the session has an
execute() method for that.

 [snip]
 As Laurence has suggested, the session persists, so
 IDatabase.configuration will only be called once per thread. Not sure if
 creating one new engine per thread is bad.

 I don't know either. I've assumed that one wouldn't want to recreate the 
 engine for each thread, but perhaps that's fine and we don't break any, 
 say, connection pooling mechanisms that way? I do not know.

Unfortunately I don't either...


 Regards,

 Martijn

 ___
 Zope-Dev maillist  -  Zope-Dev@zope.org
 http://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen
Hi there,

On Tue, Jun 17, 2008 at 2:30 PM, Brian Sutherland
[EMAIL PROTECTED] wrote:
[snip]
 Just commenting that IDatabase.engine is not used by any code external
 to the Database object. There's no use case for it to be public.

While I am not sure we have a strong use case for engine anyway, I
like methods that people can implement to affect behavior (especially
when subclassing) to be public. I don't feel comfortable implementing
something that starts with an underscore.

[snip]
 [snip]
 As Laurence has suggested, the session persists, so
 IDatabase.configuration will only be called once per thread. Not sure if
 creating one new engine per thread is bad.

 I don't know either. I've assumed that one wouldn't want to recreate the
 engine for each thread, but perhaps that's fine and we don't break any,
 say, connection pooling mechanisms that way? I do not know.

 Unfortunately I don't either...

I'll ask Mike Bayer.

Regards,

Martijn
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Brian Sutherland
On Tue, Jun 17, 2008 at 11:58:56AM +0200, Martijn Faassen wrote:
 Anyway, the balance can come out somewhere else. People are free to write 
 their own integration approaches, it's just that mine is actually  about 
 trying to make exactly this pattern work in the first place. Then when I 
 succeed people want it changed. :) Anyway, no surprise: I knew that some 
 want other patterns, and I'll be curious to see the other approaches as 
 well.

Hmm, how about this approach. It builds on and is almost compatible
with yours, but allows me a plug in point where I need it. My use case
is the simplest in that I have no local utilities and no need of
application scoped sessions (plain old thread scoping works just fine).

But I would like to have the option of using code others write for
zope/sqlalchemy and allow others to use our code.

The total code I would write in zope.sqlalchemy would be this:

class ISession(Interface):

def __call__():
Return a SQLAlchemy Session object

def Session():
return getUtility(ISession)()

I would use it in view code like:

from zope.sqlalchemy import Session

class View:

def some_view_method(self):
session = Session()
session.do_stuff()

I do believe that the application scoping approach you showed us could
be implemented by registering your session as the ISession utility.

It may be that zope.sqlalchemy provides different implementations of
sessions, but I would prefer to see no ISession utility actually
registered.

 Regards,

 Martijn

 ___
 Zope-Dev maillist  -  Zope-Dev@zope.org
 http://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen
Hi there,

Just as some context: I'm not proposing to extend zope.sqlalchemy at
all. I'm proposing to write an extension that has the configuration
pattern I sketched out. I also don't intend to write the last
SQLAlchemy integration layer; I wouldn't be so presumptious. :) That
said, if we can come up with something we all feel comfortable with,
hopefully the configuration package will be used by others as well.

On Tue, Jun 17, 2008 at 3:36 PM, Brian Sutherland
[EMAIL PROTECTED] wrote:
[snip]
 Hmm, how about this approach. It builds on and is almost compatible
 with yours, but allows me a plug in point where I need it. My use case
 is the simplest in that I have no local utilities and no need of
 application scoped sessions (plain old thread scoping works just fine).

 But I would like to have the option of using code others write for
 zope/sqlalchemy and allow others to use our code.

 The total code I would write in zope.sqlalchemy would be this:

class ISession(Interface):

def __call__():
Return a SQLAlchemy Session object

def Session():
return getUtility(ISession)()

 I would use it in view code like:

from zope.sqlalchemy import Session

class View:

def some_view_method(self):
session = Session()
session.do_stuff()

 I do believe that the application scoping approach you showed us could
 be implemented by registering your session as the ISession utility.

I'm not sure it makes sense to add an extra utility lookup each time I
get a session. Right now the scoped session just gets the thing if
it's already cached, and there's already a local utility lookup to get
the scope UID.

What about this?

class IDatabase(Interface):
def scopefunc():
  The scopefunc

def session_factory():
  The session factory

def scopefunc():
 util = component.getUtility(IDatabase)
 return util.scopefunc()

This will allow you to implement a global IDatabase utility that's
global and uses a thread-local scope only.
Similarly, we can delegate session creation to the IDatabase utility completely:

def session_factory():
   util = component.getUtility(IDatabase)
   return util.session_factory()

I think that's better than what I have now, as you can then completely
control session creation through a utility. It doesn't add a new
plugin point or utility lookup, but allows you to implement your use
case, right? We'll just need to implement a number of utilities that
fulfill the various use cases.

Regards,

Martijn
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Laurence Rowe

Martijn Faassen wrote:

Hi there,

On Tue, Jun 17, 2008 at 2:30 PM, Brian Sutherland
[EMAIL PROTECTED] wrote:
[snip]

Just commenting that IDatabase.engine is not used by any code external
to the Database object. There's no use case for it to be public.


While I am not sure we have a strong use case for engine anyway, I
like methods that people can implement to affect behavior (especially
when subclassing) to be public. I don't feel comfortable implementing
something that starts with an underscore.


Why not just have:

class IDatabase(Interface):
A utility that specifies the database.


def session_factory():
Create a new session


def id():
Get unique id for this database configuration.

This should be unique per site (application).



class Database(grok.LocalUtility):
grok.implements(IDatabase)

def session_factory(self):
engine = create_engine(
'postgres:///experiment',
convert_unicode=True)
return create_session(
bind=engine,
autocommit=True,
autoflush=True,
extension=ZopeTransactionExtension())

def id(self):
# we use the application name as the unique id. Can we use
# something more clever and universally working?
return self.__parent__.__name__


[snip]

[snip]

As Laurence has suggested, the session persists, so
IDatabase.configuration will only be called once per thread. Not sure if
creating one new engine per thread is bad.

I don't know either. I've assumed that one wouldn't want to recreate the
engine for each thread, but perhaps that's fine and we don't break any,
say, connection pooling mechanisms that way? I do not know.

Unfortunately I don't either...


I'm not sure connection pooling is really useful in a threaded 
environment with recycled sessions. You want n threads = n connections. 
If we started creating new sessions each request then things would be 
different.


Laurence

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen

Hey Laurence,

Laurence Rowe wrote:
[snip]
I'm not sure connection pooling is really useful in a threaded 
environment with recycled sessions. You want n threads = n connections. 
If we started creating new sessions each request then things would be 
different.


Mike Bayer says the following:

 you should reuse a single engine.  It contains a pool of connections,
 so recreating engines means new connections are constantly rebuilt and
 it defeats the purpose of the pool.  Other than that the creation of
 an engine is not very expensive but even a small expense is needless
 here.

I explained to him we'd only end up creating a single engine per thread 
(and app), and we'll see what he has to say about that.


I'm leaning towards trying to recycle the engines, though.

Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Brian Sutherland
On Tue, Jun 17, 2008 at 05:25:56PM +0200, Brian Sutherland wrote:
  class IDatabase(Interface):
  def scopefunc():
The scopefunc
  
  def session_factory():
The session factory
  
  def scopefunc():
   util = component.getUtility(IDatabase)
   return util.scopefunc()
  
  This will allow you to implement a global IDatabase utility that's
  global and uses a thread-local scope only.
  Similarly, we can delegate session creation to the IDatabase utility 
  completely:
  
  def session_factory():
 util = component.getUtility(IDatabase)
 return util.session_factory()
 
 I think you missed the final piece:
 
 Session = scoped_session(session_factory, scopefunc=scopefunc)
 
  I think that's better than what I have now, as you can then completely
  control session creation through a utility. 
 
 Yep
 
  It doesn't add a new
  plugin point or utility lookup, but allows you to implement your use
  case, right? We'll just need to implement a number of utilities that
  fulfill the various use cases.
 
 Yes, it looks pretty trivial to plug in a normal sqlalchemy session.

This would probably be close to what I would write for my usecase:

class Database:

implements(IDatabase)

def __init__(self, *args, **kw):
self._args = args
self._kw = kw

def scopefunc(self):
return None # use default per-thread scoping

def session_factory(self):
return sessionmaker(*self._args, **self._kw)

-- 
Brian Sutherland
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen

Brian Sutherland wrote:
[snip]

This would probably be close to what I would write for my usecase:

class Database:

implements(IDatabase)

def __init__(self, *args, **kw):
self._args = args
self._kw = kw

def scopefunc(self):
return None # use default per-thread scoping

def session_factory(self):
return sessionmaker(*self._args, **self._kw)


I don't think you can use sessionmaker, as that creates a class and you 
need to create an actual session in this place. You'd need 
create_session. You'd also need to implement your scopefunc, otherwise 
you get *no* scoping at all, not even per thread, so return 
thread.get_ident().


Like this:

class Database:
implements(IDatabase)

def __init__(self, *args, **kw):
self._args = args
self._kw = kw

def scopefunc(self):
return thread.get_ident()

def session_factory(self):
return create_session(*self._args, **self._kw)

we're ignoring the details of what creates the engine, but if you pass 
'bind' along when you create Database that should take care of it.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen

Hey,


[replying to myself]

Martijn Faassen wrote:

Laurence Rowe wrote:
[snip]
I'm not sure connection pooling is really useful in a threaded 
environment with recycled sessions. You want n threads = n 
connections. If we started creating new sessions each request then 
things would be different.


Mike Bayer says the following:

  you should reuse a single engine.  It contains a pool of connections,
  so recreating engines means new connections are constantly rebuilt and
  it defeats the purpose of the pool.  Other than that the creation of
  an engine is not very expensive but even a small expense is needless
  here.

I explained to him we'd only end up creating a single engine per thread 
(and app), and we'll see what he has to say about that.


Mike responded like this:

 yeah that also kind of defeats the connection pool's purpose as it's
 intended to handle all connections for a particular database across
 threads.  Having a single point of pooling for a certain database has
 the advantage that you can tune the total number of connections at a
 single configuration point, and also that a database restart can be
 detected just once, resulting in a bounce of the entire pool (this is
 a distinct advantage of using a pool over a non-pooled approach).

So, it looks like we need to make sure we maintain a single engine per 
database.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Laurence Rowe
2008/6/17 Michael Bayer [EMAIL PROTECTED]:

 On Jun 17, 2008, at 10:09 AM, Laurence Rowe wrote:

 I'm not sure connection pooling is really useful in a threaded environment
 with recycled sessions. You want n threads = n connections. If we started
 creating new sessions each request then things would be different.


 For this to be efficient in SQLA (if there were no pool), you'd need to bind
 each Session to a Connection individually.If you create_engine() using a
 pool like NullPool then you'd get an unpooled engine that creates new
 connections on each checkout. In theory this would be fine, although we
 don't have any current test coverage for the auto-reconnect logic using
 NullPool with persistently-checked-out Connections so we'd have to build
 that up and ensure its working.

 But if you are binding Sessions directly to Connections, theres still not
 really any disadvantage to the usual QueuePool implementation remaining in
 place.   If you set it with max_overflow=-1 it will just hand out any number
 of connections.

 I still prefer the Session bound to Engine approach, whereby each
 transaction uses a pooled connection, and returns the connection on
 rollback/commit.   That way you don't need to worry about stale connections
 hanging around for sessions/threads that fall out of use.

What connection pooling is used by default? e.g. with
create_engine('sqlite:///:memory:')

I think we are only talking about the difference between using four
pools of one connection versus one pool of four connections (assuming
the standard four threads in zope). I don't see that making a lot of
difference in practice.

Laurence
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Michael Bayer


On Jun 17, 2008, at 12:41 PM, Laurence Rowe wrote:



What connection pooling is used by default? e.g. with
create_engine('sqlite:///:memory:')


sqlite is a special case, it uses the SingletonThreadPool.  This pool  
holds onto one connection per thread.This is used in SQLite  
because of a sometimes-restriction that a sqlite connection can only  
be used in the same thread in which it was created.  The pool in  
normal use is QueuePool.



I think we are only talking about the difference between using four
pools of one connection versus one pool of four connections (assuming
the standard four threads in zope). I don't see that making a lot of
difference in practice.


in practice, a single pool of four connections means if one of those  
connections encounters a connection lost exception, the exception is  
raised, and the entire pool is recycled; meaning that only one  
exception is raised for the whole application during a database  
restart.   It also means that the total connections used by the  
application for a particular database can be configured/throttled in  
one place.   To me thats a significant difference.


___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen

Hey,

Michael Bayer wrote:
[snip comments on connection pooling]

Thanks for showing up here and participating in the discussion, Michael. 
To me it's clear we should try to reuse engines as much as we can.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martin Aspeli

Martijn Faassen wrote:

Martin Aspeli wrote:

Brian Sutherland wrote:

[snip]

For some reason this raises a warning bell in my head. I keep on
thinking: this is zope, the session is a classic case for a utility, we
should be getting it in views by an interface.

FWIW, I had the same though.

I think there's a trade-off here: we can use patterns that SQLAlchemy 
and Pylons and others use directly (use a global that isn't actually 
global) or we can use patterns that are ubiquitous in Zope (look up 
utilities by interface).


To my mind, the latter is better because it makes us internally 
consistent, and because it promotes one of the formalisms that IMHO 
makes Zope 3 easier to work with.


The former integrates smoothly with SQLAlchemy.


Sure, that's why I said trade-off. :)

It also is closer to the 
SQLAlchemy documentation.


This is a good point. But is it the same or almost the same? If it's 
the latter, the explicitly different is better than faux identity.


It's also quite likely that someone writing a 
larger application that does use the interface lookup pattern will get 
bored and write something like:


def Session():
return component.getUtility(ISession)


Mmmm, I'm not so sure, but maybe.

The Zope component architecture is not about seeing explicit calls into 
it everywhere. That's not the point of it. The point of it is about 
making applications more flexible by allowing people to plug in 
components. My approach allows you to do that.


Sure, but I also think that the CA has given us a few very basic, very 
flexible idioms (adapters, utilities, subscribers) that permeate any and 
all zope 3 applications. Internal consistency is a very good thing in a 
framework.


A not-quite-global Session variable is another pattern to achieve what 
we do elsewhere with global unnamed utilities when we use them as 
effective singletons. Having two patterns to do the same thing is not good.


Put the question another way - a new user may ask, why don't I have a 
global thing with a capital first letter like Session to look up other 
singletons?.


Anyway, the balance can come out somewhere else. People are free to 
write their own integration approaches, it's just that mine is actually 
  about trying to make exactly this pattern work in the first place. 
Then when I succeed people want it changed. :) Anyway, no surprise: I 
knew that some want other patterns, and I'll be curious to see the other 
approaches as well.


I'm not saying it's wrong, and I do think natural SQLAlchemy is a 
strong selling point, since I'd wager there are more SQLAlchemy users 
than Zope 3 users out there. I just think we need to be careful about 
what patterns we promote and how it fits with the rest of our story.


Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Laurence Rowe

Martin Aspeli wrote:

Martijn Faassen wrote:

Martin Aspeli wrote:

Brian Sutherland wrote:

[snip]

For some reason this raises a warning bell in my head. I keep on
thinking: this is zope, the session is a classic case for a utility, we
should be getting it in views by an interface.

FWIW, I had the same though.

I think there's a trade-off here: we can use patterns that SQLAlchemy 
and Pylons and others use directly (use a global that isn't 
actually global) or we can use patterns that are ubiquitous in Zope 
(look up utilities by interface).


To my mind, the latter is better because it makes us internally 
consistent, and because it promotes one of the formalisms that IMHO 
makes Zope 3 easier to work with.


The former integrates smoothly with SQLAlchemy.


Sure, that's why I said trade-off. :)


It also is closer to the SQLAlchemy documentation.


This is a good point. But is it the same or almost the same? If it's 
the latter, the explicitly different is better than faux identity.


It's also quite likely that someone writing a larger application that 
does use the interface lookup pattern will get bored and write 
something like:


def Session():
return component.getUtility(ISession)


Mmmm, I'm not so sure, but maybe.

The Zope component architecture is not about seeing explicit calls 
into it everywhere. That's not the point of it. The point of it is 
about making applications more flexible by allowing people to plug in 
components. My approach allows you to do that.


Sure, but I also think that the CA has given us a few very basic, very 
flexible idioms (adapters, utilities, subscribers) that permeate any and 
all zope 3 applications. Internal consistency is a very good thing in a 
framework.


A not-quite-global Session variable is another pattern to achieve what 
we do elsewhere with global unnamed utilities when we use them as 
effective singletons. Having two patterns to do the same thing is not good.


Put the question another way - a new user may ask, why don't I have a 
global thing with a capital first letter like Session to look up other 
singletons?.


Anyway, the balance can come out somewhere else. People are free to 
write their own integration approaches, it's just that mine is 
actually   about trying to make exactly this pattern work in the first 
place. Then when I succeed people want it changed. :) Anyway, no 
surprise: I knew that some want other patterns, and I'll be curious to 
see the other approaches as well.


I'm not saying it's wrong, and I do think natural SQLAlchemy is a 
strong selling point, since I'd wager there are more SQLAlchemy users 
than Zope 3 users out there. I just think we need to be careful about 
what patterns we promote and how it fits with the rest of our story.


I don't think scoped sessions really map well to a utility:

   session = getUtility(ISession)()

is just too ugly. Better to stick with Session() or ISession(context)

Laurence

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martin Aspeli

Laurence Rowe wrote:

Martin Aspeli wrote:

Martijn Faassen wrote:

Martin Aspeli wrote:

Brian Sutherland wrote:

[snip]

For some reason this raises a warning bell in my head. I keep on
thinking: this is zope, the session is a classic case for a utility, we
should be getting it in views by an interface.

FWIW, I had the same though.

I think there's a trade-off here: we can use patterns that SQLAlchemy 
and Pylons and others use directly (use a global that isn't 
actually global) or we can use patterns that are ubiquitous in Zope 
(look up utilities by interface).


To my mind, the latter is better because it makes us internally 
consistent, and because it promotes one of the formalisms that IMHO 
makes Zope 3 easier to work with.

The former integrates smoothly with SQLAlchemy.

Sure, that's why I said trade-off. :)


It also is closer to the SQLAlchemy documentation.
This is a good point. But is it the same or almost the same? If it's 
the latter, the explicitly different is better than faux identity.


It's also quite likely that someone writing a larger application that 
does use the interface lookup pattern will get bored and write 
something like:


def Session():
return component.getUtility(ISession)

Mmmm, I'm not so sure, but maybe.

The Zope component architecture is not about seeing explicit calls 
into it everywhere. That's not the point of it. The point of it is 
about making applications more flexible by allowing people to plug in 
components. My approach allows you to do that.
Sure, but I also think that the CA has given us a few very basic, very 
flexible idioms (adapters, utilities, subscribers) that permeate any and 
all zope 3 applications. Internal consistency is a very good thing in a 
framework.


A not-quite-global Session variable is another pattern to achieve what 
we do elsewhere with global unnamed utilities when we use them as 
effective singletons. Having two patterns to do the same thing is not good.


Put the question another way - a new user may ask, why don't I have a 
global thing with a capital first letter like Session to look up other 
singletons?.


Anyway, the balance can come out somewhere else. People are free to 
write their own integration approaches, it's just that mine is 
actually   about trying to make exactly this pattern work in the first 
place. Then when I succeed people want it changed. :) Anyway, no 
surprise: I knew that some want other patterns, and I'll be curious to 
see the other approaches as well.
I'm not saying it's wrong, and I do think natural SQLAlchemy is a 
strong selling point, since I'd wager there are more SQLAlchemy users 
than Zope 3 users out there. I just think we need to be careful about 
what patterns we promote and how it fits with the rest of our story.


I don't think scoped sessions really map well to a utility:

session = getUtility(ISession)()

is just too ugly. Better to stick with Session() or ISession(context)


Agreed.

Why can't it just be

getUtility(ISession)

?

Note that I'm only playing devil's advocate here; I don't feel very 
strongly about this.


Cheers,
Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: SQLAlchemy integration experiment

2008-06-17 Thread Martijn Faassen

Martin Aspeli wrote:

Martijn Faassen wrote:

[snip]
The Zope component architecture is not about seeing explicit calls 
into it everywhere. That's not the point of it. The point of it is 
about making applications more flexible by allowing people to plug in 
components. My approach allows you to do that.


Sure, but I also think that the CA has given us a few very basic, very 
flexible idioms (adapters, utilities, subscribers) that permeate any and 
all zope 3 applications. Internal consistency is a very good thing in a 
framework.


A not-quite-global Session variable is another pattern to achieve what 
we do elsewhere with global unnamed utilities when we use them as 
effective singletons. Having two patterns to do the same thing is not good.


It's not quite the same. While a utility can be scoped per application, 
the Session class also takes care of scoping per thread. I'm not sure 
whether utilities can be convinced to do so (sites are local per thread, 
but utilities?).


SA's ScopedSession also takes care of creating and caching individual 
sessions for each thread (and application), something utilities don't 
really do either. So you'd still have to use ScopedSession and register 
that as a local utility somehow, or do some componentry I can't wrap my 
head around yet. I have a strong suspicion it'll become a lot more 
complicated than my approach (and not more flexible). I'm trying to go 
with the flow for *both* frameworks. The code is pretty simple right 
now, and this is a good sign.


Put the question another way - a new user may ask, why don't I have a 
global thing with a capital first letter like Session to look up other 
singletons?.


A new user won't ask such a thing, as it looks like he's importing and 
instantiating a class. Classes are singletons, and they're imported. 
I've actually wished for a long time that utility lookups could be made 
to look more like imports. :)


It's a question a very experienced user might ask.

Anyway, the balance can come out somewhere else. People are free to 
write their own integration approaches, it's just that mine is 
actually   about trying to make exactly this pattern work in the first 
place. Then when I succeed people want it changed. :) Anyway, no 
surprise: I knew that some want other patterns, and I'll be curious to 
see the other approaches as well.


I'm not saying it's wrong, and I do think natural SQLAlchemy is a 
strong selling point, since I'd wager there are more SQLAlchemy users 
than Zope 3 users out there. I just think we need to be careful about 
what patterns we promote and how it fits with the rest of our story.


I want to promote the import Session and instantiate it pattern. :)

If people want to override certain behavior of session creation, *then* 
I want to promote the implement and register a utility that provides 
this interface pattern. I hope to supply the user with enough 
pre-implemented IDatabase utilities that this is rare, so that they just 
end up registering them. A simple global one, and a local one that 
allows for flexible engine configuration somehow.


What the Zope 3 integration I propose changes is how Session is set up. 
By using utilities we can offer flexibility to set it up globally or 
locally with ZODB-stored configuration. We use SA's mechanism of 
actually creating and managing sessions. I think that's a good division 
of labor: hook into SA with a bit of Zope 3 component architecture for 
configuration and pluggability, and leave the rest to SA.


Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [Checkins] SVN: zope.app.container/trunk/src/zope/app/container/browser/adding.py 1 include per line

2008-06-17 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Benji York wrote:
 On Mon, Jun 16, 2008 at 8:23 AM, Christophe Combelles [EMAIL PROTECTED] 
 wrote:
 Benji York a écrit :
 On Mon, Jun 16, 2008 at 4:42 AM, Christophe Combelles [EMAIL PROTECTED]
 wrote:
 Log message for revision 87419:
  1 include per line
 Why's that?
 Not extremely important, but it's better for diffs and it allows
 easy-sorting of import lines. (and it's pep8)
 
 It's not important (i.e., has little value), and not the status quo,
 that's why I brought it up.
 
 It's also not prescribed in PEP-8.  In fact PEP-328
 (http://www.python.org/dev/peps/pep-0328/#rationale-for-parentheses) was
 introduced -- in part -- to make it easier to import multiple things
 from a module.
 
 I don't think the change should be reverted, but I also wouldn't want
 this style to be promulgated.

The main advantage of the style is that any change to the imports in a
module creates a full-line diff, which is easeier to read.  The cost is
extra verbosity.  I prefer the full-line diffs myself.


Tres.
- --
===
Tres Seaver  +1 540-429-0999  [EMAIL PROTECTED]
Palladion Software   Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIWF4d+gerLs4ltQ4RAltMAKC/Xn6ZEvgX7koXKGHtcsii0XtiqACfWtHb
LDWIuM+GK4sK9OqXfaQnyN8=
=sZl/
-END PGP SIGNATURE-
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope] UTF8 encoding

2008-06-17 Thread Garry Saddington
I am getting a few of this type of error only coming from Windows machines, 
does anyone know how to sort this out?

ProgrammingError: ERROR: invalid byte sequence for encoding UTF8: 0xa3 HINT: 
This error can also happen if the byte sequence does not match the encoding 
expected by the server, which is controlled by client_encoding. insert into 
departmentplanning (planningcategoryid, taskid, subtasks, moneval, timescale, 
budget, success, subjectid, teacherid) VALUES (1 , 186 , 'Identify suitable 
tasks and activities for all G+T pupils and students ' , 'ADE- liaise with HS 
and GE' , 'JUL08 on' , 'SOme activities could be bought in ~£50' , 'All G+T 
pupils and students given opportunity for enhanced work' , 1787, 41 )


Regards
Garry___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


[Zope] help debugging a can't pickle error deep within a catalog reindex

2008-06-17 Thread Rob Miller

hi,

i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with over 
29,000 indexed objects.  it churns for a good long time, and eventually fails 
with a long set of tracebacks, of which i've included a sample at the end of 
this message.


i think i understand the gist of the issue... it's trying to write an object 
(probably a CatalogBrain) to the database, but this object's __dict__ contains 
a value that is of type instancemethod, which isn't allowed for persistent 
objects.


the problem is that i can't figure out which specific objects are causing the 
problem.  i've used pdb.post_mortem to get a debug prompt way down in the 
traceback, but the code goes in and out of C modules, so i'm missing a lot of 
what's happening.  and when i interactively peek at the objects that are being 
indexed when the error happens, there doesn't seem to be anything wrong, and i 
can index the objects w/ no problem.  i've even tried dropping the 
subtransaction threshold down to 1, so it will try to commit a savepoint after 
every object, but none of the objects being indexed seemed to have any problems.


i CAN verify that the instancemethod that is causing the problem renders like 
this:


bound method SessionDataManager.getSessionData of SessionDataManager at 
/session_data_manager


even that hasn't proven enough for me to concretely identify the source of the 
problem, though.


i've been working on this for a full day already, and am not sure how to 
proceed.  does anyone have any debugging tips that might help me figure out 
what, exactly, is causing the reindex attempts to blow up?


thanks!

-r




Traceback (most recent call last):
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py, 
line 296, in refreshCatalog

self.catalog_object(obj, p, pghandler=pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py, 
line 367, in catalog_object

self._increment_counter()
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py, 
line 395, in _increment_counter

self._counter.change(1)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/BTrees/Length.py, 
line 55, in change

self.value += delta
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py, 
line 890, in register

self._register(obj)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py, 
line 900, in _register

self.transaction_manager.get().join(self)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py, 
line 273, in join

self._prior_operation_failed() # doesn't return
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py, 
line 267, in _prior_operation_failed

raise TransactionFailedError(An operation previously failed, 
TransactionFailedError: An operation previously failed, with traceback:

  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZServer/PubCore/ZServerPublisher.py, 
line 25, in __init__

response=b)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 401, in publish_module

environ, debug, request, response)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 202, in publish_module_standard

response = publish(request, module_name, after_list, debug=debug)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 119, in publish

request, bind=1)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/mapply.py, 
line 88, in mapply

if debug is not None: return debug(object,args,context)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 42, in call_object

result=apply(object,args) # Type scr to step into published object.
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py, 
line 260, in manage_catalogReindex

self.refreshCatalog(clear=1, pghandler=handler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py, 
line 296, in refreshCatalog

self.catalog_object(obj, p, pghandler=pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py, 
line 385, in catalog_object

update_metadata, pghandler=pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch.py, 
line 96, in catalog_object

uid, idxs, update_metadata, pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch_utils.py, 
line 6, in call

return getattr(self, PATTERN % __name__)(*args, **kw)
  File 

[Zope] Re: help debugging a can't pickle error deep within a catalog reindex

2008-06-17 Thread Ross Patterson
Rob Miller [EMAIL PROTECTED] writes:

 hi,

 i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with
 over 29,000 indexed objects.  it churns for a good long time, and
 eventually fails with a long set of tracebacks, of which i've included
 a sample at the end of this message.

 i think i understand the gist of the issue... it's trying to write an
 object (probably a CatalogBrain) to the database, but this object's
 __dict__ contains a value that is of type instancemethod, which isn't
 allowed for persistent objects.

 the problem is that i can't figure out which specific objects are
 causing the problem.  i've used pdb.post_mortem to get a debug prompt
 way down in the traceback, but the code goes in and out of C modules,
 so i'm missing a lot of what's happening.  and when i interactively
 peek at the objects that are being indexed when the error happens,
 there doesn't seem to be anything wrong, and i can index the objects
 w/ no problem.  i've even tried dropping the subtransaction threshold
 down to 1, so it will try to commit a savepoint after every object,
 but none of the objects being indexed seemed to have any problems.

 i CAN verify that the instancemethod that is causing the problem
 renders like this:

 bound method SessionDataManager.getSessionData of SessionDataManager
 at /session_data_manager

 even that hasn't proven enough for me to concretely identify the
 source of the problem, though.

 i've been working on this for a full day already, and am not sure how
 to proceed.  does anyone have any debugging tips that might help me
 figure out what, exactly, is causing the reindex attempts to blow up?

 thanks!

Can you get a pdb.post_mortem prompt at the actual ZODB/serialize.py:416
error or only at transaction/_transaction.py:267 where some sort of
previous error is handled?

If the former, I often find it informative to step up to
ZODB/serialize.py:407 where obj.__getstate__() is called and find out
what obj is.  Is it the same object every time?  If not, is it of the
same type every time?

A next step can also be to put a pdb.set_trace() at
transaction/_transaction.py:340 in the register() method such that it's
is only called when the offending object is added to the transaction.

Hope some of that helps,
Ross

 Traceback (most recent call last):
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py,
 line 296, in refreshCatalog
 self.catalog_object(obj, p, pghandler=pghandler)
   File
 /home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py,
 line 367, in catalog_object
 self._increment_counter()
   File
 /home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py,
 line 395, in _increment_counter
 self._counter.change(1)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/BTrees/Length.py,
 line 55, in change
 self.value += delta
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py,
 line 890, in register
 self._register(obj)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py,
 line 900, in _register
 self.transaction_manager.get().join(self)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py,
 line 273, in join
 self._prior_operation_failed() # doesn't return
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py,
 line 267, in _prior_operation_failed
 raise TransactionFailedError(An operation previously failed, 
 TransactionFailedError: An operation previously failed, with traceback:

   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZServer/PubCore/ZServerPublisher.py,
 line 25, in __init__
 response=b)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py,
 line 401, in publish_module
 environ, debug, request, response)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py,
 line 202, in publish_module_standard
 response = publish(request, module_name, after_list, debug=debug)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py,
 line 119, in publish
 request, bind=1)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/mapply.py,
 line 88, in mapply
 if debug is not None: return debug(object,args,context)
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py,
 line 42, in call_object
 result=apply(object,args) # Type scr to step into published object.
   File
 /home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py,
 line 260, in manage_catalogReindex
 self.refreshCatalog(clear=1, pghandler=handler)
   File
 

Re: [Zope] UTF8 encoding

2008-06-17 Thread Andreas Jung



--On 17. Juni 2008 20:05:35 +0100 Garry Saddington 
[EMAIL PROTECTED] wrote:



I am getting a few of this type of error only coming from Windows
machines, does anyone know how to sort this out?


Sorry but you must provide a detailed description in terms of code, 
tracebacks and a description how to reproduce this error.


-aj

pgp9cNAsrAqdp.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


[Zope-DB] Reply of msg#00083

2008-06-17 Thread Suunil
Dear Ma'am/Sir :

I am interested to know more about your project for Business and Investment. 
Can you please send more details.

Regards,

Suunil ___
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db