[sqlalchemy] Re: detached, pickled sqlalchemy collection object failing after server restart
Well, it appears I have answered my own question and, yes, it was a
SQLAlchemy issue. I resolved the problem by explicitly compiling the
mapping at the end of the model initialization:
manager_mapper = orm.mapper(Manager, manager_table, ... etc.
... (all other dependent mappers)
manager_mapper.compile()
My extremely vague understanding of what is going on is that if you
are explicitly using the SQLAlchemy API to retrieve the pickled object
then compiling is done implicitly but if the API is not used then the
compiling is not done and you run into troubles. I am not sure how
the beaker session is trying to interact with the object that causes
the problem versus my own testing of interacting with the pickled
object being successful but compiling is the answer.
If someone has a more robust explanation I would be very interested to
hear it.
On Oct 21, 10:08 pm, rootsmith ke...@rootsmith.ca wrote:
Hi,
I have a pylons application using file based beaker sessions (not to
be confused with SQLAlchemy Sessions.) I retrieve a mapped object
from the database, detach it from the SQLAlchemy Session like so:
Session.expunge(object)
and save it to the beaker session (basically pickling it to a file.)
like so:
session['user'] = user
session.save()
If I restart the server and retrieve the pickled object from the
session, merge it back into the SQLAlchemy Session everything seems to
be fine.
user = Session.merge(session['user'], dont_load=True)
But then somewhere in the beaker session middleware something goes
awry before it completes the call to the controller.
Now I know this sounds like a beaker session problem but it is failing
when it is making a call to the SQLAlchemy object. Here is the stack
trace:
URL:http://127.0.0.1:5000/
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/
WebError-0.10.1-py2.6.egg/weberror/evalexception.py', line 431 in
respond
app_iter = self.application(environ, detect_start_response)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Paste-1.7.2-
py2.6.egg/paste/recursive.py', line 80 in __call__
return self.application(environ, start_response)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/middleware.py', line 73 in __call__
return self.app(environ, start_response)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/middleware.py', line 152 in __call__
return self.wrap_app(environ, session_start_response)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Routes-1.10.3-
py2.6.egg/routes/middleware.py', line 130 in __call__
response = self.app(environ, start_response)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7-
py2.6.egg/pylons/wsgiapp.py', line 125 in __call__
response = self.dispatch(controller, environ, start_response)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7-
py2.6.egg/pylons/wsgiapp.py', line 324 in dispatch
return controller(environ, start_response)
File '/Users/kevin/Documents/workspace/ryzoe/ryzoe/lib/base.py', line
54 in __call__
return WSGIController.__call__(self, environ, start_response)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7-
py2.6.egg/pylons/controllers/core.py', line 284 in __call__
return response(environ, self.start_response)
File 'build/bdist.macosx-10.6-universal/egg/webob/__init__.py', line
2182 in __call__
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/middleware.py', line 146 in session_start_response
session.persist()
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/session.py', line 600 in persist
self._session().save()
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/session.py', line 293 in save
self.namespace.release_write_lock()
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/container.py', line 149 in release_write_lock
self.close(checkcount=True)
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/container.py', line 172 in close
self.do_close()
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1-
py2.6.egg/beaker/container.py', line 536 in do_close
cPickle.dump(self.hash, fh)
File '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/copy_reg.py', line 84 in _reduce_ex
dict = getstate()
File '/Users/kevin/mydevenv/lib/python2.6/site-packages/
SQLAlchemy-0.5.6-py2.6.egg/sqlalchemy/orm/collections.py', line 607 in
__getstate__
return {'key': self.attr.key,
AttributeError: 'NoneType' object has no attribute 'key'
Here is the mapper of the object that is pickled:
orm.mapper(Manager, manager_table, inherits=User,
polymorphic_identity='manager',
properties={'residences': relation(Residence,
lazy=False,
[sqlalchemy] Re: detached, pickled sqlalchemy collection object failing after server restart
rootsmith wrote: Well, it appears I have answered my own question and, yes, it was a SQLAlchemy issue. I resolved the problem by explicitly compiling the mapping at the end of the model initialization: manager_mapper = orm.mapper(Manager, manager_table, ... etc. ... (all other dependent mappers) manager_mapper.compile() My extremely vague understanding of what is going on is that if you are explicitly using the SQLAlchemy API to retrieve the pickled object then compiling is done implicitly but if the API is not used then the compiling is not done and you run into troubles. I am not sure how the beaker session is trying to interact with the object that causes the problem versus my own testing of interacting with the pickled object being successful but compiling is the answer. If someone has a more robust explanation I would be very interested to hear it. that seems like a tricky error to reproduce. When your application starts, assuming it sets up mappers immediately, as soon as they are used all mappers are compiled. pickling/depickling wouldn't have any compilation related issues. Additionally your error seems to occur during *serialization*, which is very strange. Its not occurring to me how your application would have a hold of a mapped instance without your mappers having been compiled...unless that object was pulled from a cache previously. In any case if you throw a compile_mappers() at the end of environment.py load_environment() you shouldn't have this kind of issue. but i would be curious how you're getting it to do that. On Oct 21, 10:08 pm, rootsmith ke...@rootsmith.ca wrote: Hi, I have a pylons application using file based beaker sessions (not to be confused with SQLAlchemy Sessions.) I retrieve a mapped object from the database, detach it from the SQLAlchemy Session like so: Session.expunge(object) and save it to the beaker session (basically pickling it to a file.) like so: session['user'] = user session.save() If I restart the server and retrieve the pickled object from the session, merge it back into the SQLAlchemy Session everything seems to be fine. user = Session.merge(session['user'], dont_load=True) But then somewhere in the beaker session middleware something goes awry before it completes the call to the controller. Now I know this sounds like a beaker session problem but it is failing when it is making a call to the SQLAlchemy object. Here is the stack trace: URL:http://127.0.0.1:5000/ File '/Users/kevin/mydevenv/lib/python2.6/site-packages/ WebError-0.10.1-py2.6.egg/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Paste-1.7.2- py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Routes-1.10.3- py2.6.egg/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7- py2.6.egg/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7- py2.6.egg/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/Users/kevin/Documents/workspace/ryzoe/ryzoe/lib/base.py', line 54 in __call__ return WSGIController.__call__(self, environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7- py2.6.egg/pylons/controllers/core.py', line 284 in __call__ return response(environ, self.start_response) File 'build/bdist.macosx-10.6-universal/egg/webob/__init__.py', line 2182 in __call__ File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/middleware.py', line 146 in session_start_response session.persist() File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/session.py', line 600 in persist self._session().save() File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/session.py', line 293 in save self.namespace.release_write_lock() File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/container.py', line 149 in release_write_lock self.close(checkcount=True) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/container.py', line 172 in close self.do_close() File
[sqlalchemy] Re: detached, pickled sqlalchemy collection object failing after server restart
Thanks for the compile_mappers() call - that is a lot easier than calling compile() on all the individual mappers. What is very strange is that I can see at the point of my first calling on the object from the beaker session, I retrieve it and merge it successfully into the SQLAlchemy Session. I have put debug statements in and they get logged before the throwing of the exception which in my mind proves that the mappers are initialized successfully prior to the error. I'm really not sure what beaker is up to. Another hint that may help is that after the throwing of the exception after server restart, if I just reload the page (its a web app) everything resolves itself. It is only on initial initialization of which it recovers successfully on a subsequent call. On Oct 22, 7:35 am, Michael Bayer mike...@zzzcomputing.com wrote: rootsmith wrote: Well, it appears I have answered my own question and, yes, it was a SQLAlchemy issue. I resolved the problem by explicitly compiling the mapping at the end of the model initialization: manager_mapper = orm.mapper(Manager, manager_table, ... etc. ... (all other dependent mappers) manager_mapper.compile() My extremely vague understanding of what is going on is that if you are explicitly using the SQLAlchemy API to retrieve the pickled object then compiling is done implicitly but if the API is not used then the compiling is not done and you run into troubles. I am not sure how the beaker session is trying to interact with the object that causes the problem versus my own testing of interacting with the pickled object being successful but compiling is the answer. If someone has a more robust explanation I would be very interested to hear it. that seems like a tricky error to reproduce. When your application starts, assuming it sets up mappers immediately, as soon as they are used all mappers are compiled. pickling/depickling wouldn't have any compilation related issues. Additionally your error seems to occur during *serialization*, which is very strange. Its not occurring to me how your application would have a hold of a mapped instance without your mappers having been compiled...unless that object was pulled from a cache previously. In any case if you throw a compile_mappers() at the end of environment.py load_environment() you shouldn't have this kind of issue. but i would be curious how you're getting it to do that. On Oct 21, 10:08 pm, rootsmith ke...@rootsmith.ca wrote: Hi, I have a pylons application using file based beaker sessions (not to be confused with SQLAlchemy Sessions.) I retrieve a mapped object from the database, detach it from the SQLAlchemy Session like so: Session.expunge(object) and save it to the beaker session (basically pickling it to a file.) like so: session['user'] = user session.save() If I restart the server and retrieve the pickled object from the session, merge it back into the SQLAlchemy Session everything seems to be fine. user = Session.merge(session['user'], dont_load=True) But then somewhere in the beaker session middleware something goes awry before it completes the call to the controller. Now I know this sounds like a beaker session problem but it is failing when it is making a call to the SQLAlchemy object. Here is the stack trace: URL:http://127.0.0.1:5000/ File '/Users/kevin/mydevenv/lib/python2.6/site-packages/ WebError-0.10.1-py2.6.egg/weberror/evalexception.py', line 431 in respond app_iter = self.application(environ, detect_start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Paste-1.7.2- py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/middleware.py', line 73 in __call__ return self.app(environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Beaker-1.4.1- py2.6.egg/beaker/middleware.py', line 152 in __call__ return self.wrap_app(environ, session_start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Routes-1.10.3- py2.6.egg/routes/middleware.py', line 130 in __call__ response = self.app(environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7- py2.6.egg/pylons/wsgiapp.py', line 125 in __call__ response = self.dispatch(controller, environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7- py2.6.egg/pylons/wsgiapp.py', line 324 in dispatch return controller(environ, start_response) File '/Users/kevin/Documents/workspace/ryzoe/ryzoe/lib/base.py', line 54 in __call__ return WSGIController.__call__(self, environ, start_response) File '/Users/kevin/mydevenv/lib/python2.6/site-packages/Pylons-0.9.7- py2.6.egg/pylons/controllers/core.py', line 284 in __call__ return
