Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
(Sorry if this message appears twice in the list - I first sent it using an unregistered mail address.) Hello, thank you for your quick reply. is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. I tried this first as migrating to version 0.7 or 0.6 sounds like a significant (and currently unplanned) effort, so I hoped we could stay with 0.5 for the moment. So I identified our standard mapper calls to tables. It turned out they are invoked by a single function running in the startup phase, so I added a call to compile_mappers() at the end of this function. The function is invoked very early and even before the __init__() method of the WSGI application object - I hope this is early enough. Looking at the server logs one can see it is called right after server startup, as often as initial processes are configured. Unfortunately, there are still errors displayed. Most of the time I found 'Mapper' object has no attribute '_props'. I also tried to use the standard SQLAlchemy mapper() function instead of our wrapper mentioned in my first message (a variant of [1]). In most cases the system complained about the _props attribute as before, in rare cases the message dictionary changed size during iteration appeared. (I switched back to the standard mapper to find out if our wrapper could have an influence.) Am I correct that the results indicate we *have* to switch to 0.6/0.7 for a solution, or did I oversee something? From your message: If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. Could it be I am not loading enough modules? Does all modules mean all modules of the application, or all modules to map successfully? Thanks and regards Jochen Reference: [1] http://www.sqlalchemy.org/trac/wiki/UsageRecipes/SessionAwareMapper -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
On May 11, 2012, at 1:31 PM, Jochen Stenzel wrote: (Sorry if this message appears twice in the list - I first sent it using an unregistered mail address.) Hello, thank you for your quick reply. is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. I tried this first as migrating to version 0.7 or 0.6 sounds like a significant (and currently unplanned) effort, so I hoped we could stay with 0.5 for the moment. So I identified our standard mapper calls to tables. It turned out they are invoked by a single function running in the startup phase, so I added a call to compile_mappers() at the end of this function. The function is invoked very early and even before the __init__() method of the WSGI application object - I hope this is early enough. Looking at the server logs one can see it is called right after server startup, as often as initial processes are configured. Unfortunately, there are still errors displayed. Most of the time I found 'Mapper' object has no attribute '_props'. I also tried to use the standard SQLAlchemy mapper() function instead of our wrapper mentioned in my first message (a variant of [1]). In most cases the system complained about the _props attribute as before, in rare cases the message dictionary changed size during iteration appeared. (I switched back to the standard mapper to find out if our wrapper could have an influence.) Am I correct that the results indicate we *have* to switch to 0.6/0.7 for a solution, or did I oversee something? From your message: If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. Could it be I am not loading enough modules? Does all modules mean all modules of the application, or all modules to map successfully? yes the issue is very likely that more modules are being imported within non-main threads, and more mappers are coming in. if you get absolutely every mapper loaded up, then call compile_mappers() before any threads spawn, this kind of problem shouldn't occur. There may be other bugs in mapper configuration in 0.5 though this seems quite unusual that you're getting threading errors this frequently (especially dictionary changed size ? stack trace on that ?) . Trying 0.6 at least might be worth it. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
Could it be I am not loading enough modules? Does all modules mean all modules of the application, or all modules to map successfully? yes the issue is very likely that more modules are being imported within non-main threads, and more mappers are coming in. if you get absolutely every mapper loaded up, then call compile_mappers() before any threads spawn, this kind of problem shouldn't occur. Hm, should this include the dynamic mappers? These mappers are run with /request/ specific parameters - if a user requests to see data of a certain view, a select object (using the current view parameter value) and a related class are generated, used and released. These mappers /will/ come in with threads. So, if absolutely every mapper includes the mappers to selectables this cannot be achieved I fear ... Until now, I understood we should run all our standard mappers (to tables), finally run compile_mappers(), and could start the request handler cycle (with threaded handlers and new mappers produced on request) afterwards. There may be other bugs in mapper configuration in 0.5 though this seems quite unusual that you're getting threading errors this frequently (especially dictionary changed size ? stack trace on that ?) . I will try to produce a stack trace. (In the last instrumented debug versions, I caught all exceptions so the stack traces were not logged.) Trying 0.6 at least might be worth it. Just in case I could not describe clearly before how dynamic and request specific the function generated mappings are: would 0.6 allow us to use the pattern of dynamic mapping described above? Thanks and regards Jochen -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
Hello,
is there a problem in mapping classes to selects ([1]) /within a
function/? We are running into mapper errors reading
InvalidRequestError: One or more mappers failed to compile.
Exception was probably suppressed within a hasattr() call. Message was:
One or more mappers failed to compile. Exception was probably
suppressed within a hasattr() call. Message was: 'Mapper' object has no
attribute '_props'
when doing so and I could not find out why yet.
Here is our setup: we run a WSGI web application using Apache 2.2.14 in
MPM worker mode (a multi threading server configuration, see [2]),
Python 2.6.6, Werkzeug 0.5.1, SQLAlchemy 0.5.8 and SQLite 3 databases.
Standard tables are mapped as usual, ORM and expression language access
work well.
To provide views with variable parameters, additional dynamic
mappings are established on request, which means by a function. Here is
a simplified example:
def get_view(param):
class DBG(object): pass
dbg = select([records.c.id],
records.c.checksum==param).alias('dbg')
mapper(DBG, dbg)
return (DBG, dbg)
The function provides the select object for use in expressions and the
class for ORM use, but the error continues to show up even if the
results are never used, as in
def some_function():
class DBG(object): pass
dbg = select([records.c.id]).alias('dbg')
mapper(DBG, dbg)
return something_else
The mapper is a wrapped version of SQLAlchemies standard wrapper. It
extends the given class, calls sqlalchemy.orm.mapper() and supplies the
result of that function call, a Mapper object. I suppose it can be
treated to be sqlalchemy.orm.mapper here for simplicity.
Now this works well in a (single threaded) local environment, but on
the web server the application can become unstable when the function is
invoked answering a request. Then, in /some/ subsequent request, the
error shown above can arise, which looks as if something is passed
through from one request to the other. It can take many requests until
this happens but once the error occurred more and more threads are
infected quickly (in a stress test scenario).
The differences I see between our standard mappings and those in this
function are that standard mappers are applied once, while the function
maps multiple times. The second difference is the use of selects instead
of tables. Do you see any problems here?
Is there a known problem with this setup, or where could I continue to
search to find the reason of this problem? Is there a better practice to
be followed?
(I searched for problems in session handling but it seems to me the
application follows the related instructions, using a scoped session and
calling session.remove() at the end of request handling.)
Thanks in advance!
Jochen
References:
[1]
http://docs.sqlalchemy.org/en/rel_0_7/orm/mapper_config.html#mapping-a-class-against-arbitrary-selects
[2] http://httpd.apache.org/docs/2.0/mod/worker.html
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
On May 9, 2012, at 8:10 PM, Jochen Stenzel wrote: Hello, is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. \ -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
On Wed, May 9, 2012 at 4:11 PM, Michael Bayer mike...@zzzcomputing.com wrote: Hello, is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. Sweet. Didn't know that one. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
