Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
(Sorry if this message appears twice in the list - I first sent it using an unregistered mail address.) Hello, thank you for your quick reply. is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. I tried this first as migrating to version 0.7 or 0.6 sounds like a significant (and currently unplanned) effort, so I hoped we could stay with 0.5 for the moment. So I identified our standard mapper calls to tables. It turned out they are invoked by a single function running in the startup phase, so I added a call to compile_mappers() at the end of this function. The function is invoked very early and even before the __init__() method of the WSGI application object - I hope this is early enough. Looking at the server logs one can see it is called right after server startup, as often as initial processes are configured. Unfortunately, there are still errors displayed. Most of the time I found 'Mapper' object has no attribute '_props'. I also tried to use the standard SQLAlchemy mapper() function instead of our wrapper mentioned in my first message (a variant of [1]). In most cases the system complained about the _props attribute as before, in rare cases the message dictionary changed size during iteration appeared. (I switched back to the standard mapper to find out if our wrapper could have an influence.) Am I correct that the results indicate we *have* to switch to 0.6/0.7 for a solution, or did I oversee something? From your message: If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. Could it be I am not loading enough modules? Does all modules mean all modules of the application, or all modules to map successfully? Thanks and regards Jochen Reference: [1] http://www.sqlalchemy.org/trac/wiki/UsageRecipes/SessionAwareMapper -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
On May 11, 2012, at 1:31 PM, Jochen Stenzel wrote: (Sorry if this message appears twice in the list - I first sent it using an unregistered mail address.) Hello, thank you for your quick reply. is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. I tried this first as migrating to version 0.7 or 0.6 sounds like a significant (and currently unplanned) effort, so I hoped we could stay with 0.5 for the moment. So I identified our standard mapper calls to tables. It turned out they are invoked by a single function running in the startup phase, so I added a call to compile_mappers() at the end of this function. The function is invoked very early and even before the __init__() method of the WSGI application object - I hope this is early enough. Looking at the server logs one can see it is called right after server startup, as often as initial processes are configured. Unfortunately, there are still errors displayed. Most of the time I found 'Mapper' object has no attribute '_props'. I also tried to use the standard SQLAlchemy mapper() function instead of our wrapper mentioned in my first message (a variant of [1]). In most cases the system complained about the _props attribute as before, in rare cases the message dictionary changed size during iteration appeared. (I switched back to the standard mapper to find out if our wrapper could have an influence.) Am I correct that the results indicate we *have* to switch to 0.6/0.7 for a solution, or did I oversee something? From your message: If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. Could it be I am not loading enough modules? Does all modules mean all modules of the application, or all modules to map successfully? yes the issue is very likely that more modules are being imported within non-main threads, and more mappers are coming in. if you get absolutely every mapper loaded up, then call compile_mappers() before any threads spawn, this kind of problem shouldn't occur. There may be other bugs in mapper configuration in 0.5 though this seems quite unusual that you're getting threading errors this frequently (especially dictionary changed size ? stack trace on that ?) . Trying 0.6 at least might be worth it. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
Could it be I am not loading enough modules? Does all modules mean all modules of the application, or all modules to map successfully? yes the issue is very likely that more modules are being imported within non-main threads, and more mappers are coming in. if you get absolutely every mapper loaded up, then call compile_mappers() before any threads spawn, this kind of problem shouldn't occur. Hm, should this include the dynamic mappers? These mappers are run with /request/ specific parameters - if a user requests to see data of a certain view, a select object (using the current view parameter value) and a related class are generated, used and released. These mappers /will/ come in with threads. So, if absolutely every mapper includes the mappers to selectables this cannot be achieved I fear ... Until now, I understood we should run all our standard mappers (to tables), finally run compile_mappers(), and could start the request handler cycle (with threaded handlers and new mappers produced on request) afterwards. There may be other bugs in mapper configuration in 0.5 though this seems quite unusual that you're getting threading errors this frequently (especially dictionary changed size ? stack trace on that ?) . I will try to produce a stack trace. (In the last instrumented debug versions, I caught all exceptions so the stack traces were not logged.) Trying 0.6 at least might be worth it. Just in case I could not describe clearly before how dynamic and request specific the function generated mappings are: would 0.6 allow us to use the pattern of dynamic mapping described above? Thanks and regards Jochen -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
On May 9, 2012, at 8:10 PM, Jochen Stenzel wrote: Hello, is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. \ -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Mapper compilation errors in multi threaded web application using dynamic mapping to selects
On Wed, May 9, 2012 at 4:11 PM, Michael Bayer mike...@zzzcomputing.com wrote: Hello, is there a problem in mapping classes to selects ([1]) /within a function/? with multiple threads, where the mappers initialization may first proceed as the product of a thread running, yes. you'd want to upgrade to 0.7 for the best versions of these fixes, or at least 0.6. If you must stay on 0.5, make sure all modules are fully imported, then run compile_mappers() before starting any threads. Sweet. Didn't know that one. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
