Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Mon, Mar 05, 2012 at 04:52:36PM -0500, Michael Bayer wrote: Wow, OK great, you were able to make something work while maintaining the ClassManager approach. So the first thing is, you can skip the redefinition of Mapper(). You can put your custom ClassManager class right on a base or mixin class like this: class MySpecialMixin(object): __sa_instrumentation_manager__ = MySpecialClassManager I would be happy to do so, but is there any way I can do same w/o polluting class's __dict__? What about adding argument to Mapper.__init__? The next thing is, you could also instead subclass InstrumentationManager instead of ClassManager, which provides hooks that are a bit more public. I'd probably need to add some more hooks to it in order to fulfill this use case, though. If you look through instrumentation.py _ClassInstrumentationAdapter you can see what that adaption looks like. Yeah, I should have use it instead. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Mar 10, 2012, at 11:45 AM, Andrey Popp wrote: On Mon, Mar 05, 2012 at 04:52:36PM -0500, Michael Bayer wrote: Wow, OK great, you were able to make something work while maintaining the ClassManager approach. So the first thing is, you can skip the redefinition of Mapper(). You can put your custom ClassManager class right on a base or mixin class like this: class MySpecialMixin(object): __sa_instrumentation_manager__ = MySpecialClassManager I would be happy to do so, but is there any way I can do same w/o polluting class's __dict__? What about adding argument to Mapper.__init__? Hm yeah I suppose we can make that an option, sure! The next thing is, you could also instead subclass InstrumentationManager instead of ClassManager, which provides hooks that are a bit more public. I'd probably need to add some more hooks to it in order to fulfill this use case, though. If you look through instrumentation.py _ClassInstrumentationAdapter you can see what that adaption looks like. Yeah, I should have use it instead. this approach might have more overhead, though. I like your approach too. The InstrumentationManager stuff was kind of a one off to support some PJE stuff. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Sat, Mar 10, 2012 at 02:23:47PM -0800, Michael Bayer wrote: On Mar 10, 2012, at 11:45 AM, Andrey Popp wrote: On Mon, Mar 05, 2012 at 04:52:36PM -0500, Michael Bayer wrote: Wow, OK great, you were able to make something work while maintaining the ClassManager approach. So the first thing is, you can skip the redefinition of Mapper(). You can put your custom ClassManager class right on a base or mixin class like this: class MySpecialMixin(object): __sa_instrumentation_manager__ = MySpecialClassManager I would be happy to do so, but is there any way I can do same w/o polluting class's __dict__? What about adding argument to Mapper.__init__? Hm yeah I suppose we can make that an option, sure! The next thing is, you could also instead subclass InstrumentationManager instead of ClassManager, which provides hooks that are a bit more public. I'd probably need to add some more hooks to it in order to fulfill this use case, though. If you look through instrumentation.py _ClassInstrumentationAdapter you can see what that adaption looks like. Yeah, I should have use it instead. this approach might have more overhead, though. I like your approach too. The InstrumentationManager stuff was kind of a one off to support some PJE stuff. By the way, do you have any plans to sprint on SQLAlhemy at PyCon? I'm planning to join PyPy sprint, but will hack a day on SQLAlhemy with joy. What do you think? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Mar 10, 2012, at 3:03 PM, Andrey Popp wrote: By the way, do you have any plans to sprint on SQLAlhemy at PyCon? I'm planning to join PyPy sprint, but will hack a day on SQLAlhemy with joy. What do you think? yeah ill be here through wednesday hanging in the Pyramid-ish area -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Sat, Mar 10, 2012 at 03:24:57PM -0800, Michael Bayer wrote: On Mar 10, 2012, at 3:03 PM, Andrey Popp wrote: By the way, do you have any plans to sprint on SQLAlhemy at PyCon? I'm planning to join PyPy sprint, but will hack a day on SQLAlhemy with joy. What do you think? yeah ill be here through wednesday hanging in the Pyramid-ish area Ok, cool, let's meet there then. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
Hello, just for the record, I've built a quick and dirty hack[1] which simplifies state-management for immutable domain models (can create but can't modify). Achieved 2-2.5x speedup in pickle/unpickle (see tests module). I'm not just suggesting anyone to use this in production (though I already do :-)) so this is just FYI. [1]: https://github.com/andreypopp/saimmutable -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Sun, Feb 19, 2012 at 6:27 AM, Andrey Popp 8may...@gmail.com wrote: I've managed quite efficient inlining with bytecode magic. It's especially effective with SQLA code, since it also specializes the inlined function, removing a lot of dead code (in the call context). That's pretty interesting! I could share the code if you're interested, but it is rather hard to maintain (it's tied to the bytecode, which is version-specific in CPython) and it's experimental code, so it's rather ugly. Yeah, that would be great, thanks! Finally, I got access to it. So, this[0] is the core code. The easiest way to invoke it is to install it as an import hook (see the function install), that will process all imports from then on. The code is only tested in 2.6, and has issues with 2.7, not to mention 3.x. Might work with 2.5/2.4 though. [0] http://pastebin.com/9TN6zJKc -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Mon, Mar 05, 2012 at 06:20:33PM -0300, Claudio Freire wrote: On Sun, Feb 19, 2012 at 6:27 AM, Andrey Popp 8may...@gmail.com wrote: I've managed quite efficient inlining with bytecode magic. It's especially effective with SQLA code, since it also specializes the inlined function, removing a lot of dead code (in the call context). That's pretty interesting! I could share the code if you're interested, but it is rather hard to maintain (it's tied to the bytecode, which is version-specific in CPython) and it's experimental code, so it's rather ugly. Yeah, that would be great, thanks! Finally, I got access to it. Thanks a lot! -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
Wow, OK great, you were able to make something work while maintaining the ClassManager approach. So the first thing is, you can skip the redefinition of Mapper(). You can put your custom ClassManager class right on a base or mixin class like this: class MySpecialMixin(object): __sa_instrumentation_manager__ = MySpecialClassManager The next thing is, you could also instead subclass InstrumentationManager instead of ClassManager, which provides hooks that are a bit more public. I'd probably need to add some more hooks to it in order to fulfill this use case, though. If you look through instrumentation.py _ClassInstrumentationAdapter you can see what that adaption looks like. You can see examples of custom instrumentation in examples/custom_attributes/custom_management.py. This would be *really* neat if it actually works all the way using public API. This would probably go right into the examples/custom_attributes/ directory as a recipe to use for lower-latency read-only objects. On Mar 5, 2012, at 4:14 PM, Andrey Popp wrote: Hello, just for the record, I've built a quick and dirty hack[1] which simplifies state-management for immutable domain models (can create but can't modify). Achieved 2-2.5x speedup in pickle/unpickle (see tests module). I'm not just suggesting anyone to use this in production (though I already do :-)) so this is just FYI. [1]: https://github.com/andreypopp/saimmutable -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Feb 19, 2012, at 5:24 AM, Andrey Popp wrote: Regarding rationale let me describe where I can find this feature useful: * You have a part of you tables read-only, so you only query data from them. SQLAlchemy doesn't have to track changes on objects of classes mapped to these tables. According to this stackoverflow post[1] we can get a not so negligible performance gain here. * We can map same table on same class using different mappers (one in read-only mode and other in persistence mode). That way we can use read-only mapper to query data in case we don't want to change it and we can use persistence mapper otherwise. I believe this will also lead to better correctness of application itself. Hello, After reading your stackoverflow post (implying that a profile revealed much instrumentation that you don't need), it occurred to me that you could use SQLAlchemy to generate the SQL query which you then pass directly to an execute method on a the underlying session bind to bypass generating SQLAlchemy models. Ideally, you could further shrink the query by only requesting the columns you actually need in your calculations. This seems at least the least-instrusive approach before you jump to C. Cheers, M -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Sat, Feb 18, 2012 at 07:57:14PM -0500, Michael Bayer wrote: I don't know that customizing instrumentationmanager is all you'd need. There's lots of bookkeeping occurring with instancestate that takes time and is related to state tracking and persistence. The whole need for __setstate__ is due to the instancestate object. Not including it suggests an entirely new system that at most would consume rows from the orm.Query and route them into this alternate system. Well, I didn't want to implement entirely new system, but just to separate part of ORM which manages persistence from part of ORM which just map database columns to instance attrs. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Reducing instrumentation overhead for read-only entities
On Feb 18, 2012, at 8:05 PM, Andrey Popp wrote: On Sat, Feb 18, 2012 at 07:57:14PM -0500, Michael Bayer wrote: I don't know that customizing instrumentationmanager is all you'd need. There's lots of bookkeeping occurring with instancestate that takes time and is related to state tracking and persistence. The whole need for __setstate__ is due to the instancestate object. Not including it suggests an entirely new system that at most would consume rows from the orm.Query and route them into this alternate system. Well, I didn't want to implement entirely new system, but just to separate part of ORM which manages persistence from part of ORM which just map database columns to instance attrs. As it is now, you can get named tuples out of Query that should be serializable and don't use InstanceState. Rows from an execute() call also behave like named tuples and are serializable. So I assume here you're looking for objects that also have collections and related instances, including lazy and eager loading of those, minus the usage of an InstanceState which you're saying is too expensive to deserialize, or a simplified InstanceState that is somehow not quite as expensive to deserialize, though really you'd be talking about shaving off maybe 10-20% of function calls if it were only a simplification of InstanceState. It would be a major new feature add requiring architectural changes, tests, and most importantly a clear documentation story that makes the rationale for this alternate mode very clear and makes it totally unambiguous when this mode might be used - else the entire project is diluted by echos of too many ways to do it, too confusing, too complicated, etc. It's complicating the internals and API for an use case that may very well be completely obsolete in a year or two due to Pypy and other performance techniques. The actual performance savings may be marginal in any case. Optimizing InstanceState while maintaining it's current behavioral contract exactly would be a lot more straightforward, requiring no complications to internals, API or documentation story. Methods here include experimenting with various forms of inlining inside of __setstate__() (such as inlining the call to setup_instance() when the default instrumentation manager is in use) as well as experimenting with a C port. There's not much that happens inside of __setstate__() as it is. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
