Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
First of all thanks to all of you for your answers and time. Michael let me say that I agree 100% with all you wrote and my will/wish is to work as you wrote, but when you are inside the ORM its easy, other is if you want to interact with the ORM from outside. In my actual system I have more than 500 tables automatically generated from a UML model. Not all of them have a central role but all of them do something in the data model. If I want to fill one of the highest level object its attributes are classes their self with other attributes which are classes again; because I cannot simply send a few primitive parameters to the a function-ORM-aware and make the function create that object I am forced to create locally (in the GUI application) such object and their children (some of them can be NOT NULL so I cannot avoid it) then send the full object to the ORM interface. Apart the great advantages that an ORM offers it should be possible to install it as a service, that is separate it from the any other application: in other words I can have a library of objects (like I have) shared among several GUI/WEB applications and an ORM. The communication between the applications and the ORM should be done serializing and deserializing instances of objects in the shared library. As example you can take the remote interface for an EJB in Java (btw I am thinking about the stateless interface). In this way the ORM does have not to expose to other applicaitons it's session (or EntityManager in EJB) but just an interface. Still in other words the rules that I impose on how I manage my database (that is one or more method using session/transaction), should be independent from the way I display/control them. Again: I would like to use SA as the MODEL of a Model-View-Controller strategy and because of this I can have a myriad of views/controlles but just one separate model application acting as service. Now I am aware that my application is not decoupled as I want (Django initialize the SQL engine) so I cannot avoid SA to inject instruments in new instances but nonetheless I would expect that I may be able to work with objects in two separate universes: the GUI and the ORM. When I wrote that I feel to have missed something I was referring to this: it seems to me that I cannot use SA in a remote-like way, creating an object in the GUI and then send it to the ORM, being sure that because the ORM maps such object it knows how to persist it or eventually retrieve it because it know where to catch the key for each mapped object. For all the rest, thanks for your great software. Maurizio P.S. If you are curious about the system I am working on just take a brief look at it here http://cedadocs.badc.rl.ac.uk/905/ or herehttp://jenkins.badc.rl.ac.uk/cedaManager/cov/1 On Thursday, May 31, 2012 4:50:38 AM UTC+1, Michael Bayer wrote: On May 30, 2012, at 8:53 PM, Claudio Freire wrote: Thing is, in order to work with a large volume of objects, you're forced to do this, otherwise the session can grow uncontrollably. flush periodically, and don't maintain references to things you're done with. The Session does not strongly reference objects that have no pending changes, and they'll be garbage collected. When you separate the operation to work in batches, you almost always have some objects that have a lifespan larger than a single batch, and then a single session. Working in batches is fine. You only need a single Session for all those batches, and a single transaction.If you want several transactions, also fine, call commit() periodically. In none of these cases does the Session need to be closed, and all objects worked with thus far which are still referenced in memory can remain attached to that Session, and you wont have any detachment errors. The problems you're having are from unnecessary detachment of objects, from calling Session.close() and continuing to work with objects that have lost their owning Session, within the context of a new Session they have no association with. Another case in which an object's lifespan can exceed the session's, is when you want to implement caching with objects of your data model - cached values will have come from other sessions than the current one, and things get horribly messy. There are documented patterns for caching - see the example in examples/beaker_caching in the distro. This pattern is designed to cleanly handle the pattern of detached objects becoming re-associated with a particular session at once. The pattern is along the lines of, session is created to work with a field of objects, a set of objects is retrieved from the cache, then re-associated with the cache en-masse using the merge_result() method illustrated in the example. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To view this
Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
Seems like you have a monumental problem to overcome. I'm glad you mentioned EJB and have a Java background. In EJB, at least back when I used the very early version 1.0, the concept of the transactional nature of various service methods is defined separate from the implementation of the method itself. And again, the example of using Hibernate with Spring (or even without Spring) has a similar concept going on - the demarcation of a transaction is most commonly external to the methods that do the work. There's a great section on this in Hibernate's docs at https://community.jboss.org/wiki/SessionsAndTransactions?_sscc=t . You can see there's an emphasis on doing *many* things in a transaction, keeping the transaction/session as a resource that is present externally to a large series of operations. So SQLAlchemy hasn't made any of this up, it is emulating the same patterns that come from the J2EE world.The Session is modeled from that of Hibernate. In the Python community, designing applications via UML and then generating classes/tables from that completed design is mostly unheard of.The 500-table, automatically generated model, which as you describe uses tables on tables to represent attributes further nested on themselves using more tables, is something I've been exposed to years ago, though I never actually saw such a model achieve any success as up-front, UML design has little to do with relational database best practices. An RDBMS does best with a hand-designed schema, following standard normalization techniques but at the same time only using as many tables as are necessary to model the problem, mostly agnostic of how an object model may want to represent it and certainly removed from attempts to genericize the modeling of data in an OO sense. SQLAlchemy is designed for this latter model; while there are ways to make it work with models that have hundreds or thousands of tables, these kinds of setups are challenging, and less than ideal in any case as the database spends far too much effort querying and updating across too many tables for operations to be efficient. On May 31, 2012, at 4:10 AM, Maurizio Nagni wrote: First of all thanks to all of you for your answers and time. Michael let me say that I agree 100% with all you wrote and my will/wish is to work as you wrote, but when you are inside the ORM its easy, other is if you want to interact with the ORM from outside. In my actual system I have more than 500 tables automatically generated from a UML model. Not all of them have a central role but all of them do something in the data model. If I want to fill one of the highest level object its attributes are classes their self with other attributes which are classes again; because I cannot simply send a few primitive parameters to the a function-ORM-aware and make the function create that object I am forced to create locally (in the GUI application) such object and their children (some of them can be NOT NULL so I cannot avoid it) then send the full object to the ORM interface. Apart the great advantages that an ORM offers it should be possible to install it as a service, that is separate it from the any other application: in other words I can have a library of objects (like I have) shared among several GUI/WEB applications and an ORM. The communication between the applications and the ORM should be done serializing and deserializing instances of objects in the shared library. As example you can take the remote interface for an EJB in Java (btw I am thinking about the stateless interface). In this way the ORM does have not to expose to other applicaitons it's session (or EntityManager in EJB) but just an interface. Still in other words the rules that I impose on how I manage my database (that is one or more method using session/transaction), should be independent from the way I display/control them. Again: I would like to use SA as the MODEL of a Model-View-Controller strategy and because of this I can have a myriad of views/controlles but just one separate model application acting as service. Now I am aware that my application is not decoupled as I want (Django initialize the SQL engine) so I cannot avoid SA to inject instruments in new instances but nonetheless I would expect that I may be able to work with objects in two separate universes: the GUI and the ORM. When I wrote that I feel to have missed something I was referring to this: it seems to me that I cannot use SA in a remote-like way, creating an object in the GUI and then send it to the ORM, being sure that because the ORM maps such object it knows how to persist it or eventually retrieve it because it know where to catch the key for each mapped object. For all the rest, thanks for your great software. Maurizio P.S. If you are curious about the system I am working on just take a brief look at it
Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
On Thu, May 31, 2012 at 12:50 AM, Michael Bayer mike...@zzzcomputing.com wrote: Thing is, in order to work with a large volume of objects, you're forced to do this, otherwise the session can grow uncontrollably. flush periodically, and don't maintain references to things you're done with. The Session does not strongly reference objects that have no pending changes, and they'll be garbage collected. Problem is, I'm stuck with strongly-referencing sessions. The app comes from SA 0.3, and is heavily relying on the session as a kind of L1 cache - removing that assumption is a really huge task we haven't gotten to. We managed to upgrade it to SA 0.5, but we kept strongly-referencing sessions. The problems you're having are from unnecessary detachment of objects, from calling Session.close() and continuing to work with objects that have lost their owning Session, within the context of a new Session they have no association with. I've been solving those problems by reattaching objects to the session. Only with caches I haven't been able to do that, since cached objects will be used by many threads at once, so no single session can own them. Another case in which an object's lifespan can exceed the session's, is when you want to implement caching with objects of your data model - cached values will have come from other sessions than the current one, and things get horribly messy. There are documented patterns for caching - see the example in examples/beaker_caching in the distro. This pattern is designed to cleanly handle the pattern of detached objects becoming re-associated with a particular session at once. The pattern is along the lines of, session is created to work with a field of objects, a set of objects is retrieved from the cache, then re-associated with the cache en-masse using the merge_result() method illustrated in the example. Interesting, I hadn't seen that example. Bookmarked already :-) Thanks. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
On May 31, 2012, at 10:35 AM, Claudio Freire wrote: On Thu, May 31, 2012 at 12:50 AM, Michael Bayer mike...@zzzcomputing.com wrote: Thing is, in order to work with a large volume of objects, you're forced to do this, otherwise the session can grow uncontrollably. flush periodically, and don't maintain references to things you're done with. The Session does not strongly reference objects that have no pending changes, and they'll be garbage collected. Problem is, I'm stuck with strongly-referencing sessions. The app comes from SA 0.3, and is heavily relying on the session as a kind of L1 cache - removing that assumption is a really huge task we haven't gotten to. We managed to upgrade it to SA 0.5, but we kept strongly-referencing sessions. well you'd really need to change that, sorry.I get very close to removing the strongidentity map on each release, you're the first user I've ever encountered with a dependency on it... -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
Hello all, my curious situation is the following. A very simplified version of the code is: for data in res: obj = MyObject() ---here I fill the obj, aventually doing some query (create session, get, close) to SA sess = createSession() sess.add(obj) sess.commit() sess.close() -- do some other query (create session, get, close) and eventually persist the changes (create session, merge/add, commit close) now... on the first loop it works fine, then I receive the Parent instance warning me that a an obj inner attribute, say a contact, is not bound so cannot load contact.phone attribute. BTW I receive such message when I commit() but I am able to make it appear looking in the second loop, during a debugging session, at the specific obj attribute. What I would like to implement is to restrict the session (in a more general sense SA activity even if I know that actually SA inject some instrumentation in obj at the creation time) inside a unique class, say with some static methods (the code above involving the session is in a separate class) in order to centralize the operation toward the DB, and doing this it should act in a stateless way (createSession, do stuff, commit/rollback, close), unfortunately seems that I am missing something. Thanks Maurizio -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To view this discussion on the web visit https://groups.google.com/d/msg/sqlalchemy/-/J4THN4X57aUJ. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
well yes, the way you're doing this is entirely the opposite of how the ORM is designed to function.The Session has been developed in order to work in an intelligent manner with full graphs of interrelated objects, all coordinated under the umbrella of a transaction which applies atomicity and isolation to the full series of operations.An individual object loaded from a Session is in fact an extension of that Session's state, which is in turn an extension of the state of the current transaction within the database, all kept in sync mostly automatically. When you continuously break this connection between a whole series of interconnected objects you'll run into not just lots of problems keeping objects associated with continuously new transactions, but also horrendous performance from due to the excessive number of commits and re-acquisition of new connections/transactions. In general, the Session is typically used in a bounding sense, its lifespan beginning before you work with any objects, and ending only after you've completed the work with those objects.I frequently use the metaphor here that the Session is the table setting and the objects are the meal. Docs on why commit forces a reload by default and all the rest of this is explained pretty well at http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html . On May 30, 2012, at 4:21 AM, Maurizio Nagni wrote: Hello all, my curious situation is the following. A very simplified version of the code is: for data in res: obj = MyObject() ---here I fill the obj, aventually doing some query (create session, get, close) to SA sess = createSession() sess.add(obj) sess.commit() sess.close() -- do some other query (create session, get, close) and eventually persist the changes (create session, merge/add, commit close) now... on the first loop it works fine, then I receive the Parent instance warning me that a an obj inner attribute, say a contact, is not bound so cannot load contact.phone attribute. BTW I receive such message when I commit() but I am able to make it appear looking in the second loop, during a debugging session, at the specific obj attribute. What I would like to implement is to restrict the session (in a more general sense SA activity even if I know that actually SA inject some instrumentation in obj at the creation time) inside a unique class, say with some static methods (the code above involving the session is in a separate class) in order to centralize the operation toward the DB, and doing this it should act in a stateless way (createSession, do stuff, commit/rollback, close), unfortunately seems that I am missing something. Thanks Maurizio -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To view this discussion on the web visit https://groups.google.com/d/msg/sqlalchemy/-/J4THN4X57aUJ. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
On Wed, May 30, 2012 at 8:39 PM, Michael Bayer mike...@zzzcomputing.com wrote: well yes, the way you're doing this is entirely the opposite of how the ORM is designed to function. The Session has been developed in order to work in an intelligent manner with full graphs of interrelated objects, all coordinated under the umbrella of a transaction which applies atomicity and isolation to the full series of operations. An individual object loaded from a Session is in fact an extension of that Session's state, which is in turn an extension of the state of the current transaction within the database, all kept in sync mostly automatically. When you continuously break this connection between a whole series of interconnected objects you'll run into not just lots of problems keeping objects associated with continuously new transactions, but also horrendous performance from due to the excessive number of commits and re-acquisition of new connections/transactions. In general, the Session is typically used in a bounding sense, its lifespan beginning before you work with any objects, and ending only after you've completed the work with those objects. I frequently use the metaphor here that the Session is the table setting and the objects are the meal. Thing is, in order to work with a large volume of objects, you're forced to do this, otherwise the session can grow uncontrollably. When you separate the operation to work in batches, you almost always have some objects that have a lifespan larger than a single batch, and then a single session. Another case in which an object's lifespan can exceed the session's, is when you want to implement caching with objects of your data model - cached values will have come from other sessions than the current one, and things get horribly messy. And, finally, the long transaction pattern, which luckily is a lot easier to solve. If the OP is using that pattern, it's just a matter of reattaching detached objects to the session. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Another Parent instance is not bound to a Session; lazy load...
On May 30, 2012, at 8:53 PM, Claudio Freire wrote: Thing is, in order to work with a large volume of objects, you're forced to do this, otherwise the session can grow uncontrollably. flush periodically, and don't maintain references to things you're done with. The Session does not strongly reference objects that have no pending changes, and they'll be garbage collected. When you separate the operation to work in batches, you almost always have some objects that have a lifespan larger than a single batch, and then a single session. Working in batches is fine. You only need a single Session for all those batches, and a single transaction.If you want several transactions, also fine, call commit() periodically. In none of these cases does the Session need to be closed, and all objects worked with thus far which are still referenced in memory can remain attached to that Session, and you wont have any detachment errors. The problems you're having are from unnecessary detachment of objects, from calling Session.close() and continuing to work with objects that have lost their owning Session, within the context of a new Session they have no association with. Another case in which an object's lifespan can exceed the session's, is when you want to implement caching with objects of your data model - cached values will have come from other sessions than the current one, and things get horribly messy. There are documented patterns for caching - see the example in examples/beaker_caching in the distro. This pattern is designed to cleanly handle the pattern of detached objects becoming re-associated with a particular session at once. The pattern is along the lines of, session is created to work with a field of objects, a set of objects is retrieved from the cache, then re-associated with the cache en-masse using the merge_result() method illustrated in the example. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
