Re: [sqlalchemy] Deadlock in iterating over a session
On Sun, Mar 6, 2011 at 5:21 PM, Michael Bayer wrote: > > On Mar 6, 2011, at 8:16 PM, Lenza McElrath wrote: > > Ok cool. I actually could write my code to produce exactly that effect > fairly quickly since I was already accessing the mutable object through a > comparable_property on the model. It was surprising how easy it was > actually... SQLAlchemy really makes things awesome. > > We will start looking at moving to SQLAlchemy 0.7. It will probably be > hard to convince people to move to a "beta" version on production systems > though. When do you expect 0.7 to leave beta? > > I wonder if you could help with one other issue I have occationally > observed that might be related to this. Sometimes of these mutable objects > are None (myobject.value == None). These values are non-nullable columns > in the database, and should always be represented as an actual class in > logic. I have suspected this issue is related to objects being expired (by > explicitly called session.expire()) and then detached from the session. But > it seemed like SQLAlchemy should always either return a correct value or > raise an exception. Returning None could result in incorrect behavor for > code that does: if myobject.value: save_the_world(). Does it make sense > that this would be happening? I think I saw documentation explaining that > attribute values are loaded individually once they are expired? Could the > resurrecting of state not be happening properly when only loading a single > attribute? > > > expired attributes don't return None - they invoke a SELECT from the > database when they are accessed, or raise an error (to the chagrin of many > users) if the object is not associated with a session. > > I don't know offhand what would make your values come back as None, > wondering what you mean by "resurrecting", are you pickling the parent > objects as well ? Without looking at the source I suppose that would be > somewhat suspect though pickling/unpickling of mapped objects is well tested > and supported. > No pickling is happening. By "resurrect" I mean whatever is happening in MutableAttrInstanceState.__resurrect. I am definitely getting a None and not an error -- I happen to have an "assert myobject.value is not None". I will try to make a test case. It is hard because I don't know exactly how to trigger this code path. I ran into an issues here before with mutable columns not being updated after resurrecting (fixed in r26f423a667ca). If I recall the test case I made for that it involved modifying the attribute in one function and then flushing the session in another. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Deadlock in iterating over a session
On Mar 6, 2011, at 8:16 PM, Lenza McElrath wrote: > Ok cool. I actually could write my code to produce exactly that effect > fairly quickly since I was already accessing the mutable object through a > comparable_property on the model. It was surprising how easy it was > actually... SQLAlchemy really makes things awesome. > > We will start looking at moving to SQLAlchemy 0.7. It will probably be hard > to convince people to move to a "beta" version on production systems though. > When do you expect 0.7 to leave beta? > > I wonder if you could help with one other issue I have occationally observed > that might be related to this. Sometimes of these mutable objects are None > (myobject.value == None). These values are non-nullable columns in the > database, and should always be represented as an actual class in logic. I > have suspected this issue is related to objects being expired (by explicitly > called session.expire()) and then detached from the session. But it seemed > like SQLAlchemy should always either return a correct value or raise an > exception. Returning None could result in incorrect behavor for code that > does: if myobject.value: save_the_world(). Does it make sense that this > would be happening? I think I saw documentation explaining that attribute > values are loaded individually once they are expired? Could the resurrecting > of state not be happening properly when only loading a single attribute? expired attributes don't return None - they invoke a SELECT from the database when they are accessed, or raise an error (to the chagrin of many users) if the object is not associated with a session. I don't know offhand what would make your values come back as None, wondering what you mean by "resurrecting", are you pickling the parent objects as well ? Without looking at the source I suppose that would be somewhat suspect though pickling/unpickling of mapped objects is well tested and supported. > > On Fri, Mar 4, 2011 at 2:16 PM, Michael Bayer > wrote: > > On Mar 4, 2011, at 5:09 PM, Lenza McElrath wrote: > > > > > So there is no way to accomplish this in 0.6? I was looking at doing it > > the way I describe above, but it is not trivial to figure out which > > model/session a value is attached to. And I guess it is theoretically > > possible that a value could be connected to two models/sessions. > > Definitely scared of moving to an untested code... but looks like there are > > lots of improvements in 0.7 that might make it worth it... > > well, the on-change event required some less than trivial features so its an > 0.7 thing. The best solution of all is to not use mutable types in the first > place. I.e. if you need to change a scalar value in place, do something > like myobject.value = value.mutate(xyz). > > > > > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "sqlalchemy" group. > > To post to this group, send email to sqlalchemy@googlegroups.com. > > To unsubscribe from this group, send email to > > sqlalchemy+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/sqlalchemy?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To post to this group, send email to sqlalchemy@googlegroups.com. > To unsubscribe from this group, send email to > sqlalchemy+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To post to this group, send email to sqlalchemy@googlegroups.com. > To unsubscribe from this group, send email to > sqlalchemy+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Deadlock in iterating over a session
Ok cool. I actually could write my code to produce exactly that effect fairly quickly since I was already accessing the mutable object through a comparable_property on the model. It was surprising how easy it was actually... SQLAlchemy really makes things awesome. We will start looking at moving to SQLAlchemy 0.7. It will probably be hard to convince people to move to a "beta" version on production systems though. When do you expect 0.7 to leave beta? I wonder if you could help with one other issue I have occationally observed that might be related to this. Sometimes of these mutable objects are None (myobject.value == None). These values are non-nullable columns in the database, and should always be represented as an actual class in logic. I have suspected this issue is related to objects being expired (by explicitly called session.expire()) and then detached from the session. But it seemed like SQLAlchemy should always either return a correct value or raise an exception. Returning None could result in incorrect behavor for code that does: if myobject.value: save_the_world(). Does it make sense that this would be happening? I think I saw documentation explaining that attribute values are loaded individually once they are expired? Could the resurrecting of state not be happening properly when only loading a single attribute? On Fri, Mar 4, 2011 at 2:16 PM, Michael Bayer wrote: > > On Mar 4, 2011, at 5:09 PM, Lenza McElrath wrote: > > > > > So there is no way to accomplish this in 0.6? I was looking at doing it > the way I describe above, but it is not trivial to figure out which > model/session a value is attached to. And I guess it is theoretically > possible that a value could be connected to two models/sessions. Definitely > scared of moving to an untested code... but looks like there are lots of > improvements in 0.7 that might make it worth it... > > well, the on-change event required some less than trivial features so its > an 0.7 thing. The best solution of all is to not use mutable types in the > first place. I.e. if you need to change a scalar value in place, do > something like myobject.value = value.mutate(xyz). > > > > > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > > To post to this group, send email to sqlalchemy@googlegroups.com. > > To unsubscribe from this group, send email to > sqlalchemy+unsubscr...@googlegroups.com. > > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To post to this group, send email to sqlalchemy@googlegroups.com. > To unsubscribe from this group, send email to > sqlalchemy+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. > > -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Deadlock in iterating over a session
On Mar 4, 2011, at 5:09 PM, Lenza McElrath wrote: > > So there is no way to accomplish this in 0.6? I was looking at doing it the > way I describe above, but it is not trivial to figure out which model/session > a value is attached to. And I guess it is theoretically possible that a > value could be connected to two models/sessions. Definitely scared of moving > to an untested code... but looks like there are lots of improvements in 0.7 > that might make it worth it... well, the on-change event required some less than trivial features so its an 0.7 thing. The best solution of all is to not use mutable types in the first place. I.e. if you need to change a scalar value in place, do something like myobject.value = value.mutate(xyz). > > > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To post to this group, send email to sqlalchemy@googlegroups.com. > To unsubscribe from this group, send email to > sqlalchemy+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Deadlock in iterating over a session
Thanks for the quick response to this as usual. See me responses below. On Fri, Mar 4, 2011 at 7:32 AM, Michael Bayer wrote: > > On Mar 4, 2011, at 2:58 AM, Lenza McElrath wrote: > > Hello! I'm iterating over a session to look at all the objects: > > for obj in session: > do_something_cool(obj) > > Yesterday this caused what looks like a deadlock in SQLAlchemy code. Here > is the stack I grabbed using gdb: > >1. /python2.5/sqlalchemy/orm/session.py:1353 (Session.__iter__) >2. /python2.5/sqlalchemy/orm/identity.py:184 (WeakInstanceDict.values) >3. /python2.5/sqlalchemy/orm/identity.py:188 ( >WeakInstanceDict.itervalues) ** self._remove_mutex.acquire() >4. /python2.5/sqlalchemy/orm/state.py:501 ( >MutableAttrInstanceState.__resurrect) >5. /python2.5/sqlalchemy/orm/attributes.py:925 (Events.run) >6. /python2.5/sqlalchemy/orm/mapper.py:2424 (_event_on_resurrect) >7. /python2.5/sqlalchemy/util.py:953 (OrderedSet.__iter__) >8. /python2.5/sqlalchemy/orm/state.py:477 ( >MutableAttrInstanceState._cleanup) >9. /python2.5/sqlalchemy/orm/identity.py:139A (WeakInstanceDict.remove)** > self._remove_mutex.acquire >() > > I'm running SQLAlchemy 0.6.5. Is this a known issue or am I doing > something wrong? > > > I've never seen that before.A reentrant mutex would fix this but I > really hate to use those as they have a big performance hit and thats a very > critical section. Its true this is also related to "mutable" attributes > which is something I'd eventually like to remove entirely - they are pending > deprecation in 0.7. Is this issue consistently reproducible and can you > send me a test case ? This would be very high priority. > > Not sure if this helps but one of the triggers there is you have an object > that has "mutable" attributes on it, which is dirty, and has been garbage > collected. The "state" hangs around in the Session and when accessed > resurrects itself. At that point, it seems like some other object that was > also garbage collected starts doing the same thing before the process for > object #1 can complete, its not clear why that would happen here, thats the > point at which I'd want to pdb around to see what that's about. > This issue was in-consistently reproducible, but it is in a production system where a workaround has been applied, so now it is not-so-reproducible (hopefully). Every object has mutable attributes on it, so it is not surprising that there would be a dirty one in the session. Here is the do_something_cool function in case it provides some insight: def do_something_cool(obj, do_copy=False): for obj in session: if do_copy: ret_obj = copy.deepcopy(obj) else: ret_obj = obj ret_obj._has_been_flagged = True return ret_obj I believe do_copy should be False in this situation, but given the weirdness I would exclude the possibility that it is True. > > > On a somewhat related note, am experiencing the issue where lots of objects > in a session significantly reduces performance. It appears I am > experiencing the penalty for using MutableTypes on objects described here: > http://readthedocs.org/docs/sqlalchemy/en/latest/core/types.html#base-type-api. > The doc states: > > In order to detect changes, the ORM must create a copy of the value when it > is first accessed, so that changes to the current value can be compared > against the “clean” database-loaded value. Additionally, when the ORM checks > to see if any data requires flushing, it must scan through all instances in > the session which are known to have “mutable” attributes and compare the > current value of each one to its “clean” value. > > It doesn't seem like I should have to pay this penalty because with my type > I actually know when any updates occur. The type just supports > models.mutable_value.update(data). Is there a way to notify the > model/session that the value has been updated when someone calls update, so > the full scan of all instances is not needed? Do I need to rewrite the > update function to translate models.mutable_value.update(data) to actually > generate models.mutable_value = new_mutable_value or is there a better way > to do this? > > > very easy. Upgrade to 0.7. All has been resolved there, to support > in-place mutation detection you use the techniques described at > http://www.sqlalchemy.org/docs/07/orm/extensions/mutable.html . > > Of course you'd be beta testing something I'm not sure anyone is using yet. > Hopefully no deadlocks though ! :) (but again, I'd love to fix that > deadlock, I'll be looking at that today and a test case would be v. helpful) > So there is no way to accomplish this in 0.6? I was looking at doing it the way I describe above, but it is not trivial to figure out which model/session a value is attached to. And I guess it is theoretically possible that a value could be connected to two models/sessions. Definitely scared of moving to an untested code...
Re: [sqlalchemy] Deadlock in iterating over a session
On Mar 4, 2011, at 10:32 AM, Michael Bayer wrote: > > On Mar 4, 2011, at 2:58 AM, Lenza McElrath wrote: > >> Hello! I'm iterating over a session to look at all the objects: >> >> for obj in session: >> do_something_cool(obj) >> >> Yesterday this caused what looks like a deadlock in SQLAlchemy code. Here >> is the stack I grabbed using gdb: >> /python2.5/sqlalchemy/orm/session.py:1353 (Session.__iter__) >> /python2.5/sqlalchemy/orm/identity.py:184 (WeakInstanceDict.values) >> /python2.5/sqlalchemy/orm/identity.py:188 (WeakInstanceDict.itervalues) ** >> self._remove_mutex.acquire() >> /python2.5/sqlalchemy/orm/state.py:501 (MutableAttrInstanceState.__resurrect) >> /python2.5/sqlalchemy/orm/attributes.py:925 (Events.run) >> /python2.5/sqlalchemy/orm/mapper.py:2424 (_event_on_resurrect) >> /python2.5/sqlalchemy/util.py:953 (OrderedSet.__iter__) >> /python2.5/sqlalchemy/orm/state.py:477 (MutableAttrInstanceState._cleanup) >> /python2.5/sqlalchemy/orm/identity.py:139A (WeakInstanceDict.remove) ** >> self._remove_mutex.acquire() >> I'm running SQLAlchemy 0.6.5. Is this a known issue or am I doing something >> wrong? > > I've never seen that before.A reentrant mutex would fix this but I really > hate to use those as they have a big performance hit and thats a very > critical section. Its true this is also related to "mutable" attributes > which is something I'd eventually like to remove entirely - they are pending > deprecation in 0.7. Is this issue consistently reproducible and can you > send me a test case ? This would be very high priority. > > Not sure if this helps but one of the triggers there is you have an object > that has "mutable" attributes on it, which is dirty, and has been garbage > collected. The "state" hangs around in the Session and when accessed > resurrects itself. At that point, it seems like some other object that was > also garbage collected starts doing the same thing before the process for > object #1 can complete, its not clear why that would happen here, thats the > point at which I'd want to pdb around to see what that's about. OK I can't reproduce it, but thinking about the problem I think even using an RLock wouldn't help anyway since then you'd get a "dictionary changed size while iterating" error.So my current thoughts on this issue are at http://www.sqlalchemy.org/trac/ticket/2087 , which includes a patch against 0.7 which removes the mutexing and takes a different approach to the issue of items disappearing while the session is being iterated. If this issue is one you get often, feel free to apply it against 0.7 and try it out.The patch needs tests to prove that the mechanism actually works. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Deadlock in iterating over a session
On Mar 4, 2011, at 2:58 AM, Lenza McElrath wrote: > Hello! I'm iterating over a session to look at all the objects: > > for obj in session: > do_something_cool(obj) > > Yesterday this caused what looks like a deadlock in SQLAlchemy code. Here is > the stack I grabbed using gdb: > /python2.5/sqlalchemy/orm/session.py:1353 (Session.__iter__) > /python2.5/sqlalchemy/orm/identity.py:184 (WeakInstanceDict.values) > /python2.5/sqlalchemy/orm/identity.py:188 (WeakInstanceDict.itervalues) ** > self._remove_mutex.acquire() > /python2.5/sqlalchemy/orm/state.py:501 (MutableAttrInstanceState.__resurrect) > /python2.5/sqlalchemy/orm/attributes.py:925 (Events.run) > /python2.5/sqlalchemy/orm/mapper.py:2424 (_event_on_resurrect) > /python2.5/sqlalchemy/util.py:953 (OrderedSet.__iter__) > /python2.5/sqlalchemy/orm/state.py:477 (MutableAttrInstanceState._cleanup) > /python2.5/sqlalchemy/orm/identity.py:139A (WeakInstanceDict.remove) ** > self._remove_mutex.acquire() > I'm running SQLAlchemy 0.6.5. Is this a known issue or am I doing something > wrong? I've never seen that before.A reentrant mutex would fix this but I really hate to use those as they have a big performance hit and thats a very critical section. Its true this is also related to "mutable" attributes which is something I'd eventually like to remove entirely - they are pending deprecation in 0.7. Is this issue consistently reproducible and can you send me a test case ? This would be very high priority. Not sure if this helps but one of the triggers there is you have an object that has "mutable" attributes on it, which is dirty, and has been garbage collected. The "state" hangs around in the Session and when accessed resurrects itself. At that point, it seems like some other object that was also garbage collected starts doing the same thing before the process for object #1 can complete, its not clear why that would happen here, thats the point at which I'd want to pdb around to see what that's about. > > On a somewhat related note, am experiencing the issue where lots of objects > in a session significantly reduces performance. It appears I am experiencing > the penalty for using MutableTypes on objects described here: > http://readthedocs.org/docs/sqlalchemy/en/latest/core/types.html#base-type-api. > The doc states: > > In order to detect changes, the ORM must create a copy of the value when it > is first accessed, so that changes to the current value can be compared > against the “clean” database-loaded value. Additionally, when the ORM checks > to see if any data requires flushing, it must scan through all instances in > the session which are known to have “mutable” attributes and compare the > current value of each one to its “clean” value. > > It doesn't seem like I should have to pay this penalty because with my type I > actually know when any updates occur. The type just supports > models.mutable_value.update(data). Is there a way to notify the > model/session that the value has been updated when someone calls update, so > the full scan of all instances is not needed? Do I need to rewrite the > update function to translate models.mutable_value.update(data) to actually > generate models.mutable_value = new_mutable_value or is there a better way to > do this? very easy. Upgrade to 0.7. All has been resolved there, to support in-place mutation detection you use the techniques described at http://www.sqlalchemy.org/docs/07/orm/extensions/mutable.html . Of course you'd be beta testing something I'm not sure anyone is using yet. Hopefully no deadlocks though ! :) (but again, I'd love to fix that deadlock, I'll be looking at that today and a test case would be v. helpful) -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Deadlock in iterating over a session
Hello! I'm iterating over a session to look at all the objects: for obj in session: do_something_cool(obj) Yesterday this caused what looks like a deadlock in SQLAlchemy code. Here is the stack I grabbed using gdb: 1. /python2.5/sqlalchemy/orm/session.py:1353 (Session.__iter__) 2. /python2.5/sqlalchemy/orm/identity.py:184 (WeakInstanceDict.values) 3. /python2.5/sqlalchemy/orm/identity.py:188 (WeakInstanceDict.itervalues ) ** self._remove_mutex.acquire() 4. /python2.5/sqlalchemy/orm/state.py:501 ( MutableAttrInstanceState.__resurrect) 5. /python2.5/sqlalchemy/orm/attributes.py:925 (Events.run) 6. /python2.5/sqlalchemy/orm/mapper.py:2424 (_event_on_resurrect) 7. /python2.5/sqlalchemy/util.py:953 (OrderedSet.__iter__) 8. /python2.5/sqlalchemy/orm/state.py:477 ( MutableAttrInstanceState._cleanup) 9. /python2.5/sqlalchemy/orm/identity.py:139A (WeakInstanceDict.remove)** self._remove_mutex.acquire () I'm running SQLAlchemy 0.6.5. Is this a known issue or am I doing something wrong? On a somewhat related note, am experiencing the issue where lots of objects in a session significantly reduces performance. It appears I am experiencing the penalty for using MutableTypes on objects described here: http://readthedocs.org/docs/sqlalchemy/en/latest/core/types.html#base-type-api. The doc states: In order to detect changes, the ORM must create a copy of the value when it is first accessed, so that changes to the current value can be compared against the “clean” database-loaded value. Additionally, when the ORM checks to see if any data requires flushing, it must scan through all instances in the session which are known to have “mutable” attributes and compare the current value of each one to its “clean” value. It doesn't seem like I should have to pay this penalty because with my type I actually know when any updates occur. The type just supports models.mutable_value.update(data). Is there a way to notify the model/session that the value has been updated when someone calls update, so the full scan of all instances is not needed? Do I need to rewrite the update function to translate models.mutable_value.update(data) to actually generate models.mutable_value = new_mutable_value or is there a better way to do this? Thanks for any help! -Lenza -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
