Here is the basic outline of the web request hander. It creates an
operation, gets the return value, and either spawns a thread to execute the
operation (with non-sqlalchemy related instrumentation) or completes the
execution on the operation model. In the case I see there is a provisioning
phase and the operation return value is None.
There are two potential threading problems I see:
1) Since op._operation is an ORM, could evaluating it in a boolean context
cause the ORM to be reattached to a previously closed session?
2) if the operation returns orms they will may accessed concurrently by the
handler thread and the execute thread. This return value is sent as the
xmlrpc response and contains only python primitives. The operations we see
this on return None.
try:
op = Operation(..) # the first session is created, used, and closed
here...the close happens in a try/finally and the "transaction XXX
committed" logging happens after close() completes successfully
ret = op.get_return_value() # the operation invariably returns None in
the vast majority of cases we've seen
if has_attr(op, 'provision'): # the operations we see this with do have
a provision() method
AsyncOperation(LOGGER,
MonitoringAgent.getInstance().monitor("%s-execute" %
op.api_name())(_async_execution_logger(op.api_name())(op.execute))) #
execute exists on Operation base class and calls provision
finally:
if op._operation and not op.has_provisioning_phase: # changing the order
of this could avoid boolean evaluation of op._operation in the case we
spawned a thread
op._operation.execution_complete() # does not
get called in the operations we see this error in
On Fri, Sep 5, 2014 at 9:21 AM, Michael Bayer <mike...@zzzcomputing.com>
wrote:
>
> On Sep 5, 2014, at 12:03 PM, Lonnie Hutchinson <lonn...@skytap.com> wrote:
>
>
>
>
> On Thu, Sep 4, 2014 at 5:53 PM, Michael Bayer<mike...@zzzcomputing.com>
> wrote:
>
>>
>> > The code I say appears to expect the session_id to remain on detached
>> states in certain situations is that the check in Session._attach checks
>> not only the session_id but that the referenced session still exists in
>> _sessions:
>> > if state.session_id and \
>> > state.session_id is not self.hash_key and \
>> > state.session_id in _sessions:
>> > raise sa_exc.InvalidRequestError(
>> > "Object '%s' is already attached to session '%s' "
>> > "(this is '%s')" % (orm_util.state_str(state),
>> > state.session_id, self.hash_key))
>> >
>> > I am interested in this in the hopes that sheds light on the source of
>> the intermittent failures.
>>
>> yeah that code is intended to detect when an object is added to a session
>> that hasn't been removed from another.
>>
>>
> So, is the intent of the 'state.session_id in _sessions" clause to handle
> the situation where the session is collected while it contains instances?
>
>
> it’s to handle the case where Session A was garbage collected, not cleanly
> closed, and now an object from that now-gone session wants to be added to a
> different session. this might not even be a case we should be handling,
> perhaps if we changed the scheme for sessions to be an absolutely unique
> counter and just made it more strict. Not a change planned for now
> though...
>
>
>
>>
>> I agree, it appears to be pretty clear. However, on the application side
> I have log messages indicating I successfully commit() and close() the
> session before receiving an error saying an instance is still attached to
> it:
>
> 2014-08-13 14:14:11 tuk1m1cm1 [debug] [DEBUG] db.py:57 - transaction 8710
> started
>
> 2014-08-13 14:14:12 tuk1m1cm1 [debug] [DEBUG] db.py:143 - transaction 8710
> committed
>
> 2014-08-13 14:14:12 tuk1m1cm1 [debug] [DEBUG] db.py:57 - transaction 8719
> started
>
> 2014-08-13 14:14:13 tuk1m1cm1 [debug] [DEBUG] db.py:140 - transaction 8719
> rolled back2014-08-13 14:14:13 tuk1m1cm1 [err] [ERROR] threadutils.py:98 -
> Error executing asynchronous operation asynch-op <function execute at
> 0x5e54c08>(, {}) : Object '<Account at 0x5d49250>' is already attached to
> session '8710' (this is '8719') at Traceback (most recent call last):..
>
>
> Again I see lots terms here “threadutils”, “async operations”, things that
> suggest the application has a complex system of concurrency which may be
> misbehaving. I think you should be adding more assertions to your code to
> ensure that close() has completed fully before an object is attempted to be
> moved to another session.
>
>
>
> "transaction" refers to a session, the integer is the session.hash_key.
>
>
>> > In debugging this I have noticed that after the first session has been
>> closed() and the initializer has returned the session remains in _sessions.
>> However, if I call gc.collect() the session is removed, suggesting it just
>> hasn't been fully cleaned up yet.
>>
>> that's not really accurate. There's a collection of all the Sessions in
>> the weak referencing map _sessions, and Python's GC may not in fact remove
>> those sessions in a deterministic way. But that has no bearing upon
>> testing objects that have been removed from that session, and therefore
>> have no session_id, if they belong to that session - their session_id has
>> been cleared out.
>>
>>
> I believe this is relevant because if the session were to be GC'ed the
> "state.session_id in _sessions" would not evaluate to true and the attach
> to the session I want would succeed.
>
>
> This is essentially saying that a certain assertion being removed allows
> the code to “work”. But the assertion here is testing for a specific
> thing; that an object was not fully detached from a session before it is
> attempted to be used in another. I don’t see any indication that this
> assertion is malfunctioning.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at http://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
