Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-27 Thread Cecil Rock
Mike, We've upgraded to that latest and greatest SQLAlchemy, did some instrumentation and confirmed we were getting bit in _fast_discard. Thank you for your very prompt responses, we really appreciate your help. Cecil Rock On Mon, Feb 26, 2018 at 6:23 PM, Mike Bayer wrote: > On Mon, Feb 26, 2

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
On Mon, Feb 26, 2018 at 4:51 PM, Cecil Rock wrote: > Hi Mike, > > Another colleague of mine sent me this write up which we believe is the root > cause of our issue. > > In sqlalchemy/orm/loading.py: > > > > Function _instance() defined in _instance_processor() checks if identitykey > is in session

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Cecil Rock
Hi Mike, Another colleague of mine sent me this write up which we believe is the root cause of our issue. In sqlalchemy/orm/loading.py: Function _instance() defined in _instance_processor() checks if identitykey is in session_identity_map by calling session_identity_map.get(identitykey). This

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
I can't get it to simulate your issue though. but the test case on 4068 is pretty complicated and I don't remember how that one works either, but it looks a lot like the safer checks post 4068 may fix the issue. On Mon, Feb 26, 2018 at 4:06 PM, Cecil Rock wrote: > OH, man sorry I misunderstoo

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Cecil Rock
OH, man sorry I misunderstood that issue. I have been starring at this for too long :) Yes, that totally could be it. I'll apply that patch and should be able to confirm it or not. On Mon, Feb 26, 2018 at 1:02 PM, Mike Bayer wrote: > On Mon, Feb 26, 2018 at 3:58 PM, Cecil Rock wrote: > > Yea

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
On Mon, Feb 26, 2018 at 3:58 PM, Cecil Rock wrote: > Yeah I saw those issues, but they did not seem like they would be related to > what we're seeing :( > > I had instrumented the identity_map earlier and could not catch an instance > where it was doing the wrong thing. The GC being guilty or may

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Cecil Rock
Yeah I saw those issues, but they did not seem like they would be related to what we're seeing :( I had instrumented the identity_map earlier and could not catch an instance where it was doing the wrong thing. The GC being guilty or maybe an obscure bug higher in the stack feels more likely. It

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
On Mon, Feb 26, 2018 at 3:33 PM, Cecil Rock wrote: > Fairly old, v1.0.14. Its running in one of our older microservices. Once I > understood what was going on, I went through the fix list and could not find > anything obviously related to it. We can't reproduce this at will and > nothing obviou

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Cecil Rock
Fairly old, v1.0.14. Its running in one of our older microservices. Once I understood what was going on, I went through the fix list and could not find anything obviously related to it. We can't reproduce this at will and nothing obvious has changed, so we're worried that upgrading without u

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
um what version of SQLAlchemy is this? On Mon, Feb 26, 2018 at 3:22 PM, Cecil Rock wrote: > One of my thoughts was maybe the gc wiped out the identity_map and it gets > recreated empty. > > Here is a stack trace of when I detect the situation prior to the flush > occurring. > >> [4018/eval_0/INFO

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Cecil Rock
One of my thoughts was maybe the gc wiped out the identity_map and it gets recreated empty. Here is a stack trace of when I detect the situation prior to the flush occurring. [4018/eval_0/INFO/state/554/_modified_event] fc3a3a adding state.InstanceState object at 0x7f6a5bb989e8> as modified in

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
below are some assertions you could add, if these rules don't work exactly right then you could trip into that error condition diff --git a/lib/sqlalchemy/orm/session.py b/lib/sqlalchemy/orm/session.py index 5a27140ac..64da5072a 100644 --- a/lib/sqlalchemy/orm/session.py +++ b/lib/sqlalchemy/orm/s

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
if you can at least share the stack trace that should tell a lot On Mon, Feb 26, 2018 at 3:05 PM, Mike Bayer wrote: > On Mon, Feb 26, 2018 at 2:54 PM, Cecil Rock wrote: >> Hi, >> >> I have a question about some ORM internal state tracking stuff. If this >> isn't the right venue for this type of

Re: [sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Mike Bayer
On Mon, Feb 26, 2018 at 2:54 PM, Cecil Rock wrote: > Hi, > > I have a question about some ORM internal state tracking stuff. If this > isn't the right venue for this type of question my apologies. > > I've run into what looks like a race condition using the ORM during some our > production worklo

[sqlalchemy] InstanceState dirty but will not flush

2018-02-26 Thread Cecil Rock
Hi, I have a question about some ORM internal state tracking stuff. If this isn't the right venue for this type of question my apologies. I've run into what looks like a race condition using the ORM during some our production workloads. I am trying to figure out if our code has a bug or if