Adding commit_unless_managed() before the get() seems to fix the problem. Looking at it locally, the Gearman worker starts a transactions and just calls commit when the processing is done, over and over, never starting a new transaction. That combined with REPEATABLE-READ seems to be the culprit.
My next step was going to be to set the isolation level to READ-COMMITTED which would probably have fixed it as well. Thought I'm not sure how that would have affected performance. Will the changes to transactions in 1.6 possibly address this issue so I can remove the commit_unless_managed() call? It looks like the major change is using the underlying database's autocommit mode rather than emulating it in the ORM. I suppose I'll have to test... On Tuesday, May 28, 2013 5:43:31 PM UTC-4, Chris Conover wrote: > > I bribed the DBA to turn on the general query log briefly (which generated > 327k logs in 2 minutes and 2 seconds). Looking at the logs, I see the > transaction block for the INSERT. The SELECT is definitely coming after the > COMMIT in a different thread. I'll try the commit_unless_managed fix > quickly and if that doesn't work, I'll post the full query trace. > > On Tuesday, May 28, 2013 5:27:36 PM UTC-4, akaariai wrote: >> >> On 28 touko, 22:20, Chris Conover <clc...@gmail.com> wrote: >> > Well, you can inspect the object and see it's primary key. Surely that >> > means the INSERT is completed? So the workflow goes like this: >> > >> > foo = Foo() >> > foo.save() >> > foo.pk # new primary key is available >> > submit_gearman_task(foo.pk) >> > >> > Then in the Gearman worker: >> > >> > foo = Foo.objects.get(pk=foo_pk) # this causes a Foo.DoesNotExist >> exception >> > >> > That's what I don't understand. The primary key is assigned and >> available >> > in the application layer in one place but then the lookup using that >> > primary key fails in another place. Maybe it has something to do with >> the >> > fact that the database is set to REPEATABLE-READ isolation level versus >> > READ-COMMITTED. Still investigating... >> >> Could it be that the Gearman worker is in a transaction? When you >> issue Foo.objects.get(), if the transaction Gearman is in has started >> before the save() you will not see the new object (as MySQL uses >> REPEATABLE READ transaction). >> >> Note that any read query will start a transaction implicitly, and ORM >> writes commit implicitly started transaction. You can also use >> connection.commit_unless_managed() to commit implicitly started >> transaction (not part of public API!). So, even if you haven't >> explicitly started any transactions in Gearman worker an earlier read >> query might have started one for you. (If you find this behavior >> confusing this is one of the reasons why transaction handling will be >> changed in Django 1.6). >> >> In short: try if connection.commit_unless_managed() before the get() >> changes anything. If so, investigate why there is an ongoing >> transaction in the worker. >> >> - Anssi >> > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
