Nope...as proven earlier, this is a web2py timing issue. It's been
occurring randomly in production for some time now. I have a try/except
surrounding it, but the sucky part is that since the transaction fails, you
lose all the commits in the whole request.
On Friday, February 27, 2015 at 2:19:48 AM UTC-8, mcm wrote:
>
> Try too see if in production there is a transaction that stays open for a
> long time.
> It could be a console left open somewhere?
>
>
> 2015-02-27 8:42 GMT+01:00 Osman Masood <oama...@gmail.com <javascript:>>:
>
>> Right. The purpose of using the console was to simulate what actually
>> happens in production.
>>
>> On Thu, Feb 26, 2015 at 11:39 PM, Niphlod <nip...@gmail.com <javascript:>
>> > wrote:
>>
>>> you know that on console, after queue_task() you need to commit(), right
>>> ? If you leave the transaction hanging, no wonder that you're observing
>>> deadlocks!
>>>
>>>
>>> On Friday, February 27, 2015 at 3:38:19 AM UTC+1, Osman Masood wrote:
>>>>
>>>> I was actually able to easily reproduce this. Just open up 2 web2py
>>>> consoles (which connect to the same DB), and do:
>>>>
>>>> scheduler.queue_task('myfunc', pvars=dict(), immediate=True)
>>>>
>>>> The first one will succeed, and the second one will hang there for some
>>>> time, and then give the deadlock exception above.
>>>>
>>>> This kinda makes sense, looking at the above DB query & considering
>>>> record-level locking. We have conditional writes on all records in the
>>>> scheduler_worker table, so if two connections attempt this at the same
>>>> time, it would follow that there is a deadlock. (Correct me if I'm wrong.)
>>>> So I think the solution would be something like (on line 1303 of
>>>> scheduler.py):
>>>>
>>>> if immediate:
>>>>
>>>> - self.db(self.db.scheduler_worker.is_ticker ==
>>>> True).update(status=PICK)
>>>>
>>>> + scheduler_worker_tickers =
>>>> self.db(self.db.scheduler_worker.is_ticker
>>>> == True).select()
>>>>
>>>> + for scheduler_worker_ticker in
>>>> scheduler_worker_tickers:
>>>>
>>>> + scheduler_worker_ticker.update_record(status=PICK)
>>>>
>>>>
>>>> On Thursday, February 26, 2015 at 3:54:47 PM UTC-8, Osman Masood wrote:
>>>>>
>>>>> Thanks for the quick response!
>>>>>
>>>>> I'm using Amazon RDS (just basically MySQL), on a pretty beefy
>>>>> instance (db.m3.large), 1000 IOPS. Would be pretty surprised if that were
>>>>> the problem.
>>>>>
>>>>> Running 5 scheduler workers with the default heartbeat (3s).
>>>>>
>>>>> On Thursday, February 26, 2015 at 3:35:41 PM UTC-8, Niphlod wrote:
>>>>>>
>>>>>> it seems to me that there is database contention........ although the
>>>>>> exception **should** be trapped (i.e. try to set the status to "PICK",
>>>>>> if
>>>>>> not, well.... it's not a fatal error), there's something wrong with your
>>>>>> setup or your database is too much "underpowered" to do what you're
>>>>>> asking.
>>>>>> How many workers are running and with what heartbeat ?
>>>>>>
>>>>>> On Friday, February 27, 2015 at 12:28:28 AM UTC+1, Osman Masood wrote:
>>>>>>>
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I keep getting this error when calling scheduler.queue_task() with
>>>>>>> immediate=True:
>>>>>>>
>>>>>>> <class 'gluon.contrib.pymysql.err.InternalError'> (1213, u'Deadlock
>>>>>>> found when trying to get lock; try restarting transaction')
>>>>>>>
>>>>>>> The stack trace shows this is the culprit:
>>>>>>>
>>>>>>> self.db(self.db.scheduler_worker.is_ticker ==
>>>>>>> True).update(status=PICK)
>>>>>>>
>>>>>>> (This is a web application which uses the scheduler pretty
>>>>>>> extensively.) Seems like multiple connections are competing for access
>>>>>>> to the same scheduler_worker row.
>>>>>>>
>>>>>>> Seems to me like a web2py bug. Any help would be greatly appreciated.
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --
>>> Resources:
>>> - http://web2py.com
>>> - http://web2py.com/book (Documentation)
>>> - http://github.com/web2py/web2py (Source code)
>>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "web2py-users" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/web2py/ZBv43p3w_MM/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> web2py+un...@googlegroups.com <javascript:>.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to web2py+un...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
