Hi, I thought I replied to your question about the autocommit but it seems I didn't hit "post". Since in postgresql you can do "DELETE .... RETURNING ..." you can merge the "select" and "delete" queries in a single query without the need to execute any python code in the middle of the transaction.
But I looked into the matter in haste, now I see more problems with the builtin queue. The selected row can't disappear while the transaction is not yet committed. This would violate the isolation principle. The code that retries if the row went missing is written like it's meant for autocommit. What should happen in case of concurrent access, is having the sqlite database locked, possibly leading to a deadlock. Regarding the second issue: queue.pop can return `None` if all rows are vanished between the queue.count call and queue.pop This is probably your case. Let's continue on the issue tracker, tickets #40 and #197 because we are talking about bugs here. On Friday, 30 December 2016 21:23:59 UTC+2, k bez wrote: > > I patched two production servers with changes. > I use one server/scrapyd instance to populate shared queue with new > records and consume, and the 2nd one only consume records from shared queue. > So i have 2 scrapyd instances on different servers consume records from > same shared queue. > This scenario worked well for some hours but i am in front of this bug now. > > https://github.com/scrapy/scrapyd/issues/40 > > My error log > Traceback (most recent call last): > File > "/root/mysite/local/lib/python2.7/site-packages/twisted/internet/base.py", > line 825, in runUntilCurrent > call.func(*call.args, **call.kw) > File > "/root/mysite/local/lib/python2.7/site-packages/twisted/internet/task.py", > line 239, in __call__ > d = defer.maybeDeferred(self.f, *self.a, **self.kw) > File > "/root/mysite/local/lib/python2.7/site-packages/twisted/internet/defer.py", > line 149, in maybeDeferred > result = f(*args, **kw) > File > "/root/mysite/local/lib/python2.7/site-packages/twisted/internet/defer.py", > line 1331, in unwindGenerator > return _inlineCallbacks(None, gen, Deferred()) > --- <exception caught here> --- > File > "/root/mysite/local/lib/python2.7/site-packages/twisted/internet/defer.py", > line 1185, in _inlineCallbacks > result = g.send(result) > File > "/root/mysite/local/lib/python2.7/site-packages/scrapyd/poller.py", line > 24, in poll > returnValue(self.dq.put(self._message(msg, p))) > File > "/root/mysite/local/lib/python2.7/site-packages/scrapyd/poller.py", line > 33, in _message > d = queue_msg.copy() > exceptions.AttributeError: 'NoneType' object has no attribute 'copy' > > Instead of 2 spiders like original bug, i guess i have same bug cause of 2 > scrapyd instances with same spider queue. > Is there something that i can do to avoid it? > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
