1. Yes, just keep the connection in auto commit. Also, know that you will have to ensure your pg server's availability. 2. Leftovers. Shouldn't matter.
On 27 December 2016 at 17:59, k bez <[email protected]> wrote: > Ok I patched *sqlite.py* module and SqlitePriorityQueue class and it > seems to work fine with Postgre. > But i have some questions if smone can answer. > 1. SQLite use parameter "check_same_thread=False" for connection. But I > think Postgre con is thread-safe so i didnt use smthing like this. > 2. Whats is the use of remove and clear methods of SqlitePriorityQueue > class? Queue worked fine before patch them as i couldnt find a call for > them. > > On Monday, December 26, 2016 at 10:45:25 AM UTC+2, Nikolaos-Digenis > Karagiannis wrote: >> >> Hi, >> >> You probably looked into some old documentation. >> This setting was introduced before scrapy separated from scrapyd. >> Job queues are now implemented in scrapyd. >> A pickaxe search confirms this: >> https://github.com/scrapy/scrapy/commit/75e2c3eb338ea03e4879 >> 07fa8c99bb12317e9435 >> This was a point were many release notes are missing, >> notes clarifying what was removed from scrapy, >> because the "separation of scrapyd" alone doesn't say much. >> >> Do you use scrapyd? >> Unfortunately, the job queue class is no longer configurable. >> It shouldn't be hard however to patch scrapyd, >> either to make it configurable or to use your own fork. >> >> Check out scrapyd's repository https://github.com/scrapy/scrapyd/ >> and if you come up with something >> don't hesitate to open a PR or an issue with suggestions. >> We'll be glad to help, >> scrapyd does need its components to become less tightly coupled >> and making the job queue configurable can contribute to this. >> >> >> >> On Monday, 26 December 2016 01:49:50 UTC+2, k bez wrote: >>> >>> I have started to implement a custom job queue so i ll move from default >>> SQLite to Postgre. >>> I use this setting SPIDER_QUEUE_CLASS = 'mysite.scraper.PostgreSQLQueue' >>> but scrapyd seems to ignore it and create/use default SQLite DB. >>> I want to ask if SPIDER_QUEUE_CLASS dont work anymore? >>> Thanks in advance. >>> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "scrapy-users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/scrapy-users/V8vshXijC5c/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
