scrapy_sentry is sending errors to scrapy, not all logs. Sentry is not the right tool for what you're trying to achieve. For that purpose I will suggest you logstash with kibana3.
Cheers, Jordi 2014-03-25 21:13 GMT+11:00 <[email protected]>: > I want to jack the scrapy logs into my Sentry instance as well but cant > seem to make it report anything to Sentry. > Created a few errors in the crawler with the msg method that is available > in scrapy.log.py . > Is there anything else I can try? > > > On Tuesday, 26 November 2013 05:18:32 UTC+1, Jordi Llonch wrote: > >> Try and raise an exception anywhere in your crawler and check if it's >> logged. >> >> Cheers, >> Jordi >> >> >> 2013/11/26 Jordi Llonch <[email protected]> >> >>> Hi D, >>> >>> I am the maintainer. Have you got the SENTRY_DSN defined in the >>> settings.py or as an environment variable? Have you setup the EXTENSIONS in >>> settings.py? >>> >>> Thanks, >>> Jordi >>> >>> >>> 2013/11/26 D <[email protected]> >>> >>>> I'm trying to get scrapy-sentry to work, and even though the debug info >>>> shows that the extension has been loaded (which means settings got picked >>>> up), and no errors seem to pop up in the logs (but plenty of scrapy-related >>>> information), and the DSN entry is freshly copied from sentry api page, >>>> nothing shows up in my sentry dashboard... (hitting sentry manually >>>> through raven seems to work just fine, so connection is valid). >>>> >>>> is this project still maintained? is there anything else i can try to >>>> troubleshoot this? >>>> >>>> Thanks, >>>> -A >>>> >>>> >>>> On Monday, February 4, 2013 4:09:58 PM UTC-5, Jordi Llonch wrote: >>>>> >>>>> Hi, >>>>> >>>>> There's a new contributed package: scrapy-sentry that glues scrapy and >>>>> sentry. >>>>> https://github.com/llonchj/scrapy-sentry<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fllonchj%2Fscrapy-sentry&sa=D&sntz=1&usg=AFQjCNGKTigEmgXjYu9qwnZ4k5aj2FXqdA> >>>>> >>>>> Comments are welcome. >>>>> >>>>> Jordi >>>>> >>>>> >>>>> El miércoles, 23 de enero de 2013 16:26:19 UTC+11, Pablo Hoffman >>>>> escribió: >>>>>> >>>>>> If you want to catch only spider errors, I would consider writing a >>>>>> spider middleware and implementing the >>>>>> process_spider_exception()<http://www.google.com/url?q=http%3A%2F%2Fdoc.scrapy.org%2Fen%2Flatest%2Ftopics%2Fspider-middleware.html%23scrapy.contrib.spidermiddleware.SpiderMiddleware.process_spider_exception&sa=D&sntz=1&usg=AFQjCNEl6BhefxMrrYuC7sBvdwu1vCUFeg>method. >>>>>> >>>>>> If you want to catch all kind of errors, I would consider writing a >>>>>> custom log observer. This is an undocumented feature, but check the >>>>>> following links for more info about it: >>>>>> - source code of scrapy.log >>>>>> module<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fscrapy%2Fscrapy%2Fblob%2Fmaster%2Fscrapy%2Flog.py&sa=D&sntz=1&usg=AFQjCNFTYEGZEQlDMOeieX5Zx94apGwxpQ> >>>>>> - how twisted logging >>>>>> works<http://www.google.com/url?q=http%3A%2F%2Ftwistedmatrix.com%2Fdocuments%2F12.2.0%2Fcore%2Fhowto%2Flogging.html&sa=D&sntz=1&usg=AFQjCNHFneesMvaHIngqueJZLSpR8eE1-Q> >>>>>> (scrapy >>>>>> uses twisted logging) >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Jan 19, 2013 at 1:05 AM, clj <[email protected]> wrote: >>>>>> >>>>>>> i want to make scrapy working with sentry ( >>>>>>> https://www.getsentry.com/docs/<https://www.google.com/url?q=https%3A%2F%2Fwww.getsentry.com%2Fdocs%2F&sa=D&sntz=1&usg=AFQjCNE6nyqrOJcGu5h2UeJ3mCScF9E2ZA> >>>>>>> , an event logging system) >>>>>>> catch all errors happened in scrapy, then send to sentry. >>>>>>> >>>>>>> my thought is: >>>>>>> i can mixin some sentry's error recording code in >>>>>>> https://github.com/scrapy/scrapy/blob/master/scrapy/core/scraper.py<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fscrapy%2Fscrapy%2Fblob%2Fmaster%2Fscrapy%2Fcore%2Fscraper.py&sa=D&sntz=1&usg=AFQjCNE8_XFUVPZ4gPhHjoSGuXMdBZsj8w>at >>>>>>> line 192 and 208 >>>>>>> >>>>>>> def _log_download_errors(self, spider_failure, download_failure, >>>>>>> request, spider): >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> """Log and silence errors that come from the engine (typically >>>>>>> download >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> errors that got propagated thru here) >>>>>>> """ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> if spider_failure is download_failure: >>>>>>> errmsg = spider_failure.getErrorMessage() >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> if errmsg: >>>>>>> 192 *log.msg(format='Error downloading %(request)s: >>>>>>> %(errmsg)s',* >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> * level=log.ERROR, spider=spider, >>>>>>> request=request, errmsg=errmsg)* >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> return >>>>>>> return spider_failure >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> def _itemproc_finished(self, output, item, response, spider): >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> """ItemProcessor finished for the given ``item`` and returned >>>>>>> ``output`` >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> """ >>>>>>> self.slots[spider].itemproc_size -= 1 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> if isinstance(output, Failure): >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ex = output.value >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> if isinstance(ex, DropItem): >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> logkws = self.logformatter.dropped(item, ex, response, >>>>>>> spider) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> log.msg(spider=spider, **logkws) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> return >>>>>>> self.signals.send_catch_log_deferred(signal=signals.item_dropped, \ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> item=item, spider=spider, exception=output.value) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> else: >>>>>>> 208 *log.err(output, 'Error processing %s' % item, >>>>>>> spider=spider)* >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> else: >>>>>>> logkws = self.logformatter.scraped(output, response, spider) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> log.msg(spider=spider, **logkws) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> return >>>>>>> self.signals.send_catch_log_deferred(signal=signals.item_scraped, \ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> item=output, response=response, spider=spider) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> question: >>>>>>> 1, is my consider right? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2, a better or more nature way 。(maybe in middleware? but how?) >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "scrapy-users" group. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msg/scrapy-users/-/5SpOkH_OHEoJ. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected]. >>>>>>> For more options, visit this group at http://groups.google.com/group >>>>>>> /scrapy-users?hl=en. >>>>>>> >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "scrapy-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/scrapy-users. >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> >> -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
