I want to jack the scrapy logs into my Sentry instance as well but cant seem to make it report anything to Sentry. Created a few errors in the crawler with the msg method that is available in scrapy.log.py . Is there anything else I can try?
On Tuesday, 26 November 2013 05:18:32 UTC+1, Jordi Llonch wrote: > > Try and raise an exception anywhere in your crawler and check if it's > logged. > > Cheers, > Jordi > > > 2013/11/26 Jordi Llonch <[email protected] <javascript:>> > >> Hi D, >> >> I am the maintainer. Have you got the SENTRY_DSN defined in the >> settings.py or as an environment variable? Have you setup the EXTENSIONS in >> settings.py? >> >> Thanks, >> Jordi >> >> >> 2013/11/26 D <[email protected] <javascript:>> >> >>> I'm trying to get scrapy-sentry to work, and even though the debug info >>> shows that the extension has been loaded (which means settings got picked >>> up), and no errors seem to pop up in the logs (but plenty of scrapy-related >>> information), and the DSN entry is freshly copied from sentry api page, >>> nothing shows up in my sentry dashboard... (hitting sentry manually >>> through raven seems to work just fine, so connection is valid). >>> >>> is this project still maintained? is there anything else i can try to >>> troubleshoot this? >>> >>> Thanks, >>> -A >>> >>> >>> On Monday, February 4, 2013 4:09:58 PM UTC-5, Jordi Llonch wrote: >>>> >>>> Hi, >>>> >>>> There's a new contributed package: scrapy-sentry that glues scrapy and >>>> sentry. >>>> https://github.com/llonchj/scrapy-sentry<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fllonchj%2Fscrapy-sentry&sa=D&sntz=1&usg=AFQjCNGKTigEmgXjYu9qwnZ4k5aj2FXqdA> >>>> >>>> Comments are welcome. >>>> >>>> Jordi >>>> >>>> >>>> El miércoles, 23 de enero de 2013 16:26:19 UTC+11, Pablo Hoffman >>>> escribió: >>>>> >>>>> If you want to catch only spider errors, I would consider writing a >>>>> spider middleware and implementing the >>>>> process_spider_exception()<http://www.google.com/url?q=http%3A%2F%2Fdoc.scrapy.org%2Fen%2Flatest%2Ftopics%2Fspider-middleware.html%23scrapy.contrib.spidermiddleware.SpiderMiddleware.process_spider_exception&sa=D&sntz=1&usg=AFQjCNEl6BhefxMrrYuC7sBvdwu1vCUFeg>method. >>>>> >>>>> If you want to catch all kind of errors, I would consider writing a >>>>> custom log observer. This is an undocumented feature, but check the >>>>> following links for more info about it: >>>>> - source code of scrapy.log >>>>> module<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fscrapy%2Fscrapy%2Fblob%2Fmaster%2Fscrapy%2Flog.py&sa=D&sntz=1&usg=AFQjCNFTYEGZEQlDMOeieX5Zx94apGwxpQ> >>>>> - how twisted logging >>>>> works<http://www.google.com/url?q=http%3A%2F%2Ftwistedmatrix.com%2Fdocuments%2F12.2.0%2Fcore%2Fhowto%2Flogging.html&sa=D&sntz=1&usg=AFQjCNHFneesMvaHIngqueJZLSpR8eE1-Q> >>>>> (scrapy >>>>> uses twisted logging) >>>>> >>>>> >>>>> >>>>> On Sat, Jan 19, 2013 at 1:05 AM, clj <[email protected]> wrote: >>>>> >>>>>> i want to make scrapy working with sentry (https://www.getsentry.com/ >>>>>> docs/<https://www.google.com/url?q=https%3A%2F%2Fwww.getsentry.com%2Fdocs%2F&sa=D&sntz=1&usg=AFQjCNE6nyqrOJcGu5h2UeJ3mCScF9E2ZA> >>>>>> , an event logging system) >>>>>> catch all errors happened in scrapy, then send to sentry. >>>>>> >>>>>> my thought is: >>>>>> i can mixin some sentry's error recording code in >>>>>> https://github.com/scrapy/scrapy/blob/master/scrapy/core/scraper.py<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fscrapy%2Fscrapy%2Fblob%2Fmaster%2Fscrapy%2Fcore%2Fscraper.py&sa=D&sntz=1&usg=AFQjCNE8_XFUVPZ4gPhHjoSGuXMdBZsj8w>at >>>>>> line 192 and 208 >>>>>> >>>>>> def _log_download_errors(self, spider_failure, download_failure, >>>>>> request, spider): >>>>>> >>>>>> >>>>>> >>>>>> """Log and silence errors that come from the engine (typically >>>>>> download >>>>>> >>>>>> >>>>>> >>>>>> errors that got propagated thru here) >>>>>> """ >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> if spider_failure is download_failure: >>>>>> errmsg = spider_failure.getErrorMessage() >>>>>> >>>>>> >>>>>> >>>>>> if errmsg: >>>>>> 192 *log.msg(format='Error downloading %(request)s: >>>>>> %(errmsg)s',* >>>>>> >>>>>> >>>>>> >>>>>> * level=log.ERROR, spider=spider, >>>>>> request=request, errmsg=errmsg)* >>>>>> >>>>>> >>>>>> >>>>>> return >>>>>> return spider_failure >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> def _itemproc_finished(self, output, item, response, spider): >>>>>> >>>>>> >>>>>> >>>>>> """ItemProcessor finished for the given ``item`` and returned >>>>>> ``output`` >>>>>> >>>>>> >>>>>> >>>>>> """ >>>>>> self.slots[spider].itemproc_size -= 1 >>>>>> >>>>>> >>>>>> >>>>>> if isinstance(output, Failure): >>>>>> >>>>>> >>>>>> >>>>>> ex = output.value >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> if isinstance(ex, DropItem): >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> logkws = self.logformatter.dropped(item, ex, response, >>>>>> spider) >>>>>> >>>>>> >>>>>> >>>>>> log.msg(spider=spider, **logkws) >>>>>> >>>>>> >>>>>> >>>>>> return >>>>>> self.signals.send_catch_log_deferred(signal=signals.item_dropped, \ >>>>>> >>>>>> >>>>>> >>>>>> item=item, spider=spider, exception=output.value) >>>>>> >>>>>> >>>>>> >>>>>> else: >>>>>> 208 *log.err(output, 'Error processing %s' % item, >>>>>> spider=spider)* >>>>>> >>>>>> >>>>>> >>>>>> else: >>>>>> logkws = self.logformatter.scraped(output, response, spider) >>>>>> >>>>>> >>>>>> >>>>>> log.msg(spider=spider, **logkws) >>>>>> >>>>>> >>>>>> >>>>>> return >>>>>> self.signals.send_catch_log_deferred(signal=signals.item_scraped, \ >>>>>> >>>>>> >>>>>> >>>>>> item=output, response=response, spider=spider) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> question: >>>>>> 1, is my consider right? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> 2, a better or more nature way 。(maybe in middleware? but how?) >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "scrapy-users" group. >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> msg/scrapy-users/-/5SpOkH_OHEoJ. >>>>>> To post to this group, send email to [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at http://groups.google.com/ >>>>>> group/scrapy-users?hl=en. >>>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scrapy-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To post to this group, send email to [email protected]<javascript:> >>> . >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
