Hey Daniel, good deal, vikings are the coolest! A few comments:
first, your static class code is executed at module import time (not spider runtime) and is probably better suited living in the class's __init__ or perhaps a module function called from __init__. second, you don't need to use `ult` as a global variable, in your parse method, this is what `self` is for. also, instead of encoding `desc` to str it is probably better to decode `ult` and work in unicode. use `except:` (with no named exception) with caution, it can hide valid errors and make debugging harder And now for the Scrapy stuff: In recent versions of Scrapy `response.selector.xpath` can be written as `response.xpath`. Working with text files in your parse() method is dangerous due to the weirdness of "async" libraries like Scrapy. Ok, keep up the good work. +steven On Friday, February 5, 2016 at 10:58:40 AM UTC-7, dnl 31337 wrote: > > Hi all.. i'm new on this list and new using scrapy framework.. I'm using > scrapy to make my project, and i want to share with you guys that are more > experient developers to take a look.. > > > What is my project? > The name is Viking Boat (https://github.com/twattacker/VikingBoat). > Viking Boat is a package with lot of webcrawlers that crawls websites that > publish cve, exploits and all king of vulnerabilitys. Grep all this > information and publish them into a webpage on facebook (i'm using facebook > api here). So, you don't need to go to each website to know the new vulns. > All new vulns will be concentrated and published in one place. > > Video demo of how works: https://www.youtube.com/watch?v=PnlfCQ1beYM > > For now i make crawlers to exploit-db website.. But I will make to > others.. So any sugestion will be welcome.. > > > Thanks all.. > > daniel(twattacker) > > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
