Hello, I am creating a scraper for fun that is pulling all the statuses of businesses from a directory. I have a list of URLs which I have ran through the scrapy shell to ensure that it would pull the data as all the code is the same for each URL. For some odd reason the data is not being pulled from certain URLs, again I ran the ones that were not working through scrapy shell and have confirmed the same code would pull the statuses. Below I have pasted the message that’s being displayed from a few URLs when I run my spider. Not sure what I am doing wrong but any help would be great. I have copied my code to paste bin for you to review and test. http://pastebin.com/UBEUvbXU
2015-08-11 22:27:38 [scrapy] DEBUG: Crawled (200) <GET http://search.sunbiz.org/Inquiry/CorporationSearch/SearchResults?inquiryType=EntityName&searchNameOrder=d&searchTerm=d> (referer: None) 2015-08-11 22:27:38 [scrapy] DEBUG: Crawled (200) <GET http://search.sunbiz.org/Inquiry/CorporationSearch/SearchResults?inquiryType=EntityName&searchNameOrder=c&searchTerm=c> (referer: None) 2015-08-11 22:27:38 [scrapy] DEBUG: Crawled (200) <GET http://search.sunbiz.org/Inquiry/CorporationSearch/SearchResults?inquiryType=EntityName&searchNameOrder=g&searchTerm=g> (referer: None) 2015-08-11 22:27:38 [scrapy] DEBUG: Crawled (200) <GET http://search.sunbiz.org/Inquiry/CorporationSearch/SearchResults?inquiryType=EntityName&searchNameOrder=e&searchTerm=e> (referer: None) 2015-08-11 22:27:38 [scrapy] DEBUG: Crawled (200) <GET http://search.sunbiz.org/Inquiry/CorporationSearch/SearchResults?inquiryType=EntityName&searchNameOrder=b&searchTerm=b> (referer: None) -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
