I am building a crawler which will launch several instances with random IP addresses. I am currently using Amazon EC2 with multiple private IP addresses and each assigned to a public IP.
It is verified that it the network is configured correctly by using *curl --interface <ip_address> <url>*. Viewing the httpd log from another server proves that it is receiving requests with different IP on the same server. However, when used with bindaddress in scrapy, I found that all requests is only coming from the primary public IP. Any idea why? -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
