Hi all, Is there a good way to get at the raw request headers sent by scrapy 1.0.x? I know I can recreate a functionally equivalent request from response.request.(cookies,headers,...) but for my application I really need the exact bytes that went out on the wire.
It seems that with the old HTTP/1.0 downloader this could be done by setting DOWNLOADER_HTTPCLIENTFACTORY and overriding ScrapyHTTPPageGetter. However the new HTTP/1.1 downloader uses a twisted Agent. The HTTP/1.1 downloader seems to listen to the DOWNLOAD_HANDLERS setting, but I don't see a way to get the same effect overriding HttpDownloadHandler without duplicating a whole bunch of code from both scrapy and twisted. Any pointers would be much appreciated. Best regards, Davíð -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
