Hi, I'm currently building a crawler framework called Gungho. So far I think the project is going great. Now I'm starting to think about optimizations, and one of them happens to revolve around PoCo::Client::HTTP.
In a past experiment, I've hacked on PoCo::Client::HTTP::Request so that it writes the data content to disk -- this greatly reduced the overhead on each process that was doing the crawling, since hundreds or thousands of requests may be sent all at once. But at that time it required actually hacking on the installed PoCo::Client::HTTP::Request source, or to cut n' paste a bunch of code from said module and doing the fack -- which wasn't such a great idea, because the internal dependencies were delicate enough that upgrading PoCo::Client::HTTP caused the module to break. I'd like to do something similar in Gungho, but this time I'd like to make it so that things don't break just because we upgraded PoCo::Client::HTTP. The easiest solution seems to be just to create a new type of PoCo::Client::HTTP::Request, so that while the bulk of the code may be identical to PoCo::Client::HTTP::Request, it can lead a different life cycle that doesn't depend on the other. To make this possible, I'd like to make PoCo::Client::HTTP able to change the type of RequestFactory it creates -- or, make RequestFactory be able to produce difference types of Requests. In either case, this should be a simple 3-4 line change. I don't mind doing the hacking myself, can you please tell me where I should start? Regards, --d
