Henri et al I needed to decouple NoRobotsClient from URLConnection in order to enable it to work with any arbitrary HTTP library (HttpClient in the first place). I also could not resist the temptation to make other changes to NoRobotsClient and to vandalize your code in process.
I eliminated two-phase parsing process that required the content to be buffered in memory. I also felt it might be useful to get a complete set of rules for all defined user agents, not just one. Please review the changes and let me know if you find anything disagreeable. Oleg
