I think someting like this has already been done (apart from the daily changes you suggest) http://issues.apache.org/jira/browse/NUTCH-207
Rgrds, Thomas On 5/1/06, Fankhauser, Alain <[EMAIL PROTECTED]> wrote:
Hello I'm thinking about to create a throttle, who let us decide at witch day with wich speed (MB/S) and with wich number of connections (one thread = one connection) the fetcher fetches. That means, we have settings for every day and if there are no settings for a time at a day, then the fetcher would will make a break till we get settings. The target of this throttle is to controlle the fetcherspeed. If we fetch too fast, we just put to sleep a few (percent calculation) threads. If we are too slow, we just wake a few threads up. About using the throttle, i have also a few ideas. * The first idea is to set the throttle with -throttleDescription [path of throttleDescription] so the throttle reads in the description. * -throttleDescription without anything, so the throttle takes the path of the conf file nutch-site.xml * the user doesn't like to use my throttle, so he doesn't add any parameter. maybe you think that this is a good idea. please give me your feedback thanks and greetings Alain