I think someting like this has already been done (apart from the daily
changes you suggest) http://issues.apache.org/jira/browse/NUTCH-207
Rgrds, Thomas
On 5/1/06, Fankhauser, Alain <[EMAIL PROTECTED]> wrote:
Hello
I'm thinking about to create a throttle, who let us decide at
witch day with wich speed (MB/S) and with wich number of connections
(one thread = one connection) the fetcher fetches. That means, we have
settings for every day and if there are no settings for a time at a day,
then the fetcher would will make a break till we get settings.
The target of this throttle is to controlle the fetcherspeed.
If we fetch too fast, we just put to sleep a few (percent
calculation) threads. If we are too slow, we just wake a few threads up.
About using the throttle, i have also a few ideas.
* The first idea is to set the throttle with -throttleDescription
[path of throttleDescription] so the throttle reads in the description.
* -throttleDescription without anything, so the throttle takes the
path of the conf file nutch-site.xml
* the user doesn't like to use my throttle, so he doesn't add any
parameter.
maybe you think that this is a good idea.
please give me your feedback
thanks and greetings
Alain
