Author: jnioche Date: Fri Jun 27 07:49:05 2014 New Revision: 1605979 URL: http://svn.apache.org/r1605979 Log: NUTCH-385 Improve description of thread related configuration for Fetcher
Modified: nutch/branches/2.x/CHANGES.txt nutch/branches/2.x/conf/nutch-default.xml Modified: nutch/branches/2.x/CHANGES.txt URL: http://svn.apache.org/viewvc/nutch/branches/2.x/CHANGES.txt?rev=1605979&r1=1605978&r2=1605979&view=diff ============================================================================== --- nutch/branches/2.x/CHANGES.txt (original) +++ nutch/branches/2.x/CHANGES.txt Fri Jun 27 07:49:05 2014 @@ -2,6 +2,8 @@ Nutch Change Log Current Development +* NUTCH-385 Improve description of thread related configuration for Fetcher (jnioche,lufeng) + * NUTCH-1798 Crawl script not calling index command correctly (Aaron Bedward via jnioche) * NUTCH-1769 REST API refactoring (Fjodor Vershinin via lewismc) Modified: nutch/branches/2.x/conf/nutch-default.xml URL: http://svn.apache.org/viewvc/nutch/branches/2.x/conf/nutch-default.xml?rev=1605979&r1=1605978&r2=1605979&view=diff ============================================================================== --- nutch/branches/2.x/conf/nutch-default.xml (original) +++ nutch/branches/2.x/conf/nutch-default.xml Fri Jun 27 07:49:05 2014 @@ -607,7 +607,10 @@ <name>fetcher.server.delay</name> <value>5.0</value> <description>The number of seconds the fetcher will delay between - successive requests to the same server.</description> + successive requests to the same server. Note that this might get + overriden by a Crawl-Delay from a robots.txt and is used ONLY if + fetcher.threads.per.queue is set to 1. + </description> </property> <property> @@ -615,7 +618,7 @@ <value>0.0</value> <description>The minimum number of seconds the fetcher will delay between successive requests to the same server. This value is applicable ONLY - if fetcher.threads.per.host is greater than 1 (i.e. the host blocking + if fetcher.threads.per.queue is greater than 1 (i.e. the host blocking is turned off).</description> </property> @@ -646,7 +649,12 @@ <name>fetcher.threads.per.queue</name> <value>1</value> <description>This number is the maximum number of threads that - should be allowed to access a queue at one time.</description> + should be allowed to access a queue at one time. Setting it to + a value > 1 will cause the Crawl-Delay value from robots.txt to + be ignored and the value of fetcher.server.min.delay to be used + as a delay between successive requests to the same server instead + of fetcher.server.delay. + </description> </property> <property>