Hi thanks for the sugestion , but as i am not that much familiar with linux can u please tell me what is throttling firewall and how to deploy it , more over i would like to add that i had already change the fetcher.threads.fetch property to 1 but it was still using the same bandbith.
Sean Dean-3 wrote: > > I guess your looking to limit the bandwidth used by the Nutch crawler? > > One idea is to set a "very low" thread count for the fetcher, maybe even > going right down to 1 depending on the effect your looking for. Another > idea, which would be much more controlled is to setup a throttling > firewall on the same machine or anywhere in-between it and your gateway. > > If you go the thread route, you will need to place the below lines in your > nutch-site.xml file; > > <property> > <name>fetcher.threads.fetch</name> > <value>1</value> > <description>The number of FetcherThreads the fetcher should use. > This is also determines the maximum number of requests that are > made at once (each FetcherThread handles one > connection).</description> > </property> > > > ----- Original Message ---- > From: Harmesh <[EMAIL PROTECTED]> > To: [email protected] > Sent: Wednesday, March 7, 2007 2:01:50 AM > Subject: memory consumpition by nutch > > > while i am crawling most of the band with is used by nutch crawler is > there > any way to optimize it > -- > View this message in context: > http://www.nabble.com/memory-consumpition-by-nutch-tf3360441.html#a9347708 > Sent from the Nutch - User mailing list archive at Nabble.com. > -- View this message in context: http://www.nabble.com/memory-consumpition-by-nutch-tf3360441.html#a9349223 Sent from the Nutch - User mailing list archive at Nabble.com. ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
