Stefan Groschupf,
Thanks for this clear summarize of bandwith requirements. This could be great you to approximate server requirement for that case of crawling.
To fully make profit of a 100 MBit bandwith, how much RAM & how much threads should we have ? What kind of server would be more efficient ?
Christophe.
Stefan Groschupf wrote:
Lets do some calculation:
2 billion pages: (google has 8 billion)
100 kilobytes * 2 000 000 000 = 186.264515 terabytes per Month
1 * 100MBit per Month = 33.1776 TB
186 / 33 = 5.6
The cheapest offer for 100 MBit I found was 1000 USD per month.
So you pay 6000 USD per month just crawling without any user query.
If you _only_ have 1 million queries per day you have another 3 TB traffic.
Math.round(idea) = 20 .000 USD per Month in case all servers are in same location.
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
