There could be many things causing the slowdown on the perceived faster server.
You should be aware that under normal operation Nutch only uses one processor,
the process can jump around and execute on any of the processors you have but
never more then one at any time.
You should watch the output of "top" or something similar to see what the
slowdown might be, if your running other things on the server that might also
be the cause. For you to see an increase of 4-5 hours makes it sound like its
more then just DNS look-ups slowing you down.
Is your hosted server shared? Do they throttle your connection, processor time,
or memory?
----- Original Message ----
From: sdeck <[EMAIL PROTECTED]>
To: [email protected]
Sent: Tuesday, March 6, 2007 5:08:34 PM
Subject: Crawl slow on one machine, fast on another
Hey all,
Just curious if anyone has any ideas on what to test for on this weird
issue for me. On my laptop from home, I have a list of about 140K urls to
inject, and then I get back roughly 12000 urls to retrieve/parse/index. This
takes about an hour. Sometimes less. I have a cable modem at home. Now, when
I then push this exact same code to my hosted server (dual proc, 2 gig ram.
Basically, a real server) it takes roughly 5-6 hours to do the exact same
thing.
Has anyone run into this? Ideas on what I could check to see where the bog
down could be at?
If it was a dns issue, how would I go about checking look up times? (I have
seen this as a possible issue in the forums)
Thanks,
Scott
--
View this message in context:
http://www.nabble.com/Crawl-slow-on-one-machine%2C-fast-on-another-tf3358653.html#a9342187
Sent from the Nutch - User mailing list archive at Nabble.com.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
