When I try to do my crawl it seems like the threads get stuck in som spinwaiting mode. At first the crawl goes as planned, and I couldnt be happier. But after som time, it starts reporting more of these spinwaiting messages.
I print a log here to show you what it looks like. As you can see it gets stuck, and the queue decrease by 1 all the time. I've tried doing a smaller crawl, and what happends is that it counts down untill the "fetchQueues.totalSize" reaches 0, and then the crawl is done. But the problem is that this countdown is very slow,there's no effective crawling going on, not using eather bandwith or cpu power. Basicly, this costs way to much time, I cant let it go on like this for hours to be done. How can I fix this? after about an hour of crawling this is what the log looks like -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2526 -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2526 - fetching http://home.swipnet.se/~w-147200/ -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2525 -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2525 - fetching http://biphome.spray.se/alarsson/ -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2524 -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2524 -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2524 - fetching http://home.swipnet.se/~w-31853/html/ -activeThreads=1000, spinWaiting=1000, fetchQueues.totalSize=2523 .... -- View this message in context: http://www.nabble.com/threads-get-stuck-in-spinwaiting-tp23723825p23723825.html Sent from the Nutch - User mailing list archive at Nabble.com.
