Hello, i hope you can help me
i am doing a crawling from a country, what i did was a script for avoid an interrupted process configured for 20 threads. But when a certain page is reached, the crawler behaves rarely stoping all threads (20) . Below is the log, no one message of fetcher done is at the end and the processor is at 98% of usage. I don't know what happens, the server's page is not blocking me (tested with wget) and i have enough disk space * queue: http://www.frente.com.mx maxThreads = 1 inProgress = 0 crawlDelay = 6000 minCrawlDelay = 0 nextFetchTime = 1338578382163 now = 1338578381900 0. http://www.frente.com.mx/2012/03/01/rockabilly-radiografia-de-un-copete/ fetching http://www.frente.com.mx/2012/03/01/rockabilly-radiografia-de-un-copete/ -finishing thread FetcherThread, activeThreads=19 -finishing thread FetcherThread, activeThreads=18 -finishing thread FetcherThread, activeThreads=17 -finishing thread FetcherThread, activeThreads=16 -finishing thread FetcherThread, activeThreads=15 -finishing thread FetcherThread, activeThreads=14 -finishing thread FetcherThread, activeThreads=13 -finishing thread FetcherThread, activeThreads=12 -finishing thread FetcherThread, activeThreads=11 -finishing thread FetcherThread, activeThreads=10 -finishing thread FetcherThread, activeThreads=9 -finishing thread FetcherThread, activeThreads=8 -finishing thread FetcherThread, activeThreads=7 -activeThreads=7, spinWaiting=6, fetchQueues.totalSize=0 -finishing thread FetcherThread, activeThreads=6 -finishing thread FetcherThread, activeThreads=5 -finishing thread FetcherThread, activeThreads=4 -finishing thread FetcherThread, activeThreads=3 -finishing thread FetcherThread, activeThreads=2 -finishing thread FetcherThread, activeThreads=1 -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0 -finishing thread FetcherThread, activeThreads=0 -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0 -activeThreads=0 -- View this message in context: http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381.html Sent from the Nutch - User mailing list archive at Nabble.com.
