thanks - however, i am using the regular http-protocol plugin. everytime it
gave a few fatal npe following the hung threads and then died. :( not sure
what to do about it!
On 6/20/07, charlie w <[EMAIL PROTECTED]> wrote:
I've seen this issue when the fetcher would attempt to retrieve huge files
(>100MB). The incredible amount of time required to fetch files this
large caused the individual fetcher thread to appear hung to the thread
monitoring the fetcher status.
In my particular case, the only reason the fetcher was retrieving such a
huge file is that the content length limit is broken for the http-client
protocol plugin (it works in the regular http protocol plugin). See
Nutch-481.
On 6/19/07, Sunnyvale Fl <[EMAIL PROTECTED]> wrote:
>
> Hi all,
>
> I upgraded from Nutch 0.8.1 to 0.9 and thought the hung thread issues
are
> fixed in this version (NUTCH-344), but I am still seeing the same
problem.
> I am only running 10 threads for a focused crawl, and have the default
> fetcher.max.crawl.delay of 30. Every time it hangs, it is followed by a
> NPE
> - any clue? Thanks!
>
> 2007-06-19 14:40:37,670 WARN fetcher.Fetcher - Aborting with 3 hung
> threads.
> 2007-06-19 15:01:46,518 FATAL fetcher.Fetcher -
> java.lang.NullPointerException
> 2007-06-19 15:01:46,545 FATAL fetcher.Fetcher - at
> org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(
> FSDataInputStream.java
> :87)
> 2007-06-19 15:01:46,546 FATAL fetcher.Fetcher - at
> org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java
:125)
> 2007-06-19 15:01:46,546 FATAL fetcher.Fetcher - at
> org.apache.hadoop.io.SequenceFile$Reader.getPosition(SequenceFile.java
> :1736)
> 2007-06-19 15:01:46,546 FATAL fetcher.Fetcher - at
> org.apache.hadoop.mapred.SequenceFileRecordReader.getProgress(
> SequenceFileRecordReader.java:108)
> 2007-06-19 15:01:46,546 FATAL fetcher.Fetcher - at
> org.apache.hadoop.mapred.MapTask$1.getProgress(MapTask.java:165)
> 2007-06-19 15:01:46,546 FATAL fetcher.Fetcher - at
> org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:155)
> 2007-06-19 15:01:46,546 FATAL fetcher.Fetcher - at
> org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:115)
> 2007-06-19 15:01:46,546 FATAL fetcher.Fetcher - fetcher caught:
> java.lang.NullPointerException
>
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
