Time and again I get this error and as a result the segment remains incomplete. This wastes one iteration of the for() loop in which I am doing generate, fetch and update.
Can someone please tell me what are the measures I can take to avoid this error? And isn't it possible to make some code changes so that the whole fetch doesn't have to stop suddenly when this error occurs. Can't we do something in the code so that, the fetch still continues like in case of SocketException, in which case the fetch while(1) loop continues. If it is not possible, please tell me how can I prevent this error from happening? ----- ERROR ----- fetch of http://telephony/register.asp failed with: java.lang.OutOfMemoryError: Java heap space java.lang.NullPointerException at org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(FSDataInputStream.java:87) at org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:125) ...... at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:115) fetcher caught:java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(FSDataInputStream.java:87) at org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:125) ....... at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:115) fetcher caught:java.lang.NullPointerException Fetcher: java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604) at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:470) at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:505) at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189) at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477) ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Nutch-developers mailing list Nutch-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-developers
