Hi, I think the new sorting directly after the map job in hadoop-0.10.x causes this. I had the same problem. You could check io.sort.factor and io.sort.mb in conf/hadoop-site.xml. Maybe lower atleast io.sort.mb ?
Maybe that helps? - Espen Gal Nitzan wrote: > Hi Sean, > > Thanks for the prompt reply. > > I'm using fc6 java 1.6.0, 8GB RAM. > > I'll try your suggestion. > > Gal > > > -----Original Message----- > From: Sean Dean [mailto:[EMAIL PROTECTED] > Sent: Friday, January 19, 2007 8:25 PM > To: [email protected] > Subject: Re: java.lang.OutOfMemoryError - trunk > > What OS are you using with Nutch, and what version of JVM? > > If its Linux, paste the output of "ulimit -a", if its BSD use "limits". > > You can also try inserting "-Xms2000m" before you set the max heap, so it > would look like "-Xms2000m -Xmx2000m". > > I'm also assuming you have at least 2g free of RAM, or even more? > > ----- Original Message ---- > From: Gal Nitzan <[EMAIL PROTECTED]> > To: [email protected] > Sent: Friday, January 19, 2007 10:57:01 AM > Subject: java.lang.OutOfMemoryError - trunk > > > Thanks Sean, > > I get out of memory errors. > > I have set max heap for both nutch and hadoop 2000mb each but it doesn't > seem to affect anything. The out of memory happenes immediately after start > of a task. > > Any idea? > > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2786) > at > java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at org.apache.hadoop.io.Text.writeString(Text.java:399) > at org.apache.nutch.parse.Outlink.write(Outlink.java:52) > at org.apache.nutch.parse.ParseData.write(ParseData.java:163) > at org.apache.nutch.parse.ParseImpl.write(ParseImpl.java:55) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:323) > at org.apache.nutch.parse.ParseSegment.map(ParseSegment.java:96) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:183) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367) > > > ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
