you can also have an eye on the topN and number of fetching threads On Thu, Apr 9, 2009 at 4:48 PM, yanky young <[email protected]> wrote:
> why not just add -Xms -Xmx jvm parameters to see if it still happens > > > > 2009/4/9 srinivas jaini <[email protected]> > > > I've checked out code and am running crawl and get this error; any > > thoughts? > > environment: java 6, eclipse > > > > 009-04-08 01:22:41,658 INFO crawl.Injector (Injector.java:inject(136)) - > > Injector: starting > > 2009-04-08 01:22:41,658 INFO crawl.Injector (Injector.java:inject(137)) > - > > Injector: crawlDb: crawl/crawldb > > 2009-04-08 01:22:41,659 INFO crawl.Injector (Injector.java:inject(138)) > - > > Injector: urlDir: urls > > 2009-04-08 01:22:41,660 INFO crawl.Injector (Injector.java:inject(148)) > - > > Injector: Converting injected urls to crawl db entries. > > 2009-04-08 01:22:41,688 INFO jvm.JvmMetrics (JvmMetrics.java:init(67)) - > > Initializing JVM Metrics with processName=JobTracker, sessionId= > > 2009-04-08 01:22:41,704 WARN mapred.JobClient > > (JobClient.java:configureCommandLineOptions(547)) - Use > > GenericOptionsParser > > for parsing the arguments. Applications should implement Tool for the > same. > > 2009-04-08 01:22:41,756 WARN mapred.JobClient > > (JobClient.java:configureCommandLineOptions(697)) - No job jar file set. > > User classes may not be found. See JobConf(Class) or > > JobConf#setJar(String). > > 2009-04-08 01:22:41,794 INFO mapred.FileInputFormat > > (FileInputFormat.java:listStatus(181)) - Total input paths to process : 1 > > 2009-04-08 01:22:42,114 INFO mapred.JobClient > > (JobClient.java:runJob(1144)) > > - Running job: job_local_0001 > > 2009-04-08 01:22:42,117 INFO mapred.FileInputFormat > > (FileInputFormat.java:listStatus(181)) - Total input paths to process : 1 > > 2009-04-08 01:22:42,161 INFO mapred.MapTask (MapTask.java:run(302)) - > > numReduceTasks: 1 > > 2009-04-08 01:22:42,171 INFO mapred.MapTask (MapTask.java:<init>(493)) - > > io.sort.mb = 100 > > 2009-04-08 01:22:42,241 WARN mapred.LocalJobRunner > > (LocalJobRunner.java:run(194)) - job_local_0001 > > java.lang.OutOfMemoryError: Java heap space > > at > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:498) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > > at > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138) > > Exception in thread "main" java.io.IOException: Job failed! > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) > > at org.apache.nutch.crawl.Injector.inject(Injector.java:160) > > at org.apache.nutch.crawl.Crawl.main(Crawl.java:113) > > >
