One more thing, finally it terminates there (after some time) by giving the final Exception:
java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1217) at LbjTagger.NerTagger.main(NerTagger.java:109) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:165) at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) akhil1988 wrote: > > Thank you Jason for your reply. > > My Map class is an inner class and it is a static class. Here is the > structure of my code. > > public class NerTagger { > > public static class Map extends MapReduceBase implements > Mapper<LongWritable, Text, Text, Text>{ > private Text word = new Text(); > private static NETaggerLevel1 tagger1 = new > NETaggerLevel1(); > private static NETaggerLevel2 tagger2 = new > NETaggerLevel2(); > > Map(){ > System.out.println("HI2\n"); > > Parameters.readConfigAndLoadExternalData("Config/allLayer1.config"); > System.out.println("HI3\n"); > > Parameters.forceNewSentenceOnLineBreaks=Boolean.parseBoolean("true"); > > System.out.println("loading the tagger"); > > tagger1=(NETaggerLevel1)Classifier.binaryRead(Parameters.pathToModelFile+".level1"); > System.out.println("HI5\n"); > > tagger2=(NETaggerLevel2)Classifier.binaryRead(Parameters.pathToModelFile+".level2"); > System.out.println("Done- loading the tagger"); > } > > public void map(LongWritable key, Text value, > OutputCollector<Text, Text> output, Reporter reporter ) throws IOException > { > String inputline = value.toString(); > > /* Processing of the input pair is done here */ > } > > > public static void main(String [] args) throws Exception { > JobConf conf = new JobConf(NerTagger.class); > conf.setJobName("NerTagger"); > > conf.setOutputKeyClass(Text.class); > conf.setOutputValueClass(IntWritable.class); > > conf.setMapperClass(Map.class); > conf.setNumReduceTasks(0); > > conf.setInputFormat(TextInputFormat.class); > conf.setOutputFormat(TextOutputFormat.class); > > conf.set("mapred.job.tracker", "local"); > conf.set("fs.default.name", "file:///"); > > DistributedCache.addCacheFile(new > URI("/home/akhil1988/Ner/OriginalNer/Data/"), conf); > DistributedCache.addCacheFile(new > URI("/home/akhil1988/Ner/OriginalNer/Config/"), conf); > DistributedCache.createSymlink(conf); > > > conf.set("mapred.child.java.opts","-Xmx4096m"); > > FileInputFormat.setInputPaths(conf, new Path(args[0])); > FileOutputFormat.setOutputPath(conf, new Path(args[1])); > > System.out.println("HI1\n"); > > JobClient.runJob(conf); > } > > Jason, when the program executes HI1 and HI2 are printed but it does not > reaches HI3. In the statement > Parameters.readConfigAndLoadExternalData("Config/allLayer1.config"); it is > able to access Config/allLayer1.config file (as while executing this > statement, it prints some messages like which data it is loading, etc.) > but it gets stuck there(while loading some classifier) and never reaches > HI3. > > This program runs fine when executed normally(without mapreduce). > > Thanks, Akhil > > > > > jason hadoop wrote: >> >> Is it possible that your map class is an inner class and not static? >> >> On Tue, Jun 16, 2009 at 10:51 AM, akhil1988 <akhilan...@gmail.com> wrote: >> >>> >>> Hi All, >>> >>> I am running my mapred program in local mode by setting >>> mapred.jobtracker.local to local mode so that I can debug my code. >>> The mapred program is a direct porting of my original sequential code. >>> There >>> is no reduce phase. >>> Basically, I have just put my program in the map class. >>> >>> My program takes around 1-2 min. in instantiating the data objects which >>> are >>> present in the constructor of Map class(it loads some data model files, >>> therefore it takes some time). After the instantiation part in the >>> constrcutor of Map class the map function is supposed to process the >>> input >>> split. >>> >>> The problem is that the data objects do not get instantiated completely >>> and >>> in between(whlie it is still in constructor) the program stops giving >>> the >>> exceptions pasted at bottom. >>> The program runs fine without mapreduce and does not require more than >>> 2GB >>> memory, but in mapreduce even after doing export HADOOP_HEAPSIZE=2500(I >>> am >>> working on machines with 16GB RAM), the program fails. I have also set >>> HADOOP_OPTS="-server -XX:-UseGCOverheadLimit" as sometimes I was getting >>> GC >>> Overhead Limit Exceeded exceptions also. >>> >>> Somebody, please help me with this problem: I have trying to debug it >>> for >>> the last 3 days, but unsuccessful. Thanks! >>> >>> java.lang.OutOfMemoryError: Java heap space >>> at >>> sun.misc.FloatingDecimal.toJavaFormatString(FloatingDecimal.java:889) >>> at java.lang.Double.toString(Double.java:179) >>> at java.text.DigitList.set(DigitList.java:272) >>> at java.text.DecimalFormat.format(DecimalFormat.java:584) >>> at java.text.DecimalFormat.format(DecimalFormat.java:507) >>> at java.text.NumberFormat.format(NumberFormat.java:269) >>> at >>> org.apache.hadoop.util.StringUtils.formatPercent(StringUtils.java:110) >>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1147) >>> at LbjTagger.NerTagger.main(NerTagger.java:109) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:165) >>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) >>> >>> 09/06/16 12:34:41 WARN mapred.LocalJobRunner: job_local_0001 >>> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81) >>> at >>> org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) >>> at >>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328) >>> at >>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138) >>> Caused by: java.lang.reflect.InvocationTargetException >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at >>> >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) >>> at >>> >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >>> at >>> java.lang.reflect.Constructor.newInstance(Constructor.java:513) >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:79) >>> ... 5 more >>> Caused by: java.lang.ThreadDeath >>> at java.lang.Thread.stop(Thread.java:715) >>> at >>> org.apache.hadoop.mapred.LocalJobRunner.killJob(LocalJobRunner.java:310) >>> at >>> org.apache.hadoop.mapred.JobClient$NetworkedJob.killJob(JobClient.java:315) >>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1224) >>> at LbjTagger.NerTagger.main(NerTagger.java:109) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:165) >>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Nor-%22OOM-Java-Heap-Space%22-neither-%22GC-OverHead-Limit-Exeeceded%22-tp24059508p24059508.html >>> Sent from the Hadoop core-user mailing list archive at Nabble.com. >>> >>> >> >> >> -- >> Pro Hadoop, a book to guide you from beginner to hadoop mastery, >> http://www.amazon.com/dp/1430219424?tag=jewlerymall >> www.prohadoopbook.com a community for Hadoop Professionals >> >> > > -- View this message in context: http://www.nabble.com/Nor-%22OOM-Java-Heap-Space%22-neither-%22GC-OverHead-Limit-Exeeceded%22-tp24059508p24066426.html Sent from the Hadoop core-user mailing list archive at Nabble.com.