HI Vinod, Thanks for your support. I¹m packaging my application in Eclipse (Kepler) as a runnable jar. I can get the application to debug there in local mode.
The command line I am using for my app is: "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10² the jar permissions are -"rwxr-xr-x" The three parameters on the end of the command line are arbitrary commands from my sequential program which I will use as input when I can get this to run. I have downloaded the source and run a 2.4 version of WordCount, it¹s not getting past the map stage either. I¹m pasting in the whole job output. Deleted demo_output /usr/local/hadoop-2.4.0/jar_files $ hadoop jar /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10 Start time = 1404044126713 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process : 0 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1403953980618_0009 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources. 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application application_1403953980618_0009 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job: http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/ 14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running in uber mode : false 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0% 14/06/29 13:17:23 INFO mapreduce.Job: Task Id : attempt_1403953980618_0009_r_000000_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm pl.java:210) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja va:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 14/06/29 13:17:28 INFO mapreduce.Job: Task Id : attempt_1403953980618_0009_r_000000_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm pl.java:210) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja va:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 14/06/29 13:17:32 INFO mapreduce.Job: Task Id : attempt_1403953980618_0009_r_000000_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm pl.java:210) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja va:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100% 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed with state FAILED due to: Task failed task_1403953980618_0009_r_000000 Job failed as tasks failed. failedMaps:0 failedReduces:1 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7 Job Counters Failed reduce tasks=4 Launched reduce tasks=4 Total time spent by all maps in occupied slots (ms)=0 Total time spent by all reduces in occupied slots (ms)=7786 Total time spent by all reduce tasks (ms)=7786 Total vcore-seconds taken by all reduce tasks=7786 Total megabyte-seconds taken by all reduce tasks=7972864 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server: admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS) 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server: admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS) 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server: admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS) 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history server End time = 1404044288031 Elapsed time = 161318 ms Finished /usr/local/hadoop-2.4.0/jar_files $ Warm regards. Chris From: Vinod Kumar Vavilapalli <vino...@hortonworks.com> Reply-To: <user@hadoop.apache.org> Date: Sunday, 29 June 2014 00:20 To: <user@hadoop.apache.org> Subject: Re: Partitioning and setup errors What is happening is the client is not able to pick up the right jar to push to the cluster. It looks in the class-path for the jar that contains the class ParallelGeneticAlignment. How are you packaging your code? How are your running your job - paste the command line? +Vinod On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <stu...@chrismackenziephotography.co.uk> wrote: > Hi, > > I realise my previous question may have been a bit naïve and I also realise I > am asking an awful lot here, any advice would be greatly appreciated. > * I have been using Hadoop 2.4 in local mode and am sticking to the > mapreduce.* side of the track. > * I am using a Custom Line reader to read each sequence into a Map > * I have a partitioner class which is testing the key from the map class. > * I've tried debugging in eclipse with a breakpoint in the partitioner class > but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks) > is not being called. > Could there be any reason for that ? > > Because my map and reduce code works in local mode within eclipse, I wondered > if I may get the partitioner to work if I changed to Pseudo Distributed Mode > exporting a runnable jar from Eclipse (Kepler) > > I have several faults On my own computer Pseudo Distributed Mode and the > university clusters Pseudo Distributed Mode which I set up. I¹ve googled and > read extensively but am not seeing a solution to any of these issues. > > I have this line: > 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User > classes may not be found. See Job or Job#setJar(String). > My driver code is: > private void doParallelConcordance() throws Exception { > > Path inDir = new Path("input_sequences/10_sequences.txt"); > Path outDir = new Path("demo_output"); > > Job job = Job.getInstance(new Configuration()); > job.setJarByClass(ParallelGeneticAlignment.class); > job.setOutputKeyClass(Text.class); > job.setOutputValueClass(IntWritable.class); > > job.setInputFormatClass(CustomFileInputFormat.class); > job.setMapperClass(ConcordanceMapper.class); > job.setPartitionerClass(ConcordanceSequencePartitioner.class); > job.setReducerClass(ConcordanceReducer.class); > > FileInputFormat.addInputPath(job, inDir); > FileOutputFormat.setOutputPath(job, outDir); > > > job.waitForCompletion(true) > } > > On the university server I am getting this error: > 4/06/27 11:45:40 INFO mapreduce.Job: Task Id : > attempt_1403860966764_0003_m_000000_0, Status : FAILED > Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > par.gene.align.concordance.ConcordanceMapper not found > > On my machine the error is: > 4/06/27 12:58:03 INFO mapreduce.Job: Task Id : > attempt_1403864060032_0004_r_000000_2, Status : FAILED > Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > par.gene.align.concordance.ConcordanceReducer not found > > On the university server I get total paths to process: > 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1 > 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1 > > On my machine I get total paths to process: > 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0 > 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0 > > Being new to this community, I thought it polite to introduce myself. I¹m > planning to return to software development via an MSc at Heriot Watt > University in Edinburgh. My MSc project is based on Fosters Genetic Sequence > Alignment. I have written a sequential version my goal is now to port it to > Hadoop. > > Thanks in advance, > Regards, > > Chris MacKenzie CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
