What is happening is the client is not able to pick up the right jar to push to
the cluster. It looks in the class-path for the jar that contains the class
ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the
command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
<stu...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> I am using a Custom Line reader to read each sequence into a Map
> I have a partitioner class which is testing the key from the map class.
> I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
