seattle hadoop announce: meeting April 21st 2011 @ 7:15 pm, Avro Serialization Super Special!

Room 209! Hello Fellow Hadoopists, We are meeting at 7:15 PM April 21st at the University Heights Community Center 5031 University Way NE Seattle WA 98105 Room #209 Seattle Hadoop Distributing Computing User Meeting == Bringing Hadoopists Tog

java.lang.NullPointerException at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:1089)

Hi All, I am trying to run a map reduce job and it is running perfectly from cmd using following command hadoop jar Processor.jar arg1 arg2 but when i schedule the same job in oozie, it is giving me following exception. Apr 20, 2011 12:49:34 PM NightProcessor.ProcessorDriver main SEVERE: null

RE: AW: AW: Out-of-band writing from mapper

Hi, I tried to find the class you said and it is only available in 0.21.0 version which is not suitable for me because it doesn't work with HBase. However I manager to make it work creating MyMultipleOutputs.class which doesn't need a reporter. The deprecated classes still work so it works now.

AW: AW: Out-of-band writing from mapper

Gah, that sucks. I'm using 0.20.1-169 from Cloudera CDH2 and assumed it would be there in 0.20.2 as well. Sorry, I have no idea what happened to MultipleOutputs in 0.20.2. Regards, Christoph -Ursprüngliche Nachricht- Von: Panayotis Antonopoulos [mailto:antonopoulos...@hotmail.com] Ges

RE: AW: Out-of-band writing from mapper

Hi, You are right I am using the MultipleOutputs from the org.apache.hadoop.mapred.lib package. But there is no MultipleOutputs in the package you said. Can you please tell me which version of hadoop you are using? I use the stable version 0.20.2. Regards, Panagiotis > From: christoph.schm..

AW: Out-of-band writing from mapper

Hi, you are probably using the wrong MultipleOutputs from the org.apache.hadoop.mapred.lib package. There is another one in org.apache.hadoop.mapreduce.lib.output, which fits into the new 0.20 API. Regards, Christoph -Ursprüngliche Nachricht- Von: Panayotis Antonopoulos [mailto:antonop

RE: Out-of-band writing from mapper

I am trying to use the MultipleOutputs class using hadoop 0.20.2 and I have the following problems: 1) The JobConf and org.apache.hadoop.mapred.TextOutputFormat classes which are needed to call "MultipleOutputs.addNamedOutput( (JobConf) configuration, "text", TextOutputFormat.class, LongWritabl

Re: Out-of-band writing from mapper

Hello again, On Wed, Apr 20, 2011 at 3:00 PM, Christoph Schmitz wrote: > PS. As far as I understand, MultipleOutputs would be used in the reducer, > right? (Which I wanted to avoid for the bulk of my data.) It isn't restricted to reducer phases (and can be used in both as well -- 'Tasks' are ver

Re: Out-of-band writing from mapper

Hello Harsh, thanks for your help! I've found that this works for me in my Mapper's setup() method: --- FileSystem fs = FileSystem.get(context.getConfiguration()); // get the attempt directory Path outputDir = FileOutputFormat.getWorkOutputPath(context); Path outputPath = new Path(o

Re: Out-of-band writing from mapper

Hello Christoph, On Wed, Apr 20, 2011 at 2:12 PM, Christoph Schmitz wrote: > My question is: is there any mechanism to assist me in writing to some > designated place in the HDFS from the mapper, in a way that is recognized by > the framework (i.e. dealing with aborted tasks, speculative execut

Out-of-band writing from mapper

Hi, I need to process data in a Java MR job (using 0.20.1) in a way such that the largest part of the data is manipulated in the mapper only (i.e. some simple per-record transformation without the need for sort + shuffle), and some small pieces have to be passed on to the reducer. The mapper-on