Got past the log4j problem. Made a bash script that sets the
HADOOP_CLASSPATH and runs hadoop jar from within.
Next problem, the job is getting submitted as 'mapred' user, for some
reason, instead of the user that submitted the oozie job. Fails when
creating the staging dir..
Exception in thread "main" java.lang.RuntimeException: java.io.IOException:
The ownership on the staging directory
hdfs://{namenode}:8020/user/wand/.staging is not as expected. It is owned
by wand. The directory must be owned by the submitter mapred or by mapred
at com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by:
java.io.IOException: The ownership on the staging directory hdfs://
master22.stg.kontagent.com:8020/user/wand/.staging is not as expected. It
is owned by wand. The directory must be owned by the submitter mapred or by
mapred at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:982) at
org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976)
On Thu, May 14, 2015 at 11:16 PM, Mohammad Islam <misla...@yahoo.com.invalid
> wrote:
> This is sometime tricky and needs careful debugging.I think it could be
> related to different versions of same jar. or conflicting with other jar.
> you can try to set "ooze.launcher.mapreduce.job.classloader" in shell
> action's configuration.
> Please let us know how it goes. One important observation is : it is not
> ClassNotFoundException rather NoClassDefFoundError.
> Regards,Mohammad
>
>
>
> On Thursday, May 14, 2015 8:53 PM, Gautam <gautamkows...@gmail.com>
> wrote:
>
>
> Hello,
> I'm trying to submit a hadoop jar job via Oozie workflow. In
> particular, trying to run it using the shell action by invoking the
> /usr/bin/hadoop jar <jar_file> <MRClass>..
>
>
> I tested this job separately from the command line and it works when I just
> submit using "hadoop jar <jar_file> <class_name> " . But the same job fails
> with the following error from within Oozie flow step with this ..
>
> Exception in thread "main" java.lang.NoClassDefFoundError: Could not
> initialize class org.apache.log4j.LogManager
> at org.apache.log4j.Logger.getLogger(Logger.java:117)
> at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:128)
> at com.linkedin.camus.etl.kafka.CamusJob.<init>(CamusJob.java:124)
> at
> com.upsight.valhalla.kafka.dumper.KafkaDumper.main(KafkaDumper.java:19)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>
>
> I'v made sure there's a log4j.xml and the LogManager impl in my uber jar.
> I'd imagine others have tried submitting a MR job with "hadoop jar "
> runtime. Although, I'm also aware of the <map-reduce> action but it's too
> cumbersome for me since i set a lot of params, counters in my job client
> class (which implements the Tool abstract class).
>
>
>
> Here's my workflow step config:
>
> <action name="phShimDump">
> <shell xmlns="uri:oozie:shell-action:0.3">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <exec>/usr/bin/hadoop</exec>
> <argument>jar</argument>
> <argument>kafka-hdfs-etl/jar/kafka-hdfs-etl.jar</argument>
>
> <argument>com.upsight.valhalla.kafka.dumper.KafkaDumper</argument>
>
> <!--<argument>kafka-hdfs-etl/script/kafka-etl-console</argument>-->
> <argument>-P</argument>
>
> <argument>kafka-hdfs-etl/script/oozie/${phShimDumpConfig}</argument>
> <argument>-D</argument>
> <argument>camus.job.name=ph_up_shim_data</argument>
> <argument>-D</argument>
> <argument>zookeeper.hosts=${zookeeperHosts}</argument>
> <argument>-D</argument>
> <argument>kafka.brokers=${kafkaBrokers}</argument>
>
> <archive>/opt/kafka-hdfs-etl/kafka-hdfs-etl.tar.gz#kafka-hdfs-etl</archive>
> <capture-output/>
> </shell>
> <ok to="end" />
> <error to="kill" />
> </action>
>
>
> thanks and regards,
> -Gautam.
>
>
>
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
