Thanks Jaydeep for you quick response!
After adding oozie.action.sharelib.for.spark = spark, it threw
NoSuchMethodError exception.
Do you know how I can I add guava-16.0.1 into the oozie's class path?
>>> Invoking Spark class now >>>
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class
[org.apache.oozie.action.hadoop.SparkMain], main() threw exception,
com.google.common.reflect.TypeToken.isPrimitive()Z
java.lang.NoSuchMethodError: com.google.common.reflect.TypeToken.isPrimitive()Z
at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:142)
at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:136)
at
com.datastax.driver.core.TypeCodec$BlobCodec.<init>(TypeCodec.java:609)
at
com.datastax.driver.core.TypeCodec$BlobCodec.<clinit>(TypeCodec.java:606)
at
com.datastax.driver.core.CodecRegistry.<clinit>(CodecRegistry.java:147)
at
com.datastax.driver.core.Configuration$Builder.build(Configuration.java:259)
at
com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1135)
at com.datastax.driver.core.Cluster.<init>(Cluster.java:111)
at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178)
at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1152)
at
com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85)
at
com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150)
at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:150)
at
com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
at
com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
at
com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
at
com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
at
com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
at
com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:241)
at
com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263)
at
com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
at TestCassandra$.main(TestCassandra.scala:44)
at TestCassandra.main(TestCassandra.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
On Wed, Feb 24, 2016 at 5:00 AM, Jaydeep Vishwakarma <
jaydeep.vishwaka...@inmobi.com> wrote:
> Add spark sharelib in oozie setup it will work.
>
> Regards,
> Jaydep
>
> On Wed, Feb 24, 2016 at 3:13 PM, Liping Zhang <zlpmiche...@gmail.com>
> wrote:
>
> > Dear oozie user and dev,
> >
> > I set following values in job.properties and in action properties in CDH
> > 5.5.0 Hue oozie:
> > oozie.use.system.libpath=true
> >
> >
> oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935
> >
> >
> oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark
> >
> > *Here is the workflow file:*
> > <workflow-app name="sparktest-cassandra" xmlns="uri:oozie:workflow:0.5">
> > <start to="spark-b23b"/>
> > <kill name="Kill">
> > <message>Action failed, error
> > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > </kill>
> > <action name="spark-b23b">
> > <spark xmlns="uri:oozie:spark-action:0.1">
> > <job-tracker>${jobTracker}</job-tracker>
> > <name-node>${nameNode}</name-node>
> > <configuration>
> > <property>
> > <name>spark.executor.extraClassPath</name>
> > <value>lib/guava-16.0.1.jar</value>
> > </property>
> > <property>
> > <name>oozie.action.sharelib.for.spark</name>
> >
> > <value>/user/oozie/share/lib/lib_20151201085935/spark</value>
> > </property>
> > <property>
> > <name>oozie.use.system.libpath</name>
> > <value>true</value>
> > </property>
> > <property>
> > <name>oozie.libpath</name>
> >
> <value>/user/oozie/share/lib/lib_20151201085935</value>
> > </property>
> > </configuration>
> > <master>local[4]</master>
> > <mode>client</mode>
> > <name>sparktest-cassandra</name>
> > <class>TestCassandra</class>
> > <jar>lib/sparktest.jar</jar>
> > <spark-opts>--driver-class-path
> > /opt/cloudera/parcels/CDH/jars/guava-16.0.1.jar</spark-opts>
> > <arg>s3n://gridx-output/sparktest/ </arg>
> > <arg>10</arg>
> > <arg>3</arg>
> > <arg>2</arg>
> > </spark>
> > <ok to="End"/>
> > <error to="Kill"/>
> > </action>
> > <end name="End"/>
> > </workflow-app>
> >
> > *job.properties file:*
> > oozie.use.system.libpath=true
> > security_enabled=False
> >
> >
> oozie.libpath=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935
> >
> >
> oozie.action.sharelib.for.spark=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/oozie/share/lib/lib_20151201085935/spark
> > dryrun=False
> > jobTracker=ip-10-0-4-248.us-west-1.compute.internal:8032
> > nameNode=hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020
> >
> >
> > But I got following SparkMain class not found issue:
> >
> > <<< Invocation of Main class completed <<<
> >
> > Failing Oozie Launcher, Main class
> > [org.apache.oozie.action.hadoop.SparkMain], exception invoking main(),
> > java.lang.ClassNotFoundException: Class
> > org.apache.oozie.action.hadoop.SparkMain not found
> > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> > org.apache.oozie.action.hadoop.SparkMain not found
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
> > at
> >
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:234)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> > at
> >
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:378)
> > at
> >
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:296)
> > at
> >
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
> > at
> >
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.ClassNotFoundException: Class
> > org.apache.oozie.action.hadoop.SparkMain not found
> > at
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
> > ... 13 more
> >
> > Oozie Launcher failed, finishing Hadoop job gracefully
> >
> > Oozie Launcher, uploading action data to HDFS sequence file:
> >
> >
> hdfs://ip-10-0-4-248.us-west-1.compute.internal:8020/user/admin/oozie-oozi/0000006-160224085347053-oozie-oozi-W/spark-b23b--spark/action-data.seq
> > <
> >
> http://ip-10-0-4-248.us-west-1.compute.internal:8888/filebrowser/view=/user/admin/oozie-oozi/0000006-160224085347053-oozie-oozi-W/spark-b23b--spark/action-data.seq
> > >
> >
> > Oozie Launcher ends
> >
> > Can you help to give any suggestion? Thanks a lot!
> >
> > --
> > Cheers,
> > -----
> > Big Data - Big Wisdom - Big Value
> > --------------
> > Michelle Zhang (张莉苹)
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
Cheers,
-----
Big Data - Big Wisdom - Big Value
--------------
Michelle Zhang (张莉苹)
