Thanks. But I think I already mark all the Spark and Hadoop reps as provided.
Why the cluster's version is not used?
Any way, as I mentioned in the previous message, after changing the
hadoop-client to version 1.2.1 in my maven deps, I already pass the exception
and go to another one as indicated below. Any suggestion on this?
=================================Exception in thread "main"
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
at
org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at
org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
at
org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
at
org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala) ... 6
more
> From: so...@cloudera.com
> Date: Fri, 23 Jan 2015 10:41:12 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eyc...@hotmail.com
> CC: user@spark.apache.org
>
> So, you should not depend on Hadoop artifacts unless you use them
> directly. You should mark Hadoop and Spark deps as provided. Then the
> cluster's version is used at runtime with spark-submit. That's the
> usual way to do it, which works.
>
> If you need to embed Spark in your app and are running it outside the
> cluster for some reason, and you have to embed Hadoop and Spark code
> in your app, the version has to match. You should also use mvn
> dependency:tree to see all the dependencies coming in. There may be
> many sources of a Hadoop dep.
>
> On Fri, Jan 23, 2015 at 1:05 AM, ey-chih chow <eyc...@hotmail.com> wrote:
> > Thanks. But after I replace the maven dependence from
> >
> > <dependency>
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-client</artifactId>
> > <version>2.5.0-cdh5.2.0</version>
> > <scope>provided</scope>
> > <exclusions>
> > <exclusion>
> > <groupId>org.mortbay.jetty</groupId>
> > <artifactId>servlet-api</artifactId>
> > </exclusion>
> > <exclusion>
> > <groupId>javax.servlet</groupId>
> > <artifactId>servlet-api</artifactId>
> > </exclusion>
> > <exclusion>
> > <groupId>io.netty</groupId>
> > <artifactId>netty</artifactId>
> > </exclusion>
> > </exclusions>
> > </dependency>
> >
> > to
> >
> > <dependency>
> >
> > <groupId>org.apache.hadoop</groupId>
> >
> > <artifactId>hadoop-client</artifactId>
> >
> > <version>1.0.4</version>
> >
> > <scope>provided</scope>
> >
> > <exclusions>
> >
> > <exclusion>
> >
> > <groupId>org.mortbay.jetty</groupId>
> >
> > <artifactId>servlet-api</artifactId>
> >
> > </exclusion>
> >
> > <exclusion>
> >
> > <groupId>javax.servlet</groupId>
> >
> > <artifactId>servlet-api</artifactId>
> >
> > </exclusion>
> >
> > <exclusion>
> >
> > <groupId>io.netty</groupId>
> >
> > <artifactId>netty</artifactId>
> >
> > </exclusion>
> >
> > </exclusions>
> >
> > </dependency>
> >
> >
> > the warning message is still shown up in the namenode log. Is there any
> > other thing I need to do?
> >
> >
> > Thanks.
> >
> >
> > Ey-Chih Chow
> >
> >
