Thanks so much Shixiong! This is great. On Tue, Jun 2, 2015 at 8:26 PM Shixiong Zhu <zsxw...@gmail.com> wrote:
> Ryan - I sent a PR to fix your issue: > https://github.com/apache/spark/pull/6599 > > Edward - I have no idea why the following error happened. "ContextCleaner" > doesn't use any Hadoop API. Could you try Spark 1.3.0? It's supposed to > support both hadoop 1 and hadoop 2. > > > * "Exception in thread "Spark Context Cleaner" > java.lang.NoClassDefFoundError: 0 > at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:149)" > > > Best Regards, > Shixiong Zhu > > 2015-06-03 0:08 GMT+08:00 Ryan Williams <ryan.blake.willi...@gmail.com>: > >> I think this is causing issues upgrading ADAM >> <https://github.com/bigdatagenomics/adam> to Spark 1.3.1 (cf. adam#690 >> <https://github.com/bigdatagenomics/adam/pull/690#issuecomment-107769383>); >> attempting to build against Hadoop 1.0.4 yields errors like: >> >> 2015-06-02 15:57:44 ERROR Executor:96 - Exception in task 0.0 in stage >> 0.0 (TID 0) >> *java.lang.IncompatibleClassChangeError: Found class >> org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected* >> at >> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:95) >> at org.apache.spark.SparkHadoopWriter.commit(SparkHadoopWriter.scala:106) >> at >> org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1082) >> at >> org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >> at org.apache.spark.scheduler.Task.run(Task.scala:64) >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> 2015-06-02 15:57:44 WARN TaskSetManager:71 - Lost task 0.0 in stage 0.0 >> (TID 0, localhost): java.lang.IncompatibleClassChangeError: Found class >> org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected >> at >> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:95) >> at org.apache.spark.SparkHadoopWriter.commit(SparkHadoopWriter.scala:106) >> at >> org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1082) >> at >> org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >> at org.apache.spark.scheduler.Task.run(Task.scala:64) >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> TaskAttemptContext is a class in Hadoop 1.0.4, but an interface in Hadoop >> 2; Spark 1.3.1 expects the interface but is getting the class. >> >> It sounds like, while I *can* depend on Spark 1.3.1 and Hadoop 1.0.4, I >> then need to hope that I don't exercise certain Spark code paths that run >> afoul of differences between Hadoop 1 and 2; does that seem correct? >> >> Thanks! >> >> On Wed, May 20, 2015 at 1:52 PM Sean Owen <so...@cloudera.com> wrote: >> >>> I don't think any of those problems are related to Hadoop. Have you >>> looked at userClassPathFirst settings? >>> >>> On Wed, May 20, 2015 at 6:46 PM, Edward Sargisson <ejsa...@gmail.com> >>> wrote: >>> >>>> Hi Sean and Ted, >>>> Thanks for your replies. >>>> >>>> I don't have our current problems nicely written up as good questions >>>> yet. I'm still sorting out classpath issues, etc. >>>> In case it is of help, I'm seeing: >>>> * "Exception in thread "Spark Context Cleaner" >>>> java.lang.NoClassDefFoundError: 0 >>>> at >>>> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:149)" >>>> * We've been having clashing dependencies between a colleague and I >>>> because of the aforementioned classpath issue >>>> * The clashing dependencies are also causing issues with what jetty >>>> libraries are available in the classloader from Spark and don't clash with >>>> existing libraries we have. >>>> >>>> More anon, >>>> >>>> Cheers, >>>> Edward >>>> >>>> >>>> >>>> -------- Original Message -------- >>>> Subject: Re: spark 1.3.1 jars in repo1.maven.org Date: 2015-05-20 >>>> 00:38 From: Sean Owen <so...@cloudera.com> To: Edward Sargisson < >>>> esa...@pobox.com> Cc: user <user@spark.apache.org> >>>> >>>> >>>> Yes, the published artifacts can only refer to one version of anything >>>> (OK, modulo publishing a large number of variants under classifiers). >>>> >>>> You aren't intended to rely on Spark's transitive dependencies for >>>> anything. Compiling against the Spark API has no relation to what >>>> version of Hadoop it binds against because it's not part of any API. >>>> You mark the Spark dependency even as "provided" in your build and get >>>> all the Spark/Hadoop bindings at runtime from our cluster. >>>> >>>> What problem are you experiencing? >>>> >>>> >>>> On Wed, May 20, 2015 at 2:17 AM, Edward Sargisson <esa...@pobox.com> >>>> wrote: >>>> >>>> Hi, >>>> I'd like to confirm an observation I've just made. Specifically that >>>> spark >>>> is only available in repo1.maven.org for one Hadoop variant. >>>> >>>> The Spark source can be compiled against a number of different Hadoops >>>> using >>>> profiles. Yay. >>>> However, the spark jars in repo1.maven.org appear to be compiled >>>> against one >>>> specific Hadoop and no other differentiation is made. (I can see a >>>> difference with hadoop-client being 2.2.0 in repo1.maven.org and 1.0.4 >>>> in >>>> the version I compiled locally). >>>> >>>> The implication here is that if you have a pom file asking for >>>> spark-core_2.10 version 1.3.1 then Maven will only give you an Hadoop 2 >>>> version. Maven assumes that non-snapshot artifacts never change so >>>> trying to >>>> load an Hadoop 1 version will end in tears. >>>> >>>> This then means that if you compile code against spark-core then there >>>> will >>>> probably be classpath NoClassDefFound issues unless the Hadoop 2 >>>> version is >>>> exactly the one you want. >>>> >>>> Have I gotten this correct? >>>> >>>> It happens that our little app is using a Spark context directly from a >>>> Jetty webapp and the classpath differences were/are causing some >>>> confusion. >>>> We are currently installing a Hadoop 1 spark master and worker. >>>> >>>> Thanks a lot! >>>> Edward >>>> >>>> >>>> >>>> >>> >
