Re: spark-assembly libraries conflict with needed libraries
Thanks - that did solve my error, but instead got a different one: java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormat It seems like with that setting, spark can't find Hadoop. On 7/7/14, Koert Kuipers wrote: > spark has a setting to put user jars in front of classpath, which should do > the trick. > however i had no luck with this. see here: > > https://issues.apache.org/jira/browse/SPARK-1863 > > > > On Mon, Jul 7, 2014 at 1:31 PM, Robert James > wrote: > >> spark-submit includes a spark-assembly uber jar, which has older >> versions of many common libraries. These conflict with some of the >> dependencies we need. I have been racking my brain trying to find a >> solution (including experimenting with ProGuard), but haven't been >> able to: when we use spark-submit, we get NoMethodErrors, even though >> the code compiles fine, because the runtime classes are different than >> the compile time classes! >> >> Can someone recommend a solution? We are using scala, sbt, and >> sbt-assembly, but are happy using another tool (please provide >> instructions how to). >> >
Re: spark-assembly libraries conflict with needed libraries
I don't have experience deploying to EC2. can you use add.jar conf to add the missing jar at runtime ? I haven't tried this myself. Just a guess. On Mon, Jul 7, 2014 at 12:16 PM, Chester Chen wrote: > with "provided" scope, you need to provide the "provided" jars at the > runtime yourself. I guess in this case Hadoop jar files. > > > On Mon, Jul 7, 2014 at 12:13 PM, Robert James > wrote: > >> Thanks - that did solve my error, but instead got a different one: >> java.lang.NoClassDefFoundError: >> org/apache/hadoop/mapreduce/lib/input/FileInputFormat >> >> It seems like with that setting, spark can't find Hadoop. >> >> On 7/7/14, Koert Kuipers wrote: >> > spark has a setting to put user jars in front of classpath, which >> should do >> > the trick. >> > however i had no luck with this. see here: >> > >> > https://issues.apache.org/jira/browse/SPARK-1863 >> > >> > >> > >> > On Mon, Jul 7, 2014 at 1:31 PM, Robert James >> > wrote: >> > >> >> spark-submit includes a spark-assembly uber jar, which has older >> >> versions of many common libraries. These conflict with some of the >> >> dependencies we need. I have been racking my brain trying to find a >> >> solution (including experimenting with ProGuard), but haven't been >> >> able to: when we use spark-submit, we get NoMethodErrors, even though >> >> the code compiles fine, because the runtime classes are different than >> >> the compile time classes! >> >> >> >> Can someone recommend a solution? We are using scala, sbt, and >> >> sbt-assembly, but are happy using another tool (please provide >> >> instructions how to). >> >> >> > >> > >
Re: spark-assembly libraries conflict with needed libraries
with "provided" scope, you need to provide the "provided" jars at the runtime yourself. I guess in this case Hadoop jar files. On Mon, Jul 7, 2014 at 12:13 PM, Robert James wrote: > Thanks - that did solve my error, but instead got a different one: > java.lang.NoClassDefFoundError: > org/apache/hadoop/mapreduce/lib/input/FileInputFormat > > It seems like with that setting, spark can't find Hadoop. > > On 7/7/14, Koert Kuipers wrote: > > spark has a setting to put user jars in front of classpath, which should > do > > the trick. > > however i had no luck with this. see here: > > > > https://issues.apache.org/jira/browse/SPARK-1863 > > > > > > > > On Mon, Jul 7, 2014 at 1:31 PM, Robert James > > wrote: > > > >> spark-submit includes a spark-assembly uber jar, which has older > >> versions of many common libraries. These conflict with some of the > >> dependencies we need. I have been racking my brain trying to find a > >> solution (including experimenting with ProGuard), but haven't been > >> able to: when we use spark-submit, we get NoMethodErrors, even though > >> the code compiles fine, because the runtime classes are different than > >> the compile time classes! > >> > >> Can someone recommend a solution? We are using scala, sbt, and > >> sbt-assembly, but are happy using another tool (please provide > >> instructions how to). > >> > > >
Re: spark-assembly libraries conflict with needed libraries
Chester - I'm happy rebuilding Spark, but then how can I deploy it to EC2? On 7/7/14, Chester Chen wrote: > Have you tried to change the spark SBT scripts? You can change the > dependency scope to "provided". This similar to compile scope, except JDK > or container need to provide the dependency at runtime. > > This assume the Spark will work with the new version of common libraries. > > Of course, this is not a general solution even it works ( if may not work). > > Chester > > > > > On Mon, Jul 7, 2014 at 10:31 AM, Robert James > wrote: > >> spark-submit includes a spark-assembly uber jar, which has older >> versions of many common libraries. These conflict with some of the >> dependencies we need. I have been racking my brain trying to find a >> solution (including experimenting with ProGuard), but haven't been >> able to: when we use spark-submit, we get NoMethodErrors, even though >> the code compiles fine, because the runtime classes are different than >> the compile time classes! >> >> Can someone recommend a solution? We are using scala, sbt, and >> sbt-assembly, but are happy using another tool (please provide >> instructions how to). >> >
Re: spark-assembly libraries conflict with needed libraries
Have you tried to change the spark SBT scripts? You can change the dependency scope to "provided". This similar to compile scope, except JDK or container need to provide the dependency at runtime. This assume the Spark will work with the new version of common libraries. Of course, this is not a general solution even it works ( if may not work). Chester On Mon, Jul 7, 2014 at 10:31 AM, Robert James wrote: > spark-submit includes a spark-assembly uber jar, which has older > versions of many common libraries. These conflict with some of the > dependencies we need. I have been racking my brain trying to find a > solution (including experimenting with ProGuard), but haven't been > able to: when we use spark-submit, we get NoMethodErrors, even though > the code compiles fine, because the runtime classes are different than > the compile time classes! > > Can someone recommend a solution? We are using scala, sbt, and > sbt-assembly, but are happy using another tool (please provide > instructions how to). >
Re: spark-assembly libraries conflict with needed libraries
spark has a setting to put user jars in front of classpath, which should do the trick. however i had no luck with this. see here: https://issues.apache.org/jira/browse/SPARK-1863 On Mon, Jul 7, 2014 at 1:31 PM, Robert James wrote: > spark-submit includes a spark-assembly uber jar, which has older > versions of many common libraries. These conflict with some of the > dependencies we need. I have been racking my brain trying to find a > solution (including experimenting with ProGuard), but haven't been > able to: when we use spark-submit, we get NoMethodErrors, even though > the code compiles fine, because the runtime classes are different than > the compile time classes! > > Can someone recommend a solution? We are using scala, sbt, and > sbt-assembly, but are happy using another tool (please provide > instructions how to). >
