Ben's right, for livy interpreter, you need to specify livy.spark.jars.packages.
Specifying interpreter dependency doesn't work for livy interpreter, because livy run in yarn-cluster mode, while interpreter dependencies is downloaded in the client JVM. Felix Cheung <felixcheun...@hotmail.com>于2017年6月1日周四 上午2:52写道： > How are you setting this: > When setting my local maven repo and the "groupId:artifactId:version" in > the interpreter settings > > ------------------------------ > *From:* Theofilos Kakantousis <t...@kth.se> > *Sent:* Wednesday, May 31, 2017 1:56:07 AM > *To:* users@zeppelin.apache.org > > *Subject:* Re: Livy - add external libraries from additional maven repo > Thanks everyone for the feedback! > > Indeed %dep works only for Spark interpreter, just mentioned it to show > the interpreter behavior I expected with Livy. > When setting my local maven repo and the "groupId:artifactId:version" in > the interpreter settings, I can see the dependency (i.e. a jar file) being > downloaded to the local OS under "local-repo" directory but the dependency > is not deployed with the Spark application in YARN. > > Cheers, > Theo > > On 2017-05-31 01:19, Ben Vogan wrote: > > For what it's worth I have successfully added jar files and maven packages > to sessions using zeppelin & livy 0.3 - although not using %dep. In the > interpreter settings I set the livy.spark.jars setting for jars that are on > my HDFS cluster, and livy.spark.jars.packages for maven packages - > although only using maven central and not a local repo. > > --Ben > > On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> To add, this might be an issue with Livy. >> >> I'm seeing something similar as well. >> >> If you can get a repo with calling the Livy REST API directly it will be >> worthwhile to follow up with the Livy community separately. >> >> >> ------------------------------ >> *From:* Felix Cheung <felixcheun...@hotmail.com> >> *Sent:* Tuesday, May 30, 2017 11:34:31 AM >> *To:* users@zeppelin.apache.org; users@zeppelin.apache.org >> *Subject:* Re: Livy - add external libraries from additional maven repo >> >> if I recall, %dep only works with the built in Spark interpreter and not >> the Livy interpreter. >> >> To manage dependency win Livy you will need to set Spark conf with Livy. >> >> ------------------------------ >> *From:* Theofilos Kakantousis <t...@kth.se> >> *Sent:* Tuesday, May 30, 2017 9:05:15 AM >> *To:* users@zeppelin.apache.org >> *Subject:* Livy - add external libraries from additional maven repo >> >> Hi everyone, >> >> I'm using Zeppelin with Livy 0.4 and trying to add external libraries >> from an additional maven repo to my application according to the >> documentation available here >> <https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>. >> The example works fine, but when I set the livy.spark.jars.packages to my >> library the interpreter throws an unresolved dependency error. >> >> I have added the additional maven repository in the interpreter settings >> and have also tried setting livy.spark.jars.ivy but without luck. However, >> if I use the Spark interpreter with the following code it works fine. >> >> "%dep >> z.reset(); >> z.addRepo("my repo").url("http://myrepo"; <http://myrepo>).snapshot >> z.load("mygroup:myartifact:myversion"); >> >> Has anyone managed to do that with Livy? Thanks! >> >> Cheers, >> Theo >> > > > > -- > *BENJAMIN VOGAN* | Data Platform Team Lead > > <http://www.shopkick.com/> > <https://www.facebook.com/shopkick> <https://www.instagram.com/shopkick/> > <https://www.pinterest.com/shopkick/> <https://twitter.com/shopkickbiz> > <https://www.linkedin.com/company-beta/831240/?pathWildcard=831240> > > >
