It does not work. I have set that property to some directory where that jar is placed and all containers see it - it still says that
Caused by: java.io.IOException: Bad Request: "requirement failed: Local path /root/.livy-sessions/1ba0a04e-9c72-4ba7-82dc-90334ffec877/pijob.jar cannot be added to user sessions." Should I set that property to /root/.livy ? That seems to be very awkward to me. On Thu, Oct 26, 2017 at 10:09 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote: > If you are going to access jars outside HDFS you must have their path > listed in livy.file.local-dir-whitelist like you found. > > > *Alex Bozarth* > Software Engineer > Spark Technology Center > ------------------------------ > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth> > > > 505 Howard Street > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States&entry=gmail&source=g> > San Francisco, CA 94105 > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States&entry=gmail&source=g> > United States > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States&entry=gmail&source=g> > > > > [image: Inactive hide details for Stefan Miklosovic ---10/26/2017 09:20:41 > AM---Ok I am getting somewhere: @RunWith(JUnit4.class)]Stefan Miklosovic > ---10/26/2017 09:20:41 AM---Ok I am getting somewhere: > @RunWith(JUnit4.class) > > From: Stefan Miklosovic <mikloso...@gmail.com> > To: user@livy.incubator.apache.org > Date: 10/26/2017 09:20 AM > Subject: Re: ClassNotFoundException on job submit > ------------------------------ > > > > Ok I am getting somewhere: > > @RunWith(JUnit4.class) > public class LivyTestCase { > > private static final int SAMPLES = 10000; > > private static final String LIVY_URI = "https://urldefense. > proofpoint.com/v2/url?u=http-3A__spark-2Dmaster-3A8998&d= > DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ- > cx37DPYDyo&m=m06yDfiPVPYT_h4awyPxVA7SKfA9V1MlTzp3mQ8REcE&s=v2g- > MD9i6gEg80s1fyGAzLe9WCOhelMfEygM1WOwNeM&e="; > > @Rule > public TemporaryFolder jarFolder = new TemporaryFolder(); > > @Test > public void testPiJob() throws Exception { > > File jarFile = jarFolder.newFile("testpijob2.jar"); > > ShrinkWrap.create(JavaArchive.class) > .addClass(PiJob.class) > .as(ZipExporter.class) > .exportTo(jarFile, true); > > LivyClient client = new LivyClientBuilder() > .setURI(new URI(LIVY_URI)) > .build(); > > System.out.println("Uploading PiJob jar"); > > client.uploadJar(jarFile).get(); > > System.out.println("PiJob jar uploaded"); > > final Double result = client.submit(new PiJob(1000)).get(); > > System.out.println(result); > } > } > > But while doing so, it gives me: > > java.util.concurrent.ExecutionException: java.io.IOException: > Bad Request: "requirement failed: Local path > /root/.livy-sessions/c7dbb697-13ed-443f-a630-bc9d9a544f6b/testpijob2.jar > cannot be added to user sessions." > > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at LivyTestCase.testPiJob(LivyTestCase.java:46) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > FrameworkMethod.java:50) > at org.junit.internal.runners.model.ReflectiveCallable.run( > ReflectiveCallable.java:12) > at org.junit.runners.model.FrameworkMethod.invokeExplosively( > FrameworkMethod.java:47) > at org.junit.internal.runners.statements.InvokeMethod. > evaluate(InvokeMethod.java:17) > at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at org.junit.runners.BlockJUnit4ClassRunner.runChild( > BlockJUnit4ClassRunner.java:78) > at org.junit.runners.BlockJUnit4ClassRunner.runChild( > BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs( > JUnit4IdeaTestRunner.java:68) > at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater. > startRunnerWithArgs(IdeaTestRunner.java:47) > at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart( > JUnitStarter.java:242) > at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) > Caused by: java.io.IOException: Bad Request: "requirement failed: > Local path /root/.livy-sessions/c7dbb697-13ed-443f-a630-bc9d9a544f6b/ > testpijob2.jar > cannot be added to user sessions." > at org.apache.livy.client.http.LivyConnection.sendRequest( > LivyConnection.java:229) > at org.apache.livy.client.http.LivyConnection.post( > LivyConnection.java:192) > at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:152) > at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:149) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > I have checked that there is nothing else but that PiJob.class in that > JAR. Nothing else at all. > > I have a feeling that the path for that jar needs to be visible to all > slaves, I do not have HDFS, I have spark slaves in Docker containers > so once I want to upload it, I see it in spark master > /root/.livy-sessions, but it is not in containers ... > > Could not this be helpful? > > # List of local directories from where files are allowed to be added > to user sessions. By > # default it's empty, meaning users can only reference remote URIs > when starting their > # sessions. > # livy.file.local-dir-whitelist = > > On Thu, Oct 26, 2017 at 5:30 PM, Stefan Miklosovic <mikloso...@gmail.com> > wrote: > > I think I have to add a jar with PiJob on the classpath of Livy so it > > knows how to deserialize it .... hmmmmm > > > > On Thu, Oct 26, 2017 at 5:24 PM, Stefan Miklosovic <mikloso...@gmail.com> > wrote: > >> I have did it as you suggested and it seems to start the jobs OK and I > >> see the sessions in UI but while it is being computed (I see the job > >> is distributed on two spark slaves where in front of that there is > >> spark-master), I am computing this from my localhost: > >> > >> @RunWith(JUnit4.class) > >> public class LivyTestCase { > >> > >> private static final int SAMPLES = 10000; > >> > >> private static final String LIVY_URI = "https://urldefense. > proofpoint.com/v2/url?u=http-3A__spark-2Dmaster-3A8998&d= > DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ- > cx37DPYDyo&m=m06yDfiPVPYT_h4awyPxVA7SKfA9V1MlTzp3mQ8REcE&s=v2g- > MD9i6gEg80s1fyGAzLe9WCOhelMfEygM1WOwNeM&e="; > > >> > >> @Test > >> public void testPiJob() throws Exception { > >> > >> LivyClient client = new LivyClientBuilder() > >> .setURI(new URI(LIVY_URI)) > >> .build(); > >> > >> final Double result = client.submit(new PiJob(1000)).get(); > >> > >> System.out.println(result); > >> } > >> } > >> > >> It is a PiJob from sites examples. > >> > >> Now what I see in Livy logs is this > >> > >> org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class: > PiJob > >> at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName( > DefaultClassResolver.java:138) > >> at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver. > readClass(DefaultClassResolver.java:115) > >> at org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656) > >> at org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject( > Kryo.java:767) > >> at org.apache.livy.client.common.Serializer.deserialize( > Serializer.java:63) > >> at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39) > >> at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27) > >> at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57) > >> at org.apache.livy.rsc.driver.BypassJobWrapper.call( > BypassJobWrapper.java:42) > >> at org.apache.livy.rsc.driver.BypassJobWrapper.call( > BypassJobWrapper.java:27) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >> at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149) > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:624) > >> at java.lang.Thread.run(Thread.java:748) > >> Caused by: java.lang.ClassNotFoundException: PiJob > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >> at java.lang.Class.forName0(Native Method) > >> at java.lang.Class.forName(Class.java:348) > >> at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName( > DefaultClassResolver.java:136) > >> ... 13 more > >> > >> I know to read it - I understand there is not PiJob on the class path, > >> but why? I have just sent that class to Livy ... > >> > >> On Thu, Oct 26, 2017 at 4:17 PM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >>> You can choose to set "livy.spark.master" to "local" and > >>> "livy.spark.deploy-mode" to "client" to start Spark with local mode, > in such > >>> case YARN is not required. > >>> > >>> Otherwise if you plan to run on YARN, you have to install Hadoop and > >>> configure HADOOP_CONF_DIR in livy-env.sh. > >>> > >>> On Thu, Oct 26, 2017 at 9:40 PM, Stefan Miklosovic < > mikloso...@gmail.com> > >>> wrote: > >>>> > >>>> Hi, > >>>> > >>>> I am running Livy server in connection with Spark without Hadoop. I am > >>>> setting only SPARK_HOME and I am getting this in Livy UI logs after > >>>> job submission. > >>>> > >>>> I am using pretty much standard configuration but > >>>> livy.spark.deploy-mode = cluster > >>>> > >>>> Do I need to run with Hadoop installation as well and specify > >>>> HADOOP_CONF_DIR? > >>>> > >>>> Is not it possible to run Livy with "plain" Spark without YARN? > >>>> > >>>> stderr: > >>>> java.lang.ClassNotFoundException: > >>>> at java.lang.Class.forName0(Native Method) > >>>> at java.lang.Class.forName(Class.java:348) > >>>> at org.apache.spark.util.Utils$.classForName(Utils.scala:230) > >>>> at > >>>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:712) > >>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:180) > >>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > >>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) > >>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > >>>> > >>>> Thanks! > >>>> > >>>> -- > >>>> Stefan Miklosovic > >>> > >>> > >> > >> > >> > >> -- > >> Stefan Miklosovic > > > > > > > > -- > > Stefan Miklosovic > > > > -- > Stefan Miklosovic > > > > > -- Stefan Miklosovic
