Re: [ANNOUNCE] Apache Livy 0.6.0-incubating released
Thanks Marcelo 于2019年4月3日周三 下午12:15写道： > Nice!! > > Sent from my iPhone > > On Apr 2, 2019, at 8:21 PM, Ruslan Dautkhanov > wrote: > > Thanks a lot Marcelo ! > > Ruslan > > > > On Tue, Apr 2, 2019 at 12:24 PM Marcelo Vanzin > wrote: > >> The Apache Livy team is proud to announce the release of Apache Livy >> 0.6.0-incubating. >> >> Livy is web service that exposes a REST interface for managing long >> running Apache Spark contexts in your cluster. Livy enables >> programmatic, fault-tolerant, multi-tenant submission of Spark jobs >> from web/mobile apps (no Spark client needed). So, multiple users can >> interact with your Spark cluster concurrently and reliably. >> >> Download Apache Livy 0.6.0-incubating: >> http://livy.incubator.apache.org/download/ >> >> Release Notes: >> http://livy.incubator.apache.org/history/ >> >> For more about Livy check our website: >> http://livy.incubator.apache.org/ >> >> We would like to thank the contributors that made the release possible! >> >> >> -- >> Marcelo >> > -- > > -- > Ruslan Dautkhanov > > -- Best Regards Jeff Zhang
Re: [ANNOUNCE] Apache Livy 0.6.0-incubating released
Thanks a lot Marcelo ! Ruslan On Tue, Apr 2, 2019 at 12:24 PM Marcelo Vanzin wrote: > The Apache Livy team is proud to announce the release of Apache Livy > 0.6.0-incubating. > > Livy is web service that exposes a REST interface for managing long > running Apache Spark contexts in your cluster. Livy enables > programmatic, fault-tolerant, multi-tenant submission of Spark jobs > from web/mobile apps (no Spark client needed). So, multiple users can > interact with your Spark cluster concurrently and reliably. > > Download Apache Livy 0.6.0-incubating: > http://livy.incubator.apache.org/download/ > > Release Notes: > http://livy.incubator.apache.org/history/ > > For more about Livy check our website: > http://livy.incubator.apache.org/ > > We would like to thank the contributors that made the release possible! > > > -- > Marcelo > -- -- Ruslan Dautkhanov
Re: FileNotFoundException while testing programatic example on Livy 0.5-incubating
I think that I have figured out what the problem was. The JAR uploaded through LivyClient.uploadJar(File) will only be available on the node where Livy is running and when running Spark on Yarn in cluster mode the JAR must be available on all the nodes. The way I solved it was to copy the JAR to a path in HDFS and then add that jar to the client using LivyClient.addJar(URI) instead of LivyClient.uploadJar(File). /Pontus On Tue, Apr 2, 2019 at 3:01 PM Pontus Amberg wrote: > I've encountered a problem while testing Livy 0.5 where the upload JAR > can't be found. > > I'm using Hadoop 3.1.1, Spark 2.4.0 and Livy 0.5 in a small cluster where > Livy is running on > the NameNode/ResourceManager. Spark is running in cluster mode on Yarn. > > When I try to use the Java "programmatic" example ( > https://livy.incubator.apache.org/docs/latest/programmatic-api.html ) > I get an FileNotFoundException on the line > client.uploadJar(new File(piJar)).get(); > I can see that the JAR-file is uploaded (including and .crc file) on the > server in the "livy.session.staging-dir" > folder that I have specified in livy.conf but after while I get the > following exception on the client: > > java.util.concurrent.ExecutionException: java.io.IOException: Internal > Server Error: "java.util.concurrent.ExecutionException: > java.lang.RuntimeException: java.io.FileNotFoundException: > File > file:/usr/local/livy/uploads/31879afa-6de5-4b7a-a759-6b091031ddb0/livy-jobs-1.0.0-SNAPSHOT.jar > does not exist > > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) > > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) > > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) > > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) > org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) > > org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:88) > org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979) > org.apache.livy.rsc.driver.RSCDriver.copyFileToLocal(RSCDriver.java:506) > org.apache.livy.rsc.driver.RSCDriver.addJarOrPyFile(RSCDriver.java:476) > org.apache.livy.repl.ReplDriver.addJarOrPyFile(ReplDriver.scala:116) > > org.apache.livy.rsc.driver.JobContextImpl.addJarOrPyFile(JobContextImpl.java:151) > org.apache.livy.rsc.driver.AddJarJob.call(AddJarJob.java:39) > org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57) > org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:34) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)" > > So its a bit confusing why I get the FileNotFoundException when I can see > that the file has been uploaded. > Anyone have clue what might be causing this? > > /Pontus >
[ANNOUNCE] Apache Livy 0.6.0-incubating released
The Apache Livy team is proud to announce the release of Apache Livy 0.6.0-incubating. Livy is web service that exposes a REST interface for managing long running Apache Spark contexts in your cluster. Livy enables programmatic, fault-tolerant, multi-tenant submission of Spark jobs from web/mobile apps (no Spark client needed). So, multiple users can interact with your Spark cluster concurrently and reliably. Download Apache Livy 0.6.0-incubating: http://livy.incubator.apache.org/download/ Release Notes: http://livy.incubator.apache.org/history/ For more about Livy check our website: http://livy.incubator.apache.org/ We would like to thank the contributors that made the release possible! -- Marcelo
FileNotFoundException while testing programatic example on Livy 0.5-incubating
I've encountered a problem while testing Livy 0.5 where the upload JAR can't be found. I'm using Hadoop 3.1.1, Spark 2.4.0 and Livy 0.5 in a small cluster where Livy is running on the NameNode/ResourceManager. Spark is running in cluster mode on Yarn. When I try to use the Java "programmatic" example ( https://livy.incubator.apache.org/docs/latest/programmatic-api.html ) I get an FileNotFoundException on the line client.uploadJar(new File(piJar)).get(); I can see that the JAR-file is uploaded (including and .crc file) on the server in the "livy.session.staging-dir" folder that I have specified in livy.conf but after while I get the following exception on the client: java.util.concurrent.ExecutionException: java.io.IOException: Internal Server Error: "java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.FileNotFoundException: File file:/usr/local/livy/uploads/31879afa-6de5-4b7a-a759-6b091031ddb0/livy-jobs-1.0.0-SNAPSHOT.jar does not exist org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611) org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:88) org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979) org.apache.livy.rsc.driver.RSCDriver.copyFileToLocal(RSCDriver.java:506) org.apache.livy.rsc.driver.RSCDriver.addJarOrPyFile(RSCDriver.java:476) org.apache.livy.repl.ReplDriver.addJarOrPyFile(ReplDriver.scala:116) org.apache.livy.rsc.driver.JobContextImpl.addJarOrPyFile(JobContextImpl.java:151) org.apache.livy.rsc.driver.AddJarJob.call(AddJarJob.java:39) org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57) org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:34) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)" So its a bit confusing why I get the FileNotFoundException when I can see that the file has been uploaded. Anyone have clue what might be causing this? /Pontus
