Re: Can a livy client upload and run different jar packages multiple times?

2020-03-08 Thread Saisai Shao 
Though we haven't tried such use case before, from my understanding I think
Livy cannot support it, and such usage scenario seems not a typical usage
pattern.

adley  于2020年3月8日周日 下午9:40写道：

> Hi：
> I followed the official example-PiApp.java, and tried to submit and
> run successfully.
> Then I modified the code and called client.uploadJar (newjar) again
> after client.submit, and finally called the class implementation in newjar,
> client.submit ( newjar-newclass), the upload is successful, no error is
> reported, but it has no effect. The execution result shows that the class
> implementation in oldjar is still called.
> Can Livy support the repeated submission of jar packages under one
> client?
>

[ANNOUNCE] Apache Livy 0.7.0-incubating released

2020-02-02 Thread Saisai Shao 
The Apache Livy team is proud to announce the release of Apache Livy
0.7.0-incubating.

Livy is web service that exposes a REST interface for managing long
running Apache Spark contexts in your cluster. Livy enables
programmatic, fault-tolerant, multi-tenant submission of Spark jobs
from web/mobile apps (no Spark client needed). So, multiple users can
interact with your Spark cluster concurrently and reliably.

Download Apache Livy 0.7.0-incubating:
http://livy.incubator.apache.org/download/

Release Notes:
http://livy.incubator.apache.org/history/

For more about Livy check our website:
http://livy.incubator.apache.org/

We would like to thank the contributors that made the release possible!

=
*Disclaimer*

Apache Livy (incubating) is an effort undergoing incubation at The
Apache Software Foundation (ASF), sponsored by the name of Apache
Incubator PMC. Incubation is required of all newly accepted projects
until a further review indicates that the infrastructure,
communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the
completeness or stability of the code, it does indicate that the
project has yet to be fully endorsed by the ASF.

Re: Running local and yarn using the same livy server

2020-02-01 Thread Saisai Shao 
I don't think current Livy support such behavior, the cluster manager
specified in the conf file is a global configuration which affects all the
created sessions.

Thanks
Saisai

Ravi Shankar  于2020年1月28日周二 上午4:02写道：

> Hey guys,
> Is there a way to start different kind of spark sessions using the same
> livy server ? For instance, i want my application to tell livy whether the
> new session being requested should be started as a "local" session or a
> "yarn" session on the cluster.
>
> Ravi
>

Re: livy and security concerns

2019-10-31 Thread Saisai Shao 
Livy is just a gateway service, I think it is not Livy's responsibility to
provide a sandbox. Also it is super hard to analyze the code from Livy
side. Typically some companies will provide a sand-boxing runtime, like JVM
or others, or put Spark into container for security and isolation.

Thanks
Saisai

Meisam Fathi  于2019年11月1日周五 上午2:57写道：

> No, it does not. The code will run on a Spark session. The sandboxing
> should be supported, at least partially, by Spark. But as far as I know,
> Spark dies not have such a feature.
>
> Thanks,
> Meisam
>
> On Wed, Oct 30, 2019, 5:02 PM mhd wrk  wrote:
>
>> Considering that Livy supports Interactive Scala or Python, does it
>> provides any sand-boxing feature to protect the back-end against submitted
>> code?
>>
>

Re: Creating session in livy with jars parameter

2019-08-11 Thread Saisai Shao 
Can you please paste the exception?

Pavel Sivak  于2019年8月12日周一 上午11:52写道：

> Sure Saisai,
> First step - run livy server. I can see that server is running and web
> interface is available
> Second step - session creation. Example of body in POST request
>
>> {"kind":"spark", "jars":["local:/path_to_jar/test.jar"]}
>>
> Session is *STARTING* status, I can see that exception in LOG, but SPARK
> UI is available and I can see my jar in environment TAB
>
> But if I'm using some other library(I took guava for example) - session is
> in *IDLE* status
>
>> {"kind":"spark", "jars":["local:/path_to_guava_jar/guava.jar"]}
>
>
> I don't understand what can be the difference between my jar and some
> other jars...
>
> On Sun, Aug 11, 2019 at 10:46 PM Saisai Shao 
> wrote:
>
>> Would you mind listing the steps to reproduce your issue, and how do you
>> use REST APIs?
>>
>> Thanks
>> Saisai
>>
>> Pavel Sivak  于2019年8月10日周六 上午11:01写道：
>>
>>> Hi,
>>> My idea is to create livy session with my library in class path using
>>> "jars" parameter.
>>> I'm using REST API to create a session. After sending POST request I can
>>> see that spark session is UP, I can use Spark UI and there's my jar in the
>>> Environment TAB.
>>> But status of the session in livy is "Starting"...
>>> This is example from log file:
>>>
>>>> 19/08/09 22:10:37 INFO driver.SparkEntries: Created Spark session.
>>>> Exception in thread "main" java.lang.NullPointerException
>>>>at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
>>>>at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
>>>>at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
>>>>at 
>>>> org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
>>>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>at 
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>at 
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>at java.lang.reflect.Method.invoke(Method.java:498)
>>>>at 
>>>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>>>at 
>>>> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>>>>at 
>>>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>>>>at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>>>
>>>> Without my library livy session I'm getting livy session in Idle status.
>>> Can you help me to figure this out?
>>> Thanks
>>> --
>>> Best wishes,
>>> Pavel Sivak
>>>
>>
>
> --
> Best wishes,
> Pavel Sivak
>

Re: Creating session in livy with jars parameter

2019-08-11 Thread Saisai Shao 
Would you mind listing the steps to reproduce your issue, and how do you
use REST APIs?

Thanks
Saisai

Pavel Sivak  于2019年8月10日周六 上午11:01写道：

> Hi,
> My idea is to create livy session with my library in class path using
> "jars" parameter.
> I'm using REST API to create a session. After sending POST request I can
> see that spark session is UP, I can use Spark UI and there's my jar in the
> Environment TAB.
> But status of the session in livy is "Starting"...
> This is example from log file:
>
>> 19/08/09 22:10:37 INFO driver.SparkEntries: Created Spark session.
>> Exception in thread "main" java.lang.NullPointerException
>>  at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
>>  at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
>>  at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
>>  at 
>> org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>  at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>  at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>  at java.lang.reflect.Method.invoke(Method.java:498)
>>  at 
>> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>>  at 
>> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
>>  at 
>> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
>>  at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
>>
>> Without my library livy session I'm getting livy session in Idle status.
> Can you help me to figure this out?
> Thanks
> --
> Best wishes,
> Pavel Sivak
>

Re: Pre-registering UDTs / UDFs in Livy session

2019-08-11 Thread Saisai Shao 
Unfortunately there's no such mechanism to inject custom code when session
is started in Livy side. I think you can add some code in Spark side, Spark
has a listener hook "SparkListener", in which there has a hook `
onApplicationStart`, this hook will be called immediately after application
is started. You can take a look at SparkListener.

Thanks
Saisai

Sergii Mikhtoniuk  于2019年8月12日周一 上午6:14写道：

> Hi,
>
> I'm currently using Livy in two different contexts:
> - from Jupyter notebooks
> - from SqlLine/Beeline CLI over Thrift/JDBC connection.
>
> The data I work with includes GIS, so it sometimes necessary to register
> custom (GeoSpark) geometry UDTs and UDFs in the Spark session.
>
> For Jupyter notebook case I was able to simply add a custom step to my
> Jupyter kernel that registers UDTs after session is created, but I don't
> know how to achieve the same in JDBC client scenario.
>
> Is there any extension mechanism in Livy or Spark that would execute a
> custom code on session init or to automatically discover and register
> UDFs/UDTs?
>
> As I understand from https://issues.apache.org/jira/browse/SPARK-7768 the
> UDT mechanism is still in flux, but perhaps there's a better solution than
> to fork Livy to add my custom registration code.
>
> Any pointers are much appreciated.
>
> - Sergii
>

Re: Support for Livy with Scala 2.12

2019-06-05 Thread Saisai Shao 
 Sorry I don't have bandwidth to support 2.12, I can help to review it if
someone can do this.

Thanks
Saisai

 于2019年6月5日周三 上午10:09写道：

> Hi Saisai,
>
> I’m not familiar with Livy code.  We’re just using it for our Jupyter
> integration.
>
> I’m looking through PR for 2.11 migration that was done an year ago and it
> looks like it is mostly Pom changes.  If that’s not correct than I might
> need help to perform the upgrade.
>
> Do you have bandwidth to make this change?
>
>
>
>
> From: Saisai Shao mailto:sai.sai.s...@gmail.com>>
> Date: Tuesday, Jun 04, 2019, 8:56 PM
> To: user@livy.incubator.apache.org  user@livy.incubator.apache.org>>
> Subject: [External] Re: Support for Livy with Scala 2.12
>
> If you're familiar with Livy code, I think the effort is not so big.
> According to my previous experience on Scala 2.10 support, some codes may
> need to be changed because of version incompatible for Scala.
>
> Thanks
> Saisai
>
> mailto:santosh.dan...@ubs.com>> 于2019年6月4日周二
> 下午8:25写道：
> How much effort we need to put in to create 2.12 module? Is that just a
> change in POM files or code change is required?
>
> We have release planned for July to upgrade Jupyter and Livy to utilize
> spark 2.4.2.  This is blocking us from upgrade.
> From: Saisai Shao mailto:sai.sai.s...@gmail.com
> ><mailto:sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>>>
> Date: Monday, Jun 03, 2019, 9:02 PM
> To: user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>
> mailto:user@livy.incubator.apache.org
> ><mailto:user@livy.incubator.apache.org user@livy.incubator.apache.org>>>
> Subject: [External] Re: Support for Livy with Scala 2.12
>
> Like what we did before to support both Scala 2.10 and 2.11 in Livy, I
> think we should also have a new module to support 2.12.
>
> mailto:santosh.dan...@ubs.com> santosh.dan...@ubs.com<mailto:santosh.dan...@ubs.com>>> 于2019年6月4日周二
> 上午7:40写道：
> Yes, the spark binary we downloaded is built with default Scala 2.12.  We
> want to use databricks delta which I think only support Scala 2.12.  So,
> I'm stuck with Scala 2.12.  Moreover, Spark community is going to
> decommission Scala 2.11 completely from Spark 3.0 release.  We might need
> to prepare Livy to support Scala 2.12 by default.
>
> From: Kevin Risden [mailto:kris...@apache.org<mailto:kris...@apache.org
> ><mailto:kris...@apache.org<mailto:kris...@apache.org>>]
> Sent: Monday, June 03, 2019 6:35 PM
> To: user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org
> ><mailto:user@livy.incubator.apache.org user@livy.incubator.apache.org>>
> Subject: [External] Re: Support for Livy with Scala 2.12
>
> Looks like the issue might be Spark 2.4.2 only? From
> https://spark.apache.org/downloads.html, "Note that, Spark is pre-built
> with Scala 2.11 except version 2.4.2, which is pre-built with Scala 2.12."
> So maybe you just got unlucky with using Spark 2.4.2?
>
> Kevin Risden
>
>
> On Mon, Jun 3, 2019 at 6:19 PM  santosh.dan...@ubs.com><mailto:santosh.dan...@ubs.com santosh.dan...@ubs.com>>> wrote:
> Kevin,
>
> I'm using Livy 0.6.0.  The issues is related to not finding repl jars that
> support scala 2.12.  The error "requirement failed: Cannot find Livy REPL
> jars." is thrown because it couldn't find folder repl_2.12-jars under LIVY
> directory.
>
> I performed a test to make sure this issue is related to scala 2.12
> compatibility , I copied contents of repl_2.11-jars under Livy directory
> into new directory LIVY/repl_2.12-jars and this time I didn't get REPL jars
> exception it went ahead and created session but failed to start session due
> to rsc jars version incompatibility.
>
> LIVY Folder structure for error " requirement failed: Cannot find Livy
> REPL jars.""
>
> [/app/risk/ha02/livy]$ ls -ltr
> total 116
> -rwxr-xr-x 1 agriddev agriddev   160 Mar 19 14:39 NOTICE
> -rwxr-xr-x 1 agriddev agriddev 18665 Mar 19 14:39 LICENSE
> -rwxr-xr-x 1 agriddev agriddev   537 Mar 19 14:39 DISCLAIMER
> -rwxr-xr-x 1 agriddev agriddev 46355 Mar 19 14:42 THIRD-PARTY
> drwxr-xr-x 2 agriddev agriddev  4096 Mar 19 14:43 bin
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 repl_2.11-jars
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 rsc-jars
> drwxr-xr-x 2 agriddev agriddev 12288 Apr 14 22:37 jars
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37
> apache-livy-0.6.0-incubating-bin
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 17:37 conf
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 21:51 logs
>
> LIVY FOLDER STRUCTURE TO BYPASS "REQUIREMENT FAILED:CANNOT FIND LIVY R

Re: Support for Livy with Scala 2.12

2019-06-04 Thread Saisai Shao 
If you're familiar with Livy code, I think the effort is not so big.
According to my previous experience on Scala 2.10 support, some codes may
need to be changed because of version incompatible for Scala.

Thanks
Saisai

 于2019年6月4日周二 下午8:25写道：

> How much effort we need to put in to create 2.12 module? Is that just a
> change in POM files or code change is required?
>
> We have release planned for July to upgrade Jupyter and Livy to utilize
> spark 2.4.2.  This is blocking us from upgrade.
> From: Saisai Shao mailto:sai.sai.s...@gmail.com>>
> Date: Monday, Jun 03, 2019, 9:02 PM
> To: user@livy.incubator.apache.org  user@livy.incubator.apache.org>>
> Subject: [External] Re: Support for Livy with Scala 2.12
>
> Like what we did before to support both Scala 2.10 and 2.11 in Livy, I
> think we should also have a new module to support 2.12.
>
> mailto:santosh.dan...@ubs.com>> 于2019年6月4日周二
> 上午7:40写道：
> Yes, the spark binary we downloaded is built with default Scala 2.12.  We
> want to use databricks delta which I think only support Scala 2.12.  So,
> I'm stuck with Scala 2.12.  Moreover, Spark community is going to
> decommission Scala 2.11 completely from Spark 3.0 release.  We might need
> to prepare Livy to support Scala 2.12 by default.
>
> From: Kevin Risden [mailto:kris...@apache.org<mailto:kris...@apache.org>]
> Sent: Monday, June 03, 2019 6:35 PM
> To: user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>
> Subject: [External] Re: Support for Livy with Scala 2.12
>
> Looks like the issue might be Spark 2.4.2 only? From
> https://spark.apache.org/downloads.html, "Note that, Spark is pre-built
> with Scala 2.11 except version 2.4.2, which is pre-built with Scala 2.12."
> So maybe you just got unlucky with using Spark 2.4.2?
>
> Kevin Risden
>
>
> On Mon, Jun 3, 2019 at 6:19 PM  santosh.dan...@ubs.com>> wrote:
> Kevin,
>
> I'm using Livy 0.6.0.  The issues is related to not finding repl jars that
> support scala 2.12.  The error "requirement failed: Cannot find Livy REPL
> jars." is thrown because it couldn't find folder repl_2.12-jars under LIVY
> directory.
>
> I performed a test to make sure this issue is related to scala 2.12
> compatibility , I copied contents of repl_2.11-jars under Livy directory
> into new directory LIVY/repl_2.12-jars and this time I didn't get REPL jars
> exception it went ahead and created session but failed to start session due
> to rsc jars version incompatibility.
>
> LIVY Folder structure for error " requirement failed: Cannot find Livy
> REPL jars.""
>
> [/app/risk/ha02/livy]$ ls -ltr
> total 116
> -rwxr-xr-x 1 agriddev agriddev   160 Mar 19 14:39 NOTICE
> -rwxr-xr-x 1 agriddev agriddev 18665 Mar 19 14:39 LICENSE
> -rwxr-xr-x 1 agriddev agriddev   537 Mar 19 14:39 DISCLAIMER
> -rwxr-xr-x 1 agriddev agriddev 46355 Mar 19 14:42 THIRD-PARTY
> drwxr-xr-x 2 agriddev agriddev  4096 Mar 19 14:43 bin
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 repl_2.11-jars
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 rsc-jars
> drwxr-xr-x 2 agriddev agriddev 12288 Apr 14 22:37 jars
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37
> apache-livy-0.6.0-incubating-bin
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 17:37 conf
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 21:51 logs
>
> LIVY FOLDER STRUCTURE TO BYPASS "REQUIREMENT FAILED:CANNOT FIND LIVY REPL
> JARS"
>
> [/app/risk/ha02/livy]$ ls -ltr
> total 116
> -rwxr-xr-x 1 agriddev agriddev   160 Mar 19 14:39 NOTICE
> -rwxr-xr-x 1 agriddev agriddev 18665 Mar 19 14:39 LICENSE
> -rwxr-xr-x 1 agriddev agriddev   537 Mar 19 14:39 DISCLAIMER
> -rwxr-xr-x 1 agriddev agriddev 46355 Mar 19 14:42 THIRD-PARTY
> drwxr-xr-x 2 agriddev agriddev  4096 Mar 19 14:43 bin
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 repl_2.11-jars
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 rsc-jars
> drwxr-xr-x 2 agriddev agriddev 12288 Apr 14 22:37 jars
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37
> apache-livy-0.6.0-incubating-bin
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 17:37 conf
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 21:50 repl_2.12-jars
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 21:51 logs
>
>
>
> Error Information
>
> zip
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: Changing view acls to: agriddev
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: Changing modify acls to: agriddev
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: Changing view acls groups to:
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 I

Re: Support for Livy with Scala 2.12

2019-06-03 Thread Saisai Shao 
Like what we did before to support both Scala 2.10 and 2.11 in Livy, I
think we should also have a new module to support 2.12.

 于2019年6月4日周二 上午7:40写道：

> Yes, the spark binary we downloaded is built with default Scala 2.12.  We
> want to use databricks delta which I think only support Scala 2.12.  So,
> I'm stuck with Scala 2.12.  Moreover, Spark community is going to
> decommission Scala 2.11 completely from Spark 3.0 release.  We might need
> to prepare Livy to support Scala 2.12 by default.
>
>
>
> *From:* Kevin Risden [mailto:kris...@apache.org]
> *Sent:* Monday, June 03, 2019 6:35 PM
> *To:* user@livy.incubator.apache.org
> *Subject:* [External] Re: Support for Livy with Scala 2.12
>
>
>
> Looks like the issue might be Spark 2.4.2 only? From
> https://spark.apache.org/downloads.html, "Note that, Spark is pre-built
> with Scala 2.11 except version 2.4.2, which is pre-built with Scala 2.12."
> So maybe you just got unlucky with using Spark 2.4.2?
>
>
>
> Kevin Risden
>
>
>
>
>
> On Mon, Jun 3, 2019 at 6:19 PM  wrote:
>
> Kevin,
>
>
>
> I'm using Livy 0.6.0.  The issues is related to not finding repl jars that
> support scala 2.12.  The error "requirement failed: Cannot find Livy REPL
> jars." is thrown because it couldn't find folder repl_2.12-jars under
> LIVY directory.
>
>
>
> I performed a test to make sure this issue is related to scala 2.12
> compatibility , I copied contents of repl_2.11-jars under Livy directory
> into new directory LIVY/repl_2.12-jars and this time I didn't get REPL jars
> exception it went ahead and created session but failed to start session due
> to rsc jars version incompatibility.
>
>
>
> *LIVY Folder structure for error " requirement failed: Cannot find Livy
> REPL jars.""*
>
>
>
> [/app/risk/ha02/livy]$ ls -ltr
>
> total 116
>
> -rwxr-xr-x 1 agriddev agriddev   160 Mar 19 14:39 NOTICE
>
> -rwxr-xr-x 1 agriddev agriddev 18665 Mar 19 14:39 LICENSE
>
> -rwxr-xr-x 1 agriddev agriddev   537 Mar 19 14:39 DISCLAIMER
>
> -rwxr-xr-x 1 agriddev agriddev 46355 Mar 19 14:42 THIRD-PARTY
>
> drwxr-xr-x 2 agriddev agriddev  4096 Mar 19 14:43 bin
>
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 repl_2.11-jars
>
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 rsc-jars
>
> drwxr-xr-x 2 agriddev agriddev 12288 Apr 14 22:37 jars
>
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37
> apache-livy-0.6.0-incubating-bin
>
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 17:37 conf
>
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 21:51 logs
>
>
>
> *LIVY FOLDER STRUCTURE TO BYPASS "REQUIREMENT FAILED:CANNOT FIND LIVY REPL
> JARS"*
>
>
>
> [/app/risk/ha02/livy]$ ls -ltr
>
> total 116
>
> -rwxr-xr-x 1 agriddev agriddev   160 Mar 19 14:39 NOTICE
>
> -rwxr-xr-x 1 agriddev agriddev 18665 Mar 19 14:39 LICENSE
>
> -rwxr-xr-x 1 agriddev agriddev   537 Mar 19 14:39 DISCLAIMER
>
> -rwxr-xr-x 1 agriddev agriddev 46355 Mar 19 14:42 THIRD-PARTY
>
> drwxr-xr-x 2 agriddev agriddev  4096 Mar 19 14:43 bin
>
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 repl_2.11-jars
>
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37 rsc-jars
>
> drwxr-xr-x 2 agriddev agriddev 12288 Apr 14 22:37 jars
>
> drwxr-xr-x 2 agriddev agriddev  4096 Apr 14 22:37
> apache-livy-0.6.0-incubating-bin
>
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 17:37 conf
>
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 21:50 repl_2.12-jars
>
> drwxr-xr-x 2 agriddev agriddev  4096 Jun  3 21:51 logs
>
>
>
>
>
> Error Information
>
>
>
> zip
>
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: Changing view acls to: agriddev
>
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: Changing modify acls to: agriddev
>
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: Changing view acls groups to:
>
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: Changing modify acls groups to:
>
> 19/06/03 21:52:00 INFO LineBufferedStream: 19/06/03 21:52:00 INFO
> SecurityManager: SecurityManager: authentication disabled; ui acls
> disabled; users  with view permissions: Set(agriddev); groups with view
> permissions: Set(); users  with modify permissions: Set(agriddev); groups
> with modify permissions: Set()
>
> 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO Client:
> Submitting application application_1559316432251_0172 to ResourceManager
>
> 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO
> YarnClientImpl: Submitted application application_1559316432251_0172
>
> 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO Client:
> Application report for application_1559316432251_0172 (state: ACCEPTED)
>
> 19/06/03 21:52:01 INFO LineBufferedStream: 19/06/03 21:52:01 INFO Client:
>
> 19/06/03 21:52:01 INFO LineBufferedStream:   client token: N/A
>
> 19/06/03 21:52:01 INFO LineBufferedStream:   diagnostics: [Mon Jun 03
> 21:52:01 + 2019] Application is Activated,

Re: Livy-0.6 release?

2019-03-12 Thread Saisai Shao 
I can also help to release a new version. My only concern is that how
mature the thrift module is, shall we enable it by default or leave it
disabled?

Thanks
Saisai

Jeff Zhang  于2019年3月12日周二 上午10:54写道：

> Thanks Marcelo, I can help to test it in zeppelin side which use livy as
> one interpreter.
>
> Marcelo Vanzin  于2019年3月12日周二 上午7:25写道：
>
>> Since there isn't much activity going on from the project committers,
>> I guess I could spend some time to create a release.
>>
>> The main problem from my side is that I haven't actually used Livy in
>> a long time. So personally I have no idea of how stable the current
>> master is, and the most I can do is just run the built-in integration
>> tests. So there would be a release (assuming other PPMC members are
>> still around), but I wouldn't really be able to attest to its
>> stability. If people are ok with that...
>>
>> On Sat, Mar 2, 2019 at 6:04 AM kant kodali  wrote:
>> >
>> > Any rough timeline on 0.6? IfLivyy doesn't allow to choose a higher
>> spark version I guess that will be a blocker fora lot of people who want to
>> leverage new features from spark. Any good solution to fix this?
>> >
>> > On Mon, Feb 11, 2019 at 3:46 PM Ruslan Dautkhanov 
>> wrote:
>> >>
>> >> Got it. Thanks Marcelo.
>> >>
>> >> I see LIVY-551 is now part of the master. Hope to see Livy 0.6 perhaps
>> soon.
>> >>
>> >>
>> >> Thank you!
>> >> Ruslan Dautkhanov
>> >>
>> >>
>> >> On Tue, Feb 5, 2019 at 12:38 PM Marcelo Vanzin 
>> wrote:
>> >>>
>> >>> I think LIVY-551 is the current blocker. Unfortunately I don't think
>> >>> we're really tracking things in jira that well, as far as releases go.
>> >>> At least I'm not.
>> >>>
>> >>> On Mon, Feb 4, 2019 at 6:32 PM Ruslan Dautkhanov <
>> dautkha...@gmail.com> wrote:
>> >>> >
>> >>> > +1 for 0.6 release so folks can upgrade to Spark 2.4..
>> >>> >
>> >>> > Marcelo, what particular patches are blocking Livy 0.6 release?
>> >>> >
>> >>> > I see 3 jiras with 0.6 as Fix Version - not sure if that's correct
>> way to find blockers.
>> >>> > https://goo.gl/9axfsw
>> >>> >
>> >>> >
>> >>> > Thank you!
>> >>> > Ruslan Dautkhanov
>> >>> >
>> >>> >
>> >>> > On Mon, Jan 28, 2019 at 2:24 PM Marcelo Vanzin 
>> wrote:
>> >>> >>
>> >>> >> There are a couple of patches under review that are currently
>> blocking
>> >>> >> the release.
>> >>> >>
>> >>> >> Once those are done, we can work on releasing 0.6.
>> >>> >>
>> >>> >> On Mon, Jan 28, 2019 at 11:18 AM Roger Liu <
>> liu.ro...@microsoft.com> wrote:
>> >>> >> >
>> >>> >> > Hey there,
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > I’m wondering if we have a timeline for releasing Livy-0.6? Its
>> been a year since the last release and there are features like Spark-2.4
>> support that are not incorporated in the livy-0.5 package.
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > Thanks,
>> >>> >> >
>> >>> >> > Roger Liu
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Marcelo
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Marcelo
>>
>>
>>
>> --
>> Marcelo
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: about LIVY-424

2018-11-11 Thread Saisai Shao 
If you're hitting the problem of LIVY-424, this is exactly a Scala REPL
issue, not a Livy issue, it is hard to fix in the Livy side.

lk_hadoop  于2018年11月12日周一 上午9:37写道：

> hi,all:
> I meet this issue https://issues.apache.org/jira/browse/LIVY-424
> , anybody know how to resolve it?
> 2018-11-12
> --
> lk_hadoop
>

Re: Some questions about cached data in Livy

2018-07-11 Thread Saisai Shao 
Hi Wandong,

Livy's shared object mechanism mainly used to share objects between
different Livy jobs, this is mainly used for Job API. For example job A
create a object Foo which wants to be accessed by Job B, then user could
store this object Foo into JobContext with a provided name, after that Job
B could get this object by the name.

This is different from Spark's cache mechanism. What you mentioned above
(tmp table) is a Spark provided table cache mechanism, which is unrelated
to Livy.



Wandong Wu  于2018年7月11日周三 下午5:46写道：

> Dear Sir or Madam:
>
>   I am a Livy beginner. I use Livy, because within an interactive
> session, different spark jobs could share cached RDDs or DataFrames.
>
>   When I read some parquet files and create a table called “TmpTable”.
> The following queries will use this table. Does it mean this table has been
> cached?
>   If cached, where is the table cached? The table is cached in Livy or
> Spark cluster?
>
>   Spark also supports cache function.  When I read some parquet files
> and create a table called “TmpTable2”. I add such code: sql_ctx.cacheTable(
> *'tmpTable2'*).
>   In the next query using this table. It will be cached in Spark
> cluster. Then the following queries could use this cached table.
>
>   What is the difference between cached in Livy and cached in Spark
> cluster?
>
> Thanks!
>
> Yours
> Wandong
>
>

Re: Does Livy Support Basic Auth?

2018-07-01 Thread Saisai Shao 
No, it doesn't support, you should change the code to add a specific Filter
to make it work.

Harsch, Tim  于2018年6月30日周六 上午12:49写道：

> I thought I read somewhere that Livy supports basic auth, but don't see
> configurations for it.  Does it?
>

Re: Query regarding livy session

2018-06-03 Thread Saisai Shao 
This is by design that Livy doesn't support concurrent execution of
different requests. Because Livy doesn't know if the subsequent requests
are semantically dependent or not.

Rajeshkumar R  于2018年6月3日周日 下午12:08写道：

> Hello
>> I have a livy session which is linked to a spark context. This spark
>> context is holding to data from a table in memory. My idea is to use this
>> data shareble across multiple users using same livy session and context.
>> But livy is not allowing two request at the sametime to the same session.
>> The second request is in queue when the first is being processed. Is this
>> the way livy is designed or there is some issue in our code?
>>
>> Thanks
>> Rajesh
>>
>>
>>

Re: Predef does not have a member classOf

2018-02-22 Thread Saisai Shao 
This seems like a Scala version issue (as far as I can see from the stack).
Would you please describe the details about how you use Livy and what
configurations you set?

Thanks
Jerry

2018-02-23 6:30 GMT+08:00 David Espinosa :

> Hi all,
>
> I'm a new user in Livy. I've created a scala app that runs well with a
> Spark 2.2.1. The version of scala used is 2.11.12..
> I have tried to create a session in Livy 0.5.0 using this jar, and once
> the Spark Session is created an error raises and the session goes dead.
>
> This is a part of the error Stak:
>
> "18/02/22 17:00:23 INFO SparkEntries: Spark context finished
> initialization in 2282ms",
> "18/02/22 17:00:24 INFO SparkEntries: Created Spark session.",
> "Exception in thread \"main\" scala.reflect.internal.FatalError:
> object Predef does not have a member classOf",
> "\tat scala.reflect.internal.Definitions$DefinitionsClass.scala$
> reflect$internal$Definitions$DefinitionsClass$$fatalMissing
> Symbol(Definitions.scala:1186)",
> "\tat scala.reflect.internal.Definitions$DefinitionsClass.getMembe
> r(Definitions.scala:1203)",
> "\tat scala.reflect.internal.Definitions$DefinitionsClass.getMembe
> rMethod(Definitions.scala:1238)",
>
> Has somebody found and fixed a problem like this?
>
> Thanks in advance,
> David
>
>
>

Re: Is multiple languages support in 0.5.0 only work with rest-api?

2018-02-06 Thread Saisai Shao 
Sorry I don't fully understand your saying. I think you can use Scala/Java
API to do data transformation and register as a table, and then use Python
API to do ML pipeline, not sure what is your specific scenario.

Thanks
Jerry

2018-02-06 18:09 GMT+08:00 simsicon <simsi...@gmail.com>:

> Thanks Jerry, please let me give an example like, if I am going to submit
> an application that using Scala API, but I would like to prepend a
> customization transformation function using Python at the beginning of the
> application, what should I do exactly? And is SparkSession in SparkContext
>
> <https://livy.incubator.apache.org/docs/latest/api/scala/index.html#org.apache.livy.scalaapi.ScalaJobContext>refers
> to the same concept of session in REST api?
>
> On Tue, Feb 6, 2018 at 3:22 PM, Saisai Shao <sai.sai.s...@gmail.com>
> wrote:
>
>> IIUC, I assume you're using JobAPI rather than REST APIs. Yes, you can
>> create a session using Java/Scala job API, and then manipulate with Python
>> APIs.
>>
>> Thanks
>> Jerry
>>
>> 2018-02-06 13:43 GMT+08:00 simsicon <simsi...@gmail.com>:
>>
>>> Hi,
>>>
>>> I am pretty new to livy, and I am looking into the livy to do some
>>> research if livy is capable as infrastructure of a data mining platform,
>>> which means we may want to pipe different language operators together, so I
>>> am excited to see the multiple languages support in the latest release
>>> 0.5.0.
>>>
>>> I wonder if we can use scala api to create a session, then post
>>> different language statement to that session? Does this make any sense?
>>>
>>> Thanks
>>> Z
>>>
>>> --
>>>
>>> Thanks Regards
>>>
>>> Zhang Zheng
>>>
>>
>>
>
>
> --
>
> Thanks Regards
>
> Zhang Zheng
>

Re: Is multiple languages support in 0.5.0 only work with rest-api?

2018-02-05 Thread Saisai Shao 
IIUC, I assume you're using JobAPI rather than REST APIs. Yes, you can
create a session using Java/Scala job API, and then manipulate with Python
APIs.

Thanks
Jerry

2018-02-06 13:43 GMT+08:00 simsicon :

> Hi,
>
> I am pretty new to livy, and I am looking into the livy to do some
> research if livy is capable as infrastructure of a data mining platform,
> which means we may want to pipe different language operators together, so I
> am excited to see the multiple languages support in the latest release
> 0.5.0.
>
> I wonder if we can use scala api to create a session, then post different
> language statement to that session? Does this make any sense?
>
> Thanks
> Z
>
> --
>
> Thanks Regards
>
> Zhang Zheng
>

Re: In POST /sessions, what's the difference between "jars" and "spark.jar" in conf ?

2018-01-28 Thread Saisai Shao 
I think they are basically the same, you can use either. Why don't you take
a try.

Thanks
Jerry

2018-01-29 12:37 GMT+08:00 Kyunam Kim :

> Basically, which option is equivalent to "spark-submit --jar
> myLib0,jar,myLib1.jar" ?
>
> Thanks,
> Q
>

Re: How to set driverMemory, driverCores, executorMemory using livy?

2017-12-06 Thread Saisai Shao 
Using this API "public LivyClientBuilder setConf(String key, String
value)"  to set Spark configurations you wanted.

2017-12-06 15:34 GMT+08:00 kant kodali :

> Hi All,
>
>
> I do see POST /sessions API where I can pass driverMemory, driverCores,
> executorMemory as part of the request body but I am using programmatic API
> to submit upload the Jar and submit my job so how do I set values
> for driverMemory, driverCores, executorMemory ?
>
> Thanks!
>
>

Re: Does Apache Livy support Spark Structured Streaming 2.2.0?

2017-11-28 Thread Saisai Shao 
Livy doesn't support WebSocket. I think for your scenario, you need to use
Livy's JobAPI instead of interactive query, Livy doesn't push results back
to client in real time. So this maybe slightly different from what you want
(IIUC).

2017-11-29 14:34 GMT+08:00 kant kodali <kanth...@gmail.com>:

> Nice! so if I submit a streaming query over REST can I get the results
> back over REST or Websocket?
>
> On Tue, Nov 28, 2017 at 10:29 PM, Saisai Shao <sai.sai.s...@gmail.com>
> wrote:
>
>> Livy doesn't add any restriction on how user uses Spark API, so of course
>> Structured Streaming is supported.
>>
>> 2017-11-29 14:21 GMT+08:00 kant kodali <kanth...@gmail.com>:
>>
>>> Hi All,
>>>
>>> Does Apache Livy support Spark Structured Streaming 2.2.0? If so, any
>>> examples please? preferably in Java.
>>>
>>> Thanks,
>>> kant
>>>
>>
>>
>

Re: How to access Python script's stdout when running in Yarn cluster mode?

2017-11-27 Thread Saisai Shao 
I think you have plenty of ways to get application log, either via command
line or programmatically, even with yarn RM UI. Since there already has
several ways to get application log, so currently we don't have a plan to
address this.

2017-11-27 20:04 GMT+08:00 Partridge, Lucas (GE Aviation) <
lucas.partri...@ge.com>:

> Thanks Jerry.
>
>
>
> “Currently there's no Livy api for you to get application log via REST
> API.”
>
> - That’s a real shame. Are there any plans to address this?  Because this
> really limits the usefulness of Livy when I’m using Yarn cluster mode.  It
> seems I have to make a choice between running my client in a scalable
> manner (yarn cluster) or being able to see the logs programmatically (yarn
> client), but not both.  My client is multi-threaded; I don’t want it to
> have to host multiple concurrent Spark driver applications but it looks
> like I might have no choice about this.
>
>
>
> Thanks, Lucas.
>
>
>
> *From:* Saisai Shao [mailto:sai.sai.s...@gmail.com]
> *Sent:* 27 November 2017 02:19
> *To:* user@livy.incubator.apache.org
> *Subject:* EXT: Re: How to access Python script's stdout when running in
> Yarn cluster mode?
>
>
>
> Since you're running with yarn cluster mode, the output from your python
> script should be part of your yarn application log. you can get it via yarn
> command like yarn log -applicationId , or others like Yarn
> UI. Currently there's no Livy api for you to get application log via REST
> API.
>
>
>
> Thanks
>
> Jerry
>
>
>
> 2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <
> lucas.partri...@ge.com>:
>
> Hi,
>
>
>
> I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines
> from a Python script I’m running on Spark in Yarn cluster mode.
> Unfortunately the stdout from my Python script is not included in the log
> lines returned by GET /batches/{batchId}/log!
>
>
>
> Is this by design, or an unfortunate by-product of running in Yarn cluster
> mode?
>
>
>
> If this is intentional does anyone know how I can access the stdout from
> my Python script via Livy please? Preferably without having to change my
> REST client (a Java app) to use Yarn’s client deployment mode.
>
>
>
> Thanks,
>
> Lucas.
>
>
>
>
>

Re: How to access Python script's stdout when running in Yarn cluster mode?

2017-11-26 Thread Saisai Shao 
Since you're running with yarn cluster mode, the output from your python
script should be part of your yarn application log. you can get it via yarn
command like yarn log -applicationId , or others like Yarn
UI. Currently there's no Livy api for you to get application log via REST
API.

Thanks
Jerry

2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <
lucas.partri...@ge.com>:

> Hi,
>
>
>
> I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines
> from a Python script I’m running on Spark in Yarn cluster mode.
> Unfortunately the stdout from my Python script is not included in the log
> lines returned by GET /batches/{batchId}/log!
>
>
>
> Is this by design, or an unfortunate by-product of running in Yarn cluster
> mode?
>
>
>
> If this is intentional does anyone know how I can access the stdout from
> my Python script via Livy please? Preferably without having to change my
> REST client (a Java app) to use Yarn’s client deployment mode.
>
>
>
> Thanks,
>
> Lucas.
>
>
>

Re: ClassNotFoundException on job submit

2017-10-26 Thread Saisai Shao 
You can choose to set "livy.spark.master" to "local" and
"livy.spark.deploy-mode" to "client" to start Spark with local mode, in
such case YARN is not required.

Otherwise if you plan to run on YARN, you have to install Hadoop and
configure HADOOP_CONF_DIR in livy-env.sh.

On Thu, Oct 26, 2017 at 9:40 PM, Stefan Miklosovic 
wrote:

> Hi,
>
> I am running Livy server in connection with Spark without Hadoop. I am
> setting only SPARK_HOME and I am getting this in Livy UI logs after
> job submission.
>
> I am using pretty much standard configuration but
> livy.spark.deploy-mode = cluster
>
> Do I need to run with Hadoop installation as well and specify
> HADOOP_CONF_DIR?
>
> Is not it possible to run Livy with "plain" Spark without YARN?
>
> stderr:
> java.lang.ClassNotFoundException:
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:712)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Thanks!
>
> --
> Stefan Miklosovic
>

Re: NoSuchFileException: spark-internal - when creating interactive session on remote spark

2017-10-16 Thread Saisai Shao 
Would you please provide more information about how you create a Livy
session? As for now, Livy only supports spark on yarn and local mode
officially, we don't test on standalone cluster mode, so maybe there's some
issues in it.

On Mon, Oct 16, 2017 at 4:29 AM, Junaid Nasir  wrote:

> Hi everyone,
> I am using livy (livy-0.4.0-incubating-bin), and a remote spark
> (spark-2.2.0-bin-hadoop2.7) stand alone cluster. livy's setting is default
> except setting livy.spark.master = spark://10.128.1.1:6066 and
> livy.spark.deploy-mode = cluster (have tried with default deploy-mode but
> then it throws a different error)
> when i try to create an interactive spark session on remote cluster it
> creates the driver and immediately sends it to error state.
>
> livy console error
>
>   17/10/15 20:18:08 INFO LineBufferedStream: stdout: 17/10/15 
> 20:18:08 INFO RestSubmissionClient: Submitting a request to launch an 
> application in spark://10.128.1.1:6066.
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO 
> RestSubmissionClient: Submission successfully created as 
> driver-20171015201808-0002. Polling submission state...
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO 
> RestSubmissionClient: Submitting a request for the status of submission 
> driver-20171015201808-0002 in spark://10.128.1.1:6066.
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO 
> RestSubmissionClient: State of driver driver-20171015201808-0002 is now ERROR.
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO 
> RestSubmissionClient: Driver is running on worker 
> worker-20171015195836-10.128.1.1-38097 at 10.128.1.1:38097.
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 ERROR 
> RestSubmissionClient: Exception from the cluster:
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 
> java.nio.file.NoSuchFileException: spark-internal
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> java.nio.file.Files.copy(Files.java:1274)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:625)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> org.apache.spark.util.Utils$.copyFile(Utils.scala:596)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> org.apache.spark.util.Utils$.doFetchFile(Utils.scala:681)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> org.apache.spark.util.Utils$.fetchFile(Utils.scala:480)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:  
> org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO 
> RestSubmissionClient: Server responded with CreateSubmissionResponse:
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: {
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "action" : 
> "CreateSubmissionResponse",
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "message" : "Driver 
> successfully submitted as driver-20171015201808-0002",
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "serverSparkVersion" : 
> "2.2.0",
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "submissionId" : 
> "driver-20171015201808-0002",
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "success" : true
> 17/10/15 20:18:09 INFO LineBufferedStream: stdout: }
>
> [image: Mixmax]
>  Not using Mixmax yet?
> 
>
> can anyone please guide me how to resolve this?
>
> I have tried setting up spark locally and it works fine. so i guess
> problem is with uploading jar files to remote cluster. I am using GCE for
> both spark cluster and livy server but they have all ports (on internal
> network open)
>
> spark worker log show this too
>
>   17/10/15 20:18:08 INFO Worker: Asked to launch driver 
>

Re: Spark cassandra connector with livy

2017-10-09 Thread Saisai Shao 
Please set "spark.jars.packages" to the package you wanted in batch POST
protocol "conf" field.

Thanks
Jerry

On Tue, Oct 10, 2017 at 3:05 AM, Junaid Nasir  wrote:

> more info regarding the problem.
> when i pass {"kind": "pyspark","jars":["datastax:
> spark-cassandra-connector:2.0.1-s_2.11"]} via post to /sessions
> session status changes to dead. livy logs show this
>
>   17/10/09 18:45:15 WARN ContextLauncher: Child process 
> exited with code 1.
> 17/10/09 18:45:15 ERROR RSCClient: Failed to connect to context.
> java.io.IOException: Child process exited with code 1.
> at 
> com.cloudera.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:383)
> at 
> com.cloudera.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:432)
> at java.lang.Thread.run(Thread.java:748)
> 17/10/09 18:45:15 INFO RSCClient: Failing pending job 
> 115078db-7ec3-4a5c-b74e-e24d6d811413 due to shutdown.
> 17/10/09 18:45:15 WARN InteractiveSession: (Fail to get rsc 
> uri,java.util.concurrent.ExecutionException: java.io.IOException: Child 
> process exited with code 1.)
> 17/10/09 18:45:15 INFO InteractiveSession: Stopping InteractiveSession 1...
> 17/10/09 18:45:15 INFO InteractiveSession: Stopped InteractiveSession 1.
>
> [image: Mixmax]
>  Not using Mixmax yet?
> 
>
>
>
> On Mon, Oct 9, 2017 5:11 PM, Junaid Nasir jna...@an10.io wrote:
>
>> Hi,
>>
>> How can I include Cassandra-connect-package
>>  to Livy spark
>> session?
>>
>> I use the following spark-submit command
>>
>> /spark-submit --master spark://10.128.1.1:7077 --packages
>> datastax:spark-cassandra-connector:2.0.1-s_2.11 --conf
>> spark.cassandra.connection.host="10.128.1.1,10.128.1.2,10.128.1.3"
>>
>>
>> how can i translate it to livy's session request or add this to livy conf
>> file?
>>
>> Also how to provide list of string and Map of key=val as mentioned in livy's
>> documentation
>> 
>>
>> Regards,
>> Junaid Nasir
>>
>

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Saisai Shao 
If we're using session name, how do we guarantee the uniqueness of this
name?

Thanks
Jerry

On Tue, Sep 12, 2017 at 4:51 AM, Alex Bozarth  wrote:

> I would agree with Marcelo's comment the JIRA that this isn't a good
> feature for livy, but I'll take a look at your impl if you open a PR and
> see if it changes my mind.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* 
> *GitHub: **github.com/ajbozarth* 
>
>
> 505 Howard Street
> 
> San Francisco, CA 94105
> 
> United States
> 
>
>
>
> [image: Inactive hide details for Meisam Fathi ---09/11/2017 10:23:49
> AM---+ dev Is there any interest in adding this feature to Livy?]Meisam
> Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding
> this feature to Livy? I can send a PR
>
> From: Meisam Fathi 
> To: "user@livy.incubator.apache.org" , "
> d...@livy.incubator.apache.org" 
> Date: 09/11/2017 10:23 AM
> Subject: Re: user defined sessionId / URI for Livy sessions
> --
>
>
>
> + dev
> Is there any interest in adding this feature to Livy? I can send a PR
>
> Ideally, it would be helpful if we could mint a session ID with a PUT
> > request, something like PUT /sessions/foobar, where "foobar" is the newly
> > created sessionId.
> >
> > I suggest we make session names unique and nonnumeric values (to
> guarantee
> a session name does not clash with another session name or session ID).
>
> Design doc:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-
> 2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=
> xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
> JIRA ticket: https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-
> juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=
>
>
> Thanks,
> Meisam
>
>
>
>

Re: Input file as an argument og a Spark code

2017-07-24 Thread Saisai Shao 
I think you have to make this csv file accessible from Spark cluster,
putting to HDFS is one possible solution.

On Tue, Jul 25, 2017 at 1:26 AM, Joaquín Silva  wrote:

> Hello,
>
>
>
> I'm building a BASH program (using Curl)  that should run a Spark code
> remotely using Livy. But one of the code argument  is a CSV file, how can I
> make that spark reads this file?. The file is going to be in client side,
> not in the Spark cluster machines.
>
>
>
> Regards,
>
>
>
> Joaquín Silva
>
>
>

Re: Scala 2.11 and Hive

2017-07-03 Thread Saisai Shao 
Hi Mobin,

Livy supports Scala 2.11 as well as Spark 2.0+. Can you please elaborate
your problem?

Thanks
Jerry

On Sun, Jul 2, 2017 at 9:19 PM, Mobin Ranjbar 
wrote:

> Hi there,
>
>
> I have a problem around using Livy and Apache Spark(Scala 2.10). I have
> Apache Hive that does not support Spark with Scala 2.10. I have to use Hive
> with Spark with Scala 2.11 but Livy does not support it. How can I have
> them together without having two different versions of Spark?
>
>
> Thanks in Advance,
>
>
> Mobin
>