Re: What if Livy Server dies?

In the comments bellow, by batch I mean a Livy job that is not interactive. This is not to be confused with batch vs streaming jobs in Spark. A Livy batch job could be a Spark streaming or batch job. A Livy interactive job could be a Spark batch or streaming job as well: On Wed, May 29, 2019 at

Re: Using external libraries in request

Hi Argenis, What do you exactly mean by every "every request"? Do you mean every interactive session? Thanks, Meisam On Sat, May 4, 2019 at 11:46 AM Argenis Leon wrote: > Hi, > I want to use a library https://github.com/ironmussa/optimus in every > request, but I don't want to instantiate it

Re: Livy with Standalone Spark Master

Yes. Livy works with standalone Spark, except for the recovery features. The recovery features need Yarn. Thanks, Meisam On Sat, Apr 20, 2019, 2:48 PM Pat Ferrel wrote: > Does Livy work with a Standalone Spark Master? > >

Re: Accessing Detailed Livy Session Information (session name?)

/v1 by default. @dev mailing list: This behavior is not documented in livy.conf nor on the website. It might be a good idea to document it somewhere. Thanks, Meisam On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi wrote: > Hi Peter, > > Livy 0.6 has a new feature to give each session a name

Re: Accessing Detailed Livy Session Information (session name?)

Hi Peter, Livy 0.6 has a new feature to give each session a name: https://github.com/apache/incubator-livy/pull/48 Would this feature be useful in your usecase? Thanks, Meisam On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) wrote: > Greetings, > > > > I have a custom service that connects

Re: Run spark application in different clusters

Currently this feature is not available in Livy. The values of HADOOP_CONF_DIR and YARN_CONF_DIR are read and set when Livy starts and never change. Thanks, Meisam On Thu, Feb 21, 2019, 12:47 AM Praveen Muthusamy wrote: > Hi, > > Currently HADOOP_CONF_DIR and YARN_CONF_DIR are used to find out

Re: Retain livy batch session info longer

I propose the following configurations to support this feature: # How long to retain an inactive interactive session before cleaning it up livy.server.session.timeout # How long to retain an interactive session that ran successfully livy.server.session.success.retaintion # How long to retain an

Re: Some questions about RscDriver

> > > Within an interactive session, Livy communicate with Spark through RPC. > According to some architecture diagrams, LivyServer has a RscClient, and > Spark has a RscDriver. My understanding is that RscDriver is one of > components belonging to Spark, and RscDriver has existed before the Livy

Re: job execution logs

If you are using YARN the driver logs are always available at YARN resource manager. You can also see the logs from /sessions/{sessionid}/logs Thanks, Meisam On Sun, Jun 24, 2018 at 1:37 PM Abbass wrote: > Guys any idea about where Livy keeps job execution logs (Spark driver > logs) ? > > I

Re: What happens if Livy server crashes ? All the spark jobs are gone?

If you are running on cluster mode, the application should keep running on YRAN. On Tue, Mar 20, 2018 at 3:34 PM kant kodali <kanth...@gmail.com> wrote: > @Meisam Fathi I am running with yarn and zookeeper as a state store. I > spawned a job via livy that reads from kafka and wri

Re: When I submit a livy job am I running in a client mode or cluster mode ?

client mode and cluster mode are for YARN. if you are using Spark standalone, your application will run in "client" mode. Thanks, Meisam On Wed, Mar 7, 2018 at 9:41 PM kant kodali wrote: > livy.spark.master yarn ?? Do I need to have yarn ? can I use spark > standalone

Re: Livy Installation

On Mon, Feb 12, 2018 at 2:01 AM Vinod Kancharana wrote: > Is there a recommended cluster server (edge node or master node or cluster > node) for Livy Installation? Right now, I installed Livy on the master node > running Yarn Resource Manager. > > I am not the best

Re: How to submit the job via REST ?

You should compile and package PiJar before running this code snippet. It does not need to be a separate app/project. You can put the PiJob code right next to the code snippet to run it. MVN/sbt/gradle can create the jar for you and I assume there is a way to call them programmatically, but that

Re: Livy POST Sessions can not work with conf

I am curious to know if this is resolved yet. I see that in the original email you used "conf" : {"spark.dynamicAllocation.enabled":false ,"spark.shuffle.service.enabled":false}. But in the second email, you used "conf" : {"spark.dynamicAllocation.enabled":"false", "spark.

Re: Spark job via livy issue from a docker container

Hi Sarjeet! Is livy.rsc.rpc.server.address set in the conf/livy-client.conf? Or in conf/livy.conf if you are using an older version of Livy? Thanks, Meisam 2017-10-19 12:44:59,688 WARN [Driver] rsc.RSCConf: Your hostname, node1.lab > (*valid hostname*), resolves to a loopback address, but we

Re: user defined sessionId / URI for Livy sessions

> > > E.g. something like this may be useful in an active-active livy > configuration because its not clear how sequential numeric id's would work > in that context. Perhaps UUID's would also suffice for the active-active > setup. > For active-active, sequential numeric IDs could be generated by

Re: user defined sessionId / URI for Livy sessions

> If we're using session name, how do we guarantee the uniqueness of this > name? > If the requested session name already exist, Livy returns an error and does not create the session. Thanks, Meisam

Re: user defined sessionId / URI for Livy sessions

+ dev Is there any interest in adding this feature to Livy? I can send a PR Ideally, it would be helpful if we could mint a session ID with a PUT > request, something like PUT /sessions/foobar, where "foobar" is the newly > created sessionId. > > I suggest we make session names unique and