Re: Use existing SparkSession in POST/batches request

Interactive sessions also allow you to upload jars. Consider trying to transform your batch job, or wrapping your job, into interactive code by giving your batch job an entry point method. class JumboBatchRunner { public static runBatch( SparkContext sc, String startDate, String endDate, St

Re: about LIVY-424

While it's true LIVY-424 creates a session leak due to REPL leak in Scala it's not the only thing that can. I've run hundreds of simple scala commands and the leak is only mild/moderate. However, some scala commands can be really problematic. For instance import org.apache.spark.sql._ ru

Re: Impersonation Error

Vamsi, It may be helpful if you read the thread on user impersonation from Tuesday, July 10, 2018. Do you have hive sql enabled? What related configurations do you have set in livy.conf ? Are you using kerberos as well? From: srungarapu vamsi Reply-To: "user@livy.incubator.apache.org" Date

Re: How to tune Livy for fast queries

oming from. ____ From: Harsch, Tim Sent: Thursday, August 2, 2018 8:28:58 AM To: user@livy.incubator.apache.org Subject: Re: How to tune Livy for fast queries Thank you Saisai for your response. I did have a chance to investigate further and I should giv

Re: How to tune Livy for fast queries

_ Probably some network cost should also be counted in. There's no such configuration for tuning. If you find some performance issue, you can create a JIRA or even a patch to fix Livy. Harsch, Tim mailto:tim.har...@teradata.com>> 于2018年8月1日周三 上午8:04写道： I have a Li

Re: Security Questions

Hi Harun, Probably the best feature that is going to help you with Livy is controlling the bind interface. By default it is set to 0.0.0.0. If you set it to 127.0.0.1 then only apps calling livy by that IP will be allowed in.This will put much of the onus of protecting access to the ho

How to tune Livy for fast queries

I have a Livy application that I'm trying to tune as I'm seeing some performance issue when the queries are fast queries. I've wrapped my queries with a timer that logs the time taken. The spark code executed typically takes 50ms to 150ms. I'm querying Livy every 500ms looking for my respons

Re: Query: Local Path cannot read by Livy (Running with Docker)

Path cannot read by Livy (Running with Docker) [External Email] Yes, that is the exact same scenario I have, just instead of jar files I am using CSV files stored in my client machine's file system and no protocol is supporting to access those files. On Sat, Jul 28, 2018 at 3:42 AM, H

Re: Query: Local Path cannot read by Livy (Running with Docker)

When you say not able to use my files in Spark Session what do you mean? I think I’m seeing a similar problem. If I submit jars using a client machine that is not the same as the server (so file system is not shared) and I submit the jar it’s looks like it gets transmitted to the server but th

Re: Apache livy and ajax

Indeed. Both Jupyter notebooks can interface with Livy using spark magic https://github.com/jupyter-incubator/sparkmagic. Zeppelin can as well, but I am less versed in how to configure it. -- Tim Harsch Principal Software Engineer, Research & Development Teradata 20

Re: user impersonation in Livy not working

text to true in livy.conf Harsch, Tim mailto:tim.har...@teradata.com>>于2018年7月10日周二 下午11:53写道： Hi, I have been unable to get proxyUsers to work with Livy. I seem to have Kerberos configured correctly, as evidenced by the last beeline command. When I start a livy session the server

user impersonation in Livy not working

Hi, I have been unable to get proxyUsers to work with Livy. I seem to have Kerberos configured correctly, as evidenced by the last beeline command. When I start a livy session the server log will usually say something like: “owner: kylo, proxyUser: Some(dladmin)” so it seems the server should

Does Livy Support Basic Auth?

I thought I read somewhere that Livy supports basic auth, but don't see configurations for it. Does it?

Re: Query regarding livy session

If you can guarantee that the statements you are executing are semantically independent, you could create a pool of sessions and execute on any idle session. Not sure in terms of resources how efficient that would be, depends on the cluster configuration in part. From: Saisai Shao Reply-To:

Re: Spark scala examples

Hi Nirav, There are a few ways to use Livy: Shared sessions and Batch sessions are covered in the REST API docs (https://livy.incubator.apache.org/docs/latest/rest-api.html) and GetHue.com wrote some cool articles. I created a little bash framework for testing out some example code I found at

Re: new user to livy/spark, basic use case questions

Results for any given script, or script segments, sent to Livy are retrievable in a subsequent call to the statements endpoint. The results are available as long as the session and server are still alive. If you want a script to write results to a datasource, so they are more permanent, you co

%json magick of dataframe drops the datatype information

Hi, I'm trying to understand why, when I use %json magic, dataType is empty? As you can see in this gist: https://gist.github.com/harschware/a29dc3224d3dea6bb749790244d39d72 all structs in the schema come back with just "n

completion endpoint

Hi, I noticed this endpoint in the REST API documentation. POST /sessions/{sessionId}/completion I've experimented with it a bit, and having a little trouble getting the results to match my expectations. Can someone provide some clues on where I might find example code, or is there some UI