[jira] [Updated] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated HIVE-16391: --- Fix Version/s: 1.2.3 Affects Version/s: 1.2.2 Attachment: HIVE-16391.patch

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501667#comment-16501667 ] Saisai Shao commented on HIVE-16391: I see, thanks. Will upload the p

Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or a cluster global value ?

2018-06-05 Thread Saisai Shao
atch (if it is delayed), which will lead to unexpected results. thomas lavocat 于2018年6月5日周二 下午7:48写道: > > On 05/06/2018 13:44, Saisai Shao wrote: > > You need to read the code, this is an undocumented configuration. > > I'm on it right now, but, Spark is a big piece of so

Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or a cluster global value ?

2018-06-05 Thread Saisai Shao
. > > On 05/06/2018 11:24, Saisai Shao wrote: > > spark.streaming.concurrentJobs is a driver side internal configuration, > this means that how many streaming jobs can be submitted concurrently in > one batch. Usually this should not be configured by user, unless you're > fami

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501561#comment-16501561 ] Saisai Shao commented on HIVE-16391: Seems there's no permission for me to

[jira] [Comment Edited] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501561#comment-16501561 ] Saisai Shao edited comment on HIVE-16391 at 6/5/18 10:1

Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or a cluster global value ?

2018-06-05 Thread Saisai Shao
spark.streaming.concurrentJobs is a driver side internal configuration, this means that how many streaming jobs can be submitted concurrently in one batch. Usually this should not be configured by user, unless you're familiar with Spark Streaming internals, and know the implication of this configur

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501415#comment-16501415 ] Saisai Shao commented on HIVE-16391: I'm not sure if submitting a PR is a

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501285#comment-16501285 ] Saisai Shao commented on HIVE-16391: Hi [~joshrosen] I'm trying to make

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

2018-06-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501225#comment-16501225 ] Saisai Shao commented on SPARK-20202: - OK, for the 1st, I've alread

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

2018-06-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499707#comment-16499707 ] Saisai Shao commented on SPARK-20202: - What is our plan to to fix this issue,

Re: Query regarding livy session

2018-06-03 Thread Saisai Shao
This is by design that Livy doesn't support concurrent execution of different requests. Because Livy doesn't know if the subsequent requests are semantically dependent or not. Rajeshkumar R 于2018年6月3日周日 下午12:08写道: > Hello >> I have a livy session which is linked to a spark context. This spark >>

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-06-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497931#comment-16497931 ] Saisai Shao commented on SPARK-18673: - Thanks Steve, looking forward to your in

[jira] [Commented] (SPARK-24355) Improve Spark shuffle server responsiveness to non-ChunkFetch requests

2018-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497626#comment-16497626 ] Saisai Shao commented on SPARK-24355: - Do we have a test result before and a

[jira] [Commented] (SPARK-24448) File not found on the address SparkFiles.get returns on standalone cluster

2018-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497548#comment-16497548 ] Saisai Shao commented on SPARK-24448: - Does it only happen in standalone clu

[jira] [Assigned] (LIVY-475) Support of Hadoop CredentialProvider API for livy.keystore.password

2018-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-475: Assignee: Ivan Dzikovsky > Support of Hadoop CredentialProvider API for livy.keystore.passw

[jira] [Resolved] (LIVY-475) Support of Hadoop CredentialProvider API for livy.keystore.password

2018-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-475. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 99 [https://github.com

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496187#comment-16496187 ] Saisai Shao commented on SPARK-18673: - I created a PR for this issue [h

[jira] [Resolved] (SPARK-23991) data loss when allocateBlocksToBatch

2018-05-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23991. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-23991) data loss when allocateBlocksToBatch

2018-05-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23991: --- Assignee: Gabor Somogyi > data loss when allocateBlocksToBa

[jira] [Resolved] (LIVY-473) Minor refactor of integration test to remove some old codes

2018-05-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-473. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 97 [https://github.com

[jira] [Assigned] (LIVY-473) Minor refactor of integration test to remove some old codes

2018-05-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-473: Assignee: Saisai Shao > Minor refactor of integration test to remove some old co

[jira] [Created] (LIVY-473) Minor refactor of integration test to remove some old codes

2018-05-28 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-473: Summary: Minor refactor of integration test to remove some old codes Key: LIVY-473 URL: https://issues.apache.org/jira/browse/LIVY-473 Project: Livy Issue Type

Re: bug in livy when jar has been uploaded

2018-05-27 Thread Saisai Shao
Please create a JIRA and put the issues and steps to reproduce on the JIRA. Thanks Jerry Harsch, Tim 于2018年5月26日周六 上午3:49写道: > Hi, > I posted a question on stack overflow (see link below). Livy is throwing > an error for any spark code I try. The error comes after uploading a jar > to the sh

[jira] [Resolved] (LIVY-472) Improve the logs for fail-to-create session

2018-05-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-472. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 96

[jira] [Assigned] (LIVY-472) Improve the logs for fail-to-create session

2018-05-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-472: Assignee: Saisai Shao > Improve the logs for fail-to-create sess

[jira] [Created] (SPARK-24377) Make --py-files work in non pyspark application

2018-05-23 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-24377: --- Summary: Make --py-files work in non pyspark application Key: SPARK-24377 URL: https://issues.apache.org/jira/browse/SPARK-24377 Project: Spark Issue Type

[jira] [Commented] (LIVY-471) New session creation API set to support resource uploading

2018-05-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486896#comment-16486896 ] Saisai Shao commented on LIVY-471: -- {quote}Using file name provided by user is bad

[jira] [Created] (LIVY-472) Improve the logs for fail-to-create session

2018-05-22 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-472: Summary: Improve the logs for fail-to-create session Key: LIVY-472 URL: https://issues.apache.org/jira/browse/LIVY-472 Project: Livy Issue Type: Improvement

[jira] [Assigned] (LIVY-468) Reverse proxy support for webui

2018-05-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-468: Assignee: Michal Wcislo > Reverse proxy support for we

[jira] [Commented] (LIVY-468) Reverse proxy support for webui

2018-05-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486615#comment-16486615 ] Saisai Shao commented on LIVY-468: -- Ok, I will do it. > Reverse proxy support fo

[jira] [Commented] (LIVY-452) Differentiate FAILED and KILLED states

2018-05-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482107#comment-16482107 ] Saisai Shao commented on LIVY-452: -- Issue resolved by pull request 92 [https://github

[jira] [Resolved] (LIVY-452) Differentiate FAILED and KILLED states

2018-05-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-452. -- Resolution: Fixed Fix Version/s: 0.6.0 > Differentiate FAILED and KILLED sta

[jira] [Assigned] (LIVY-452) Differentiate FAILED and KILLED states

2018-05-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-452: Assignee: Alexey Romanenko > Differentiate FAILED and KILLED sta

[jira] [Commented] (LIVY-471) New session creation API set to support resource uploading

2018-05-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481379#comment-16481379 ] Saisai Shao commented on LIVY-471: -- Requesting a new session id doesn't in

[jira] [Commented] (LIVY-471) New session creation API set to support resource uploading

2018-05-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481375#comment-16481375 ] Saisai Shao commented on LIVY-471: -- I have a local PR for solution 2, which was sim

[jira] [Updated] (LIVY-471) New session creation API set to support resource uploading

2018-05-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-471: - Description: Already post in mail list. In our current API design to create interactive / batch session

[jira] [Updated] (LIVY-471) New session creation API set to support resource uploading

2018-05-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-471: - Description: Already post in mail list. In our current API design to create interactive / batch session

[jira] [Created] (LIVY-471) New session creation API set to support resource uploading

2018-05-17 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-471: Summary: New session creation API set to support resource uploading Key: LIVY-471 URL: https://issues.apache.org/jira/browse/LIVY-471 Project: Livy Issue Type

Re: [VOTE] Spark 2.3.1 (RC1)

2018-05-16 Thread Saisai Shao
+1, checked new py4j related changes. Marcelo Vanzin 于2018年5月17日周四 上午5:41写道: > This is actually in 2.3, jira is just missing the version. > > https://github.com/apache/spark/pull/20765 > > On Wed, May 16, 2018 at 2:14 PM, kant kodali wrote: > > I am not sure how SPARK-23406 is a new feature. si

[jira] [Commented] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

2018-05-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475480#comment-16475480 ] Saisai Shao commented on SPARK-24241: - Issue resolved by pull request 21290 h

[jira] [Resolved] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

2018-05-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24241. - Resolution: Fixed > Do not fail fast when dynamic resource allocation enabled with 0 execu

[jira] [Updated] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

2018-05-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24241: Fix Version/s: 2.4.0 > Do not fail fast when dynamic resource allocation enabled with 0 execu

[jira] [Assigned] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

2018-05-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24241: --- Assignee: Kent Yao > Do not fail fast when dynamic resource allocation enabled wit

[jira] [Resolved] (LIVY-431) Livy batch mode ignore proxyUser parameter

2018-05-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-431. -- Resolution: Not A Problem > Livy batch mode ignore proxyUser parame

[jira] [Commented] (LIVY-431) Livy batch mode ignore proxyUser parameter

2018-05-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475255#comment-16475255 ] Saisai Shao commented on LIVY-431: -- I think you should enable impersonation first in

[jira] [Commented] (LIVY-469) "shared" session kind is undocumented

2018-05-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475221#comment-16475221 ] Saisai Shao commented on LIVY-469: -- It is by intention. Starting from 0.5 session kin

About new API set to create interactive and batch session

2018-05-14 Thread Saisai Shao
Hi Team, In our current API design to create interactive / batch session, we assume end user could upload jars, pyFiles and related dependencies to HDFS before creating the session, and we use one POST request to create session. But usually end user may not have the permission to access the HDFS i

[jira] [Assigned] (SPARK-24182) Improve error message for client mode when AM fails

2018-05-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24182: --- Assignee: Marcelo Vanzin > Improve error message for client mode when AM fa

[jira] [Resolved] (SPARK-24182) Improve error message for client mode when AM fails

2018-05-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24182. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21243 [https

Re: Livy kills spark streaming job after one hour

2018-05-10 Thread Saisai Shao
Which version of Livy are you using? This bug should be fixed long ago, I suggest you to try latest released version to see if this issue still exists. 姚德义 于2018年5月10日周四 下午5:43写道: > Hi, > I submit a spark streaming job by Livy, but after one hour Livy kills the > job. At that time, streaming job

[jira] [Created] (SPARK-24219) Improve the docker build script to avoid copying everything in example

2018-05-08 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-24219: --- Summary: Improve the docker build script to avoid copying everything in example Key: SPARK-24219 URL: https://issues.apache.org/jira/browse/SPARK-24219 Project: Spark

[jira] [Resolved] (SPARK-24188) /api/v1/version not working

2018-05-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24188. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24188) /api/v1/version not working

2018-05-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24188: --- Assignee: Marcelo Vanzin > /api/v1/version not work

[jira] [Commented] (SPARK-24174) Expose Hadoop config as part of /environment API

2018-05-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463575#comment-16463575 ] Saisai Shao commented on SPARK-24174: - I believe Hadoop web UI already expose

[jira] [Assigned] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available

2018-05-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24136: --- Assignee: Arun Mahadevan > MemoryStreamDataReader.next should skip sleeping if record

[jira] [Resolved] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available

2018-05-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24136. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21207 [https

[jira] [Commented] (LIVY-464) Cannot connect to Datastax standalone cluster DSE 5.1.5

2018-05-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463465#comment-16463465 ] Saisai Shao commented on LIVY-464: -- For now Livy can only support local mode and yarn

Re: Globally Unique/ User defined session Ids

2018-05-03 Thread Saisai Shao
I think we already supported single node HA for Livy by using incremented session id? Can you please elaborate your scenario and advantages of using GUID? Thanks Jerry Alex Bozarth 于2018年5月4日周五 上午4:13写道: > This sounds similar to a previous email chain from a few months ago. I > think your answe

[jira] [Resolved] (LIVY-466) RSCDriver throws exception during RPC shutdown

2018-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-466. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 90

[jira] [Resolved] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

2018-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24110. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21178 [https

[jira] [Assigned] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

2018-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24110: --- Assignee: Saisai Shao > Avoid calling UGI loginUserFromKeytab in ThriftSer

Re: [Spark Streaming]: Does DStream workload run over Spark SQL engine?

2018-05-02 Thread Saisai Shao
No, the underlying of DStream is RDD, so it will not leverage any SparkSQL related feature. I think you should use Structured Streaming instead, which is based on SparkSQL. Khaled Zaouk 于2018年5月2日周三 下午4:51写道: > Hi, > > I have a question regarding the execution engine of Spark Streaming > (DStrea

[jira] [Created] (LIVY-466) RSCDriver throws exception during RPC shutdown

2018-05-02 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-466: Summary: RSCDriver throws exception during RPC shutdown Key: LIVY-466 URL: https://issues.apache.org/jira/browse/LIVY-466 Project: Livy Issue Type: Bug

[jira] [Resolved] (LIVY-461) Cannot upload Jar file using LivyClientBuilder in Scala

2018-05-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-461. -- Resolution: Cannot Reproduce > Cannot upload Jar file using LivyClientBuilder in Sc

[jira] [Commented] (LIVY-461) Cannot upload Jar file using LivyClientBuilder in Scala

2018-05-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460558#comment-16460558 ] Saisai Shao commented on LIVY-461: -- I just verified with PiApp example included in

[jira] [Commented] (LIVY-465) Client closed before SASL negotiation finished

2018-05-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460370#comment-16460370 ] Saisai Shao commented on LIVY-465: -- # RSC configuration should be set through ses

[jira] [Updated] (LIVY-465) Client closed before SASL negotiation finished

2018-05-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-465: - Priority: Minor (was: Blocker) > Client closed before SASL negotiation finis

[jira] [Updated] (LIVY-465) Client closed before SASL negotiation finished

2018-05-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-465: - Issue Type: Question (was: Bug) > Client closed before SASL negotiation finis

[jira] [Commented] (LIVY-463) Port for accessing livy server

2018-05-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460333#comment-16460333 ] Saisai Shao commented on LIVY-463: -- Such information is already logged on the log

[jira] [Updated] (LIVY-463) Port for accessing livy server

2018-05-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-463: - Fix Version/s: (was: 0.5.1) > Port for accessing livy ser

[jira] [Assigned] (SPARK-23688) Refactor tests away from rate source

2018-04-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23688: --- Assignee: Jungtaek Lim > Refactor tests away from rate sou

[jira] [Resolved] (SPARK-23688) Refactor tests away from rate source

2018-04-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23688. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21152 [https

[jira] [Assigned] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-04-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23830: --- Assignee: Eric Maynard > Spark on YARN in cluster deploy mode fail w

[jira] [Resolved] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-04-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23830. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21168 [https

[jira] [Created] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

2018-04-27 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-24110: --- Summary: Avoid calling UGI loginUserFromKeytab in ThriftServer Key: SPARK-24110 URL: https://issues.apache.org/jira/browse/SPARK-24110 Project: Spark Issue

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-04-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455621#comment-16455621 ] Saisai Shao commented on SPARK-23830: - I agree [~emaynard]. > Spark on

[jira] [Commented] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

2018-04-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453521#comment-16453521 ] Saisai Shao commented on SPARK-24062: - Issue resolved by pull request 21138 [h

[jira] [Resolved] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

2018-04-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24062. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.4.0

[jira] [Assigned] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

2018-04-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-460: Assignee: Saisai Shao > when using livy with spark2.3, something wrong with the stdout and std

[jira] [Resolved] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

2018-04-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-460. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 89 [https://github.com

[jira] [Updated] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

2018-04-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-460: - Priority: Trivial (was: Major) > when using livy with spark2.3, something wrong with the stdout and std

[jira] [Updated] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

2018-04-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-460: - Issue Type: Improvement (was: Bug) > when using livy with spark2.3, something wrong with the stdout

[jira] [Commented] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

2018-04-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451522#comment-16451522 ] Saisai Shao commented on LIVY-460: -- I think the word "stdout" is misleadin

[jira] [Created] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

2018-04-24 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-24062: --- Summary: SASL encryption cannot be worked in ThriftServer Key: SPARK-24062 URL: https://issues.apache.org/jira/browse/SPARK-24062 Project: Spark Issue Type

[jira] [Assigned] (SPARK-22372) Make YARN client extend SparkApplication

2018-04-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22372: --- Assignee: Marcelo Vanzin > Make YARN client extend SparkApplicat

[jira] [Commented] (SPARK-24016) Yarn does not update node blacklist in static allocation

2018-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445252#comment-16445252 ] Saisai Shao commented on SPARK-24016: - I think this can be useful if we ena

[jira] [Commented] (LIVY-459) First submission via livy-python-api fails with KryoException, subsequent retries succeed

2018-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445239#comment-16445239 ] Saisai Shao commented on LIVY-459: -- Thanks for the report. Are you going to fix this i

[jira] [Commented] (LIVY-424) Session memory keeps growing as session persists

2018-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443705#comment-16443705 ] Saisai Shao commented on LIVY-424: -- This seems like a Scala issue, and looks like no

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443495#comment-16443495 ] Saisai Shao commented on SPARK-24008: - The case you provided seems not so v

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443475#comment-16443475 ] Saisai Shao commented on SPARK-24008: - Why do you need to broadcast {{SQLCon

[jira] [Resolved] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24014. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24014) Add onStreamingStarted method to StreamingListener

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24014: --- Assignee: Liang-Chi Hsieh > Add onStreamingStarted method to StreamingListe

[jira] [Resolved] (LIVY-458) Upgrade jackson version from 2.9.2 to 2.9.5

2018-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-458. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 87 [https://github.com

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441957#comment-16441957 ] Saisai Shao commented on SPARK-23989: - I've no idea what are you trying t

[jira] [Commented] (SPARK-23843) Deploy yarn meets incorrect LOCALIZED_CONF_DIR

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441930#comment-16441930 ] Saisai Shao commented on SPARK-23843: - I think this issue is due to your "n

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441919#comment-16441919 ] Saisai Shao commented on SPARK-23830: - What is the reason to use {{class}} ins

[jira] [Commented] (SPARK-24001) Multinode cluster

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441918#comment-16441918 ] Saisai Shao commented on SPARK-24001: - Question should go to mail list. > Mu

[jira] [Resolved] (SPARK-24001) Multinode cluster

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24001. - Resolution: Invalid > Multinode cluster > -- > >

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441906#comment-16441906 ] Saisai Shao commented on SPARK-23989: - Please provide a reproducible case. Did

<    1   2   3   4   5   6   7   8   9   10   >