[jira] [Resolved] (LIVY-409) Improve User Experience in livy-shell

[ https://issues.apache.org/jira/browse/LIVY-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-409. -- Resolution: Fixed Fix Version/s: 0.5.0 Issue resolved by pull request 55 [https://github.com

[jira] [Resolved] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22290. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19509 [https

[jira] [Assigned] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22290: --- Assignee: Marcelo Vanzin > Starting second context in same JVM fails to get new H

[jira] [Commented] (AMBARI-22260) Update Spark2 log4j default settings to latest

[ https://issues.apache.org/jira/browse/AMBARI-22260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210628#comment-16210628 ] Saisai Shao commented on AMBARI-22260: -- Please help to review [~sumitmohanty] [~jluniya], thanks

Review Request 63138: Update Spark2 log4j default settings to latest

/diff/1/ Testing --- Manually verification on refresh install of HDP 2.6 and upgrade from HDP 2.5. Thanks, Saisai Shao

[jira] [Created] (AMBARI-22260) Update Spark2 log4j default settings to latest

Saisai Shao created AMBARI-22260: Summary: Update Spark2 log4j default settings to latest Key: AMBARI-22260 URL: https://issues.apache.org/jira/browse/AMBARI-22260 Project: Ambari Issue Type

[jira] [Updated] (LIVY-411) Session cannot be started when Python or R package is missing

[ https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-411: - Reporter: Yesha Vora (was: Saisai Shao) > Session cannot be started when Python or R package is miss

[jira] [Updated] (LIVY-411) Session cannot be started when Python or R package is missing

[ https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-411: - Description: In Livy 0.5.0, we supported multiple languages in one session, but it requires

[jira] [Created] (LIVY-411) Session cannot be started when Python or R package is missing

Saisai Shao created LIVY-411: Summary: Session cannot be started when Python or R package is missing Key: LIVY-411 URL: https://issues.apache.org/jira/browse/LIVY-411 Project: Livy Issue Type

Re: NoSuchFileException: spark-internal - when creating interactive session on remote spark

Would you please provide more information about how you create a Livy session? As for now, Livy only supports spark on yarn and local mode officially, we don't test on standalone cluster mode, so maybe there's some issues in it. On Mon, Oct 16, 2017 at 4:29 AM, Junaid Nasir

[jira] [Commented] (SPARK-22242) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205018#comment-16205018 ] Saisai Shao commented on SPARK-22242: - It is not resolved, but it is the same type of problem

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202900#comment-16202900 ] Saisai Shao commented on SPARK-9: - {quote} I don't think that limited familiarity with a new

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201950#comment-16201950 ] Saisai Shao commented on SPARK-9: - My concern is about how to maintain this code in the community

[jira] [Commented] (SPARK-22062) BlockManager does not account for memory consumed by remote fetches

[ https://issues.apache.org/jira/browse/SPARK-22062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201305#comment-16201305 ] Saisai Shao commented on SPARK-22062: - Yes, there potentially has OOM problem, but I think this kind

[jira] [Commented] (SPARK-22243) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200372#comment-16200372 ] Saisai Shao commented on SPARK-22243: - Yes, it is a related issue regarding to Spark Streaming

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198674#comment-16198674 ] Saisai Shao commented on SPARK-21737: - I was trying to understand how Spark communicate with Mesos

[jira] [Assigned] (LIVY-7) Add autocompletion API

[ https://issues.apache.org/jira/browse/LIVY-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-7: -- Assignee: Pascal Pellmont > Add autocompletion API > -- > >

[jira] [Commented] (SPARK-22199) Spark Job on YARN fails with executors "Slave registration failed"

[ https://issues.apache.org/jira/browse/SPARK-22199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198258#comment-16198258 ] Saisai Shao commented on SPARK-22199: - Can you please list the steps to reproduce this issue? Also

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198204#comment-16198204 ] Saisai Shao commented on SPARK-21737: - Hi [~tgraves], I'm trying to understand the design

Re: Spark cassandra connector with livy

Please set "spark.jars.packages" to the package you wanted in batch POST protocol "conf" field. Thanks Jerry On Tue, Oct 10, 2017 at 3:05 AM, Junaid Nasir wrote: > more info regarding the problem. > when i pass {"kind": "pyspark","jars":["datastax: >

[jira] [Assigned] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22074: --- Assignee: Li Yuanjian > Task killed by other attempt task should not be resubmit

[jira] [Resolved] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22074. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19287 [https

[jira] [Assigned] (SPARK-22135) metrics in spark-dispatcher not being registered properly

[ https://issues.apache.org/jira/browse/SPARK-22135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22135: --- Assignee: paul mackles > metrics in spark-dispatcher not being registered prope

[jira] [Resolved] (SPARK-22135) metrics in spark-dispatcher not being registered properly

[ https://issues.apache.org/jira/browse/SPARK-22135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22135. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull

[jira] [Updated] (LIVY-408) Upgrade Netty version to avoid some security issues

[ https://issues.apache.org/jira/browse/LIVY-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-408: - Fix Version/s: (was: 0.5.0) > Upgrade Netty version to avoid some security iss

[jira] [Updated] (LIVY-408) Upgrade Netty version to avoid some security issues

[ https://issues.apache.org/jira/browse/LIVY-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-408: - Priority: Minor (was: Major) > Upgrade Netty version to avoid some security iss

[jira] [Created] (LIVY-408) Upgrade Netty version to avoid some security issues

Saisai Shao created LIVY-408: Summary: Upgrade Netty version to avoid some security issues Key: LIVY-408 URL: https://issues.apache.org/jira/browse/LIVY-408 Project: Livy Issue Type: Improvement

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183581#comment-16183581 ] Saisai Shao commented on SPARK-22151: - Checking the Yarn client code, looks like there's specific

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183563#comment-16183563 ] Saisai Shao commented on SPARK-21737: - [~yoonlee95], are you still working on this thing? I think

[jira] [Updated] (SPARK-22123) Add latest failure reason for task set blacklist

[ https://issues.apache.org/jira/browse/SPARK-22123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22123: Priority: Minor (was: Major) > Add latest failure reason for task set blackl

[jira] [Assigned] (SPARK-22123) Add latest failure reason for task set blacklist

[ https://issues.apache.org/jira/browse/SPARK-22123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22123: --- Assignee: zhoukang > Add latest failure reason for task set blackl

[jira] [Resolved] (SPARK-22123) Add latest failure reason for task set blacklist

[ https://issues.apache.org/jira/browse/SPARK-22123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22123. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19338 [https

[jira] [Resolved] (SPARK-20785) Spark should provide jump links and add (count) in the SQL web ui.

[ https://issues.apache.org/jira/browse/SPARK-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-20785. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19346 [https

[jira] [Assigned] (SPARK-20785) Spark should provide jump links and add (count) in the SQL web ui.

[ https://issues.apache.org/jira/browse/SPARK-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-20785: --- Assignee: guoxiaolongzte > Spark should provide jump links and add (count) in the SQL

[jira] [Commented] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182062#comment-16182062 ] Saisai Shao commented on SPARK-22074: - So if I understand correctly, this happens when speculation

[jira] [Commented] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181924#comment-16181924 ] Saisai Shao commented on SPARK-22074: - Hey [~XuanYuan], I'm a little confused why

[jira] [Commented] (SPARK-9103) Tracking spark's memory usage

[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181905#comment-16181905 ] Saisai Shao commented on SPARK-9103: Hi [~irashid], thanks a lot for your response. I agree that your

[jira] [Commented] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180506#comment-16180506 ] Saisai Shao commented on SPARK-22074: - Hi [~XuanYuan], can you please help me to understand your

[jira] [Closed] (LIVY-403) Add getCompletion for Interpreter

[ https://issues.apache.org/jira/browse/LIVY-403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed LIVY-403. Resolution: Duplicate > Add getCompletion for Interpre

[jira] [Commented] (SPARK-9103) Tracking spark's memory usage

[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176073#comment-16176073 ] Saisai Shao commented on SPARK-9103: [~irashid] I would like to hear your suggestion on displaying

[jira] [Assigned] (SPARK-21934) Expose Netty memory usage via Metrics System

[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21934: --- Assignee: Saisai Shao > Expose Netty memory usage via Metrics Sys

[jira] [Resolved] (SPARK-21934) Expose Netty memory usage via Metrics System

[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21934. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19160 [https

Re: How to get the test logs from travis

Zeppelin does, they kind cat a lot of the logs and > related files which then get appended to the build results. > > https://github.com/apache/zeppelin/blob/master/.travis.yml > > On Wed, Sep 20, 2017 at 8:22 PM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > > &

How to get the test logs from travis

Hi Team, Currently it is quite painful to figure out the cause of failure on travis test. Do you know how to get test logs from travis, is there a way supported by travis to either upload environment to some places, or log on to travis to dig the files. Previously we uploaded logs to azure when

[jira] [Updated] (LIVY-386) Refactor Livy core module with Java

[ https://issues.apache.org/jira/browse/LIVY-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-386: - Summary: Refactor Livy core module with Java (was: Refactor Livy code module with Java) > Refactor L

[jira] [Assigned] (SPARK-22030) GraphiteSink fails to re-connect to Graphite instances behind an ELB or any other auto-scaled LB

[ https://issues.apache.org/jira/browse/SPARK-22030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22030: --- Assignee: Alex Mikhailau > GraphiteSink fails to re-connect to Graphite instances beh

[jira] [Updated] (SPARK-22030) GraphiteSink fails to re-connect to Graphite instances behind an ELB or any other auto-scaled LB

[ https://issues.apache.org/jira/browse/SPARK-22030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22030: Priority: Minor (was: Critical) > GraphiteSink fails to re-connect to Graphite instances beh

[jira] [Resolved] (SPARK-22030) GraphiteSink fails to re-connect to Graphite instances behind an ELB or any other auto-scaled LB

[ https://issues.apache.org/jira/browse/SPARK-22030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22030. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19210 [https

[jira] [Assigned] (LIVY-396) Livy does not map YARN app states correctly

[ https://issues.apache.org/jira/browse/LIVY-396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-396: Assignee: Meisam Fathi > Livy does not map YARN app states correc

[jira] [Resolved] (LIVY-396) Livy does not map YARN app states correctly

[ https://issues.apache.org/jira/browse/LIVY-396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-396. -- Resolution: Fixed Issue resolved by pull request 39 [https://github.com/apache/incubator-livy/pull/39

[jira] [Updated] (LIVY-403) Add getCompletion for Interpreter

[ https://issues.apache.org/jira/browse/LIVY-403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-403: - Component/s: REPL > Add getCompletion for Interpre

[jira] [Updated] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21902: Priority: Trivial (was: Major) > BlockManager.doPut will hide actually exception when except

[jira] [Updated] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21902: Issue Type: Improvement (was: Wish) > BlockManager.doPut will hide actually exception w

[jira] [Assigned] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21902: --- Assignee: zhoukang > BlockManager.doPut will hide actually exception when exception thr

[jira] [Resolved] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21902. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19171 [https

[jira] [Resolved] (SPARK-21922) When executor failed and task metrics have not send to driver,the status will always be 'RUNNING' and the duration will be 'CurrentTime - launchTime'

[ https://issues.apache.org/jira/browse/SPARK-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21922. - Resolution: Fixed Assignee: zhoukang Target Version/s: 2.3.0 > W

[jira] [Updated] (SPARK-21922) When executor failed and task metrics have not send to driver,the status will always be 'RUNNING' and the duration will be 'CurrentTime - launchTime'

[ https://issues.apache.org/jira/browse/SPARK-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21922: Fix Version/s: 2.3.0 > When executor failed and task metrics have not send to driver,the sta

[jira] [Updated] (SPARK-21922) When executor failed and task metrics have not send to driver,the status will always be 'RUNNING' and the duration will be 'CurrentTime - launchTime'

[ https://issues.apache.org/jira/browse/SPARK-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21922: Target Version/s: (was: 2.3.0) > When executor failed and task metrics have not send to dri

[jira] [Commented] (SPARK-21513) SQL to_json should support all column types

[ https://issues.apache.org/jira/browse/SPARK-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164000#comment-16164000 ] Saisai Shao commented on SPARK-21513: - [~goldmedal] is it your correct JIRA name? [~hyukjin.kwon] I

[jira] [Assigned] (SPARK-21513) SQL to_json should support all column types

[ https://issues.apache.org/jira/browse/SPARK-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21513: --- Assignee: Jia-Xuan Liu > SQL to_json should support all column ty

Re: user defined sessionId / URI for Livy sessions

I see. So based on this, we should manage a data structure in Livy Server to keep all the live sessions' name. Also regarding to session recovery, we should persist this structure to the reliable storage and recover after restart. I'm not pretty sure if it is a good feature or not. First because

Re: user defined sessionId / URI for Livy sessions

If we're using session name, how do we guarantee the uniqueness of this name? Thanks Jerry On Tue, Sep 12, 2017 at 4:51 AM, Alex Bozarth wrote: > I would agree with Marcelo's comment the JIRA that this isn't a good > feature for livy, but I'll take a look at your impl if

Re: user defined sessionId / URI for Livy sessions

If we're using session name, how do we guarantee the uniqueness of this name? Thanks Jerry On Tue, Sep 12, 2017 at 4:51 AM, Alex Bozarth wrote: > I would agree with Marcelo's comment the JIRA that this isn't a good > feature for livy, but I'll take a look at your impl if

[jira] [Commented] (SPARK-21943) When I use rest Api (/ applications / [app-id] / jobs / [job-id]) to view some of the jobs that are running jobs, the returned json information is missing the “descrip

[ https://issues.apache.org/jira/browse/SPARK-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161008#comment-16161008 ] Saisai Shao commented on SPARK-21943: - If you're trying to report bugs, I think you should provide

[jira] [Commented] (SPARK-21943) When I use rest Api (/ applications / [app-id] / jobs / [job-id]) to view some of the jobs that are running jobs, the returned json information is missing the “descrip

[ https://issues.apache.org/jira/browse/SPARK-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160814#comment-16160814 ] Saisai Shao commented on SPARK-21943: - >From the code, it says that job description is gotten f

[jira] [Closed] (SPARK-21906) No need to runAsSparkUser to switch UserGroupInformation in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-21906. --- Resolution: Not A Problem > No need to runAsSparkUser to switch UserGroupInformation in YARN m

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

I guess you're using Capacity Scheduler with DefaultResourceCalculator, which doesn't count cpu cores into resource calculation, this "1" you saw is actually meaningless. If you want to also calculate cpu resource, you should choose DominantResourceCalculator. Thanks Jerry On Sat, Sep 9, 2017 at

[jira] [Assigned] (SPARK-20098) DataType's typeName method returns with 'StructF' in case of StructField

[ https://issues.apache.org/jira/browse/SPARK-20098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-20098: --- Assignee: Peter Szalai > DataType's typeName method returns with 'StructF' in c

[jira] [Commented] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158320#comment-16158320 ] Saisai Shao commented on SPARK-21942: - Personally I would like to fail fast if such things happened

[jira] [Commented] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158271#comment-16158271 ] Saisai Shao commented on SPARK-21942: - {quote} https://github.com/search?utf8=%E2%9C%93=filename

[jira] [Assigned] (SPARK-21939) Use TimeLimits instead of Timeouts

[ https://issues.apache.org/jira/browse/SPARK-21939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21939: --- Assignee: Dongjoon Hyun > Use TimeLimits instead of Timeo

[jira] [Resolved] (SPARK-21939) Use TimeLimits instead of Timeouts

[ https://issues.apache.org/jira/browse/SPARK-21939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21939. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19150 [https

[jira] [Commented] (SPARK-21942) ix DiskBlockManager crashing when a root local folder has been externally deleted by OS

[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156914#comment-16156914 ] Saisai Shao commented on SPARK-21942: - I think in a production environment, user should always

[jira] [Updated] (SPARK-9104) expose network layer memory usage

[ https://issues.apache.org/jira/browse/SPARK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-9104: --- Summary: expose network layer memory usage (was: expose network layer memory usage in shuffle part

[jira] [Updated] (SPARK-21934) Expose Netty memory usage via Metrics System

[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21934: Issue Type: Sub-task (was: New Feature) Parent: SPARK-9103 > Expose Netty memory us

[jira] [Created] (SPARK-21934) Expose Netty memory usage via Metrics System

Saisai Shao created SPARK-21934: --- Summary: Expose Netty memory usage via Metrics System Key: SPARK-21934 URL: https://issues.apache.org/jira/browse/SPARK-21934 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154744#comment-16154744 ] Saisai Shao commented on SPARK-21917: - Thanks [~vanzin], I think your suggestion is great

[jira] [Assigned] (SPARK-18061) Spark Thriftserver needs to create SPNego principal

[ https://issues.apache.org/jira/browse/SPARK-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-18061: --- Assignee: Saisai Shao > Spark Thriftserver needs to create SPNego princi

[jira] [Resolved] (SPARK-18061) Spark Thriftserver needs to create SPNego principal

[ https://issues.apache.org/jira/browse/SPARK-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-18061. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18628 [https

[jira] [Comment Edited] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153141#comment-16153141 ] Saisai Shao edited comment on SPARK-21917 at 9/5/17 8:36 AM: - I'm inclining

[jira] [Commented] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153141#comment-16153141 ] Saisai Shao commented on SPARK-21917: - I'm inclining to choose option 1, the only overhead

[jira] [Created] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

Saisai Shao created SPARK-21917: --- Summary: Remote http(s) resources is not supported in YARN mode Key: SPARK-21917 URL: https://issues.apache.org/jira/browse/SPARK-21917 Project: Spark Issue

[ANNOUNCE] Apache Livy 0.4.0-incubating released

The Apache Livy team is proud to announce Apache Livy version 0.4.0-incubating. This is the first Livy release after entering the Apache Incubator. Livy is web service that exposes a REST interface for managing long running Apache Spark contexts in your cluster. With Livy, new applications can

[ANNOUNCE] Apache Livy 0.4.0-incubating released

The Apache Livy team is proud to announce Apache Livy version 0.4.0-incubating. This is the first Livy release after entering the Apache Incubator. Livy is web service that exposes a REST interface for managing long running Apache Spark contexts in your cluster. With Livy, new applications can

Re: The process of Livy 0.4.0-incubating release

-- > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > [image: Inactive hide details for Saisai Shao ---09/0

Re: Port to open for submitting Spark on Yarn application

I think spark.yarn.am.port is not used any more, so you don't need to consider this. If you're running Spark on YARN, I think some YARN RM port to submit applications should also be reachable via firewall, as well as HDFS port to upload resources. Also in the Spark side, executors will be

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150258#comment-16150258 ] Saisai Shao commented on SPARK-21888: - Jars added by "--jars" will be added to client

[jira] [Commented] (SPARK-21882) OutputMetrics doesn't count written bytes correctly in the saveAsHadoopDataset function

[ https://issues.apache.org/jira/browse/SPARK-21882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150260#comment-16150260 ] Saisai Shao commented on SPARK-21882: - Please submit the patch to Github Apache Spark repo

Re: The process of Livy 0.4.0-incubating release

ajbozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > [image: Inactive hide details for Jeff Zhang ---08/30/2017 05:53:15 PM---I > think we'd better to announce after the artifacts are publis]Jeff Zhang > ---08/30/2017 05:5

[jira] [Assigned] (SPARK-21658) Adds the default None for value in na.replace in PySpark to match

[ https://issues.apache.org/jira/browse/SPARK-21658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21658: --- Assignee: Chin Han Yu > Adds the default None for value in na.replace in PySpark to ma

[jira] [Commented] (SPARK-21658) Adds the default None for value in na.replace in PySpark to match

[ https://issues.apache.org/jira/browse/SPARK-21658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150059#comment-16150059 ] Saisai Shao commented on SPARK-21658: - Done :). > Adds the default None for value in na.repl

Re: Moving Scala 2.12 forward one step

Hi Sean, Do we have a planned target version for Scala 2.12 support? Several other projects like Zeppelin, Livy which rely on Spark repl also require changes to support this Scala 2.12. Thanks Jerry On Thu, Aug 31, 2017 at 5:55 PM, Sean Owen wrote: > No, this doesn't let

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148713#comment-16148713 ] Saisai Shao commented on SPARK-11574: - Thanks a lot [~srowen]! > Spark should support StatsD s

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148701#comment-16148701 ] Saisai Shao commented on SPARK-11574: - Ping [~srowen]. Hi Sean I cannot assign this JIRA to Xiaofeng

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148333#comment-16148333 ] Saisai Shao commented on SPARK-11574: - Maybe it is my permission issue, will ask PMC to handle

[jira] [Assigned] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-17321: --- Assignee: Saisai Shao > YARN shuffle service should use good disk f

[jira] [Resolved] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-17321. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19032 [https

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148292#comment-16148292 ] Saisai Shao commented on SPARK-11574: - Hi Xiaofeng, is your JIRA username still available, I cannot

[jira] [Resolved] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-11574. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 9518 [https

Re: The process of Livy 0.4.0-incubating release

.ibm.com* <ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth> > > > 505 Howard Street > San Francisco, CA 94105 > United States > > > > [image: Inactive hide details for Saisai Shao ---08/30/2017 12:05:24 > AM---

<    3   4   5   6   7   8   9   10   11   12   >