[jira] [Commented] (SPARK-9104) expose network layer memory usage in shuffle part

[ https://issues.apache.org/jira/browse/SPARK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125331#comment-16125331 ] Saisai Shao commented on SPARK-9104: I think it is quite useful to get the memory details of Netty, so

[jira] [Commented] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

[ https://issues.apache.org/jira/browse/SPARK-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125115#comment-16125115 ] Saisai Shao commented on SPARK-21714: - I noticed this issue before and tried to fix it, but the

[jira] [Commented] (SPARK-21660) Yarn ShuffleService failed to start when the chosen directory become read-only

[ https://issues.apache.org/jira/browse/SPARK-21660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121125#comment-16121125 ] Saisai Shao commented on SPARK-21660: - Will yarn NM handle this bad disk problem and return a good

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114291#comment-16114291 ] Saisai Shao commented on SPARK-21618: - Hi Steve, I'm not quite following your comments. You mean that

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113806#comment-16113806 ] Saisai Shao commented on SPARK-21618: - [~benmayne] If you try the master branch of Spark with

[jira] [Commented] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

[ https://issues.apache.org/jira/browse/SPARK-21570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112036#comment-16112036 ] Saisai Shao commented on SPARK-21570: - Sorry I'm not familiar with NFS/Lustre FS, does this kind of

[jira] [Commented] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

[ https://issues.apache.org/jira/browse/SPARK-21570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110617#comment-16110617 ] Saisai Shao commented on SPARK-21570: - This __spark_libs_xxx.zip is created by Spark on yarn to zip

[jira] [Commented] (SPARK-21521) History service requires user is in any group

[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099629#comment-16099629 ] Saisai Shao commented on SPARK-21521: - I I think we should have a special logics to treat special

[jira] [Commented] (SPARK-21521) History service requires user is in any group

[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099392#comment-16099392 ] Saisai Shao commented on SPARK-21521: - [~vanzin], I guess so, in the current logics of

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095447#comment-16095447 ] Saisai Shao commented on SPARK-21460: - I think this is basically a ListenerBus issue, not a dynamic

[jira] [Created] (SPARK-21475) Change the usage of FileInputStream/OutputStream to Files.newInput/OutputStream in the critical path

Saisai Shao created SPARK-21475: --- Summary: Change the usage of FileInputStream/OutputStream to Files.newInput/OutputStream in the critical path Key: SPARK-21475 URL:

[jira] [Commented] (SPARK-21407) Support spnego for ThriftServer thrift/http auth

[ https://issues.apache.org/jira/browse/SPARK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088151#comment-16088151 ] Saisai Shao commented on SPARK-21407: - Sorry Marcelo, haven't noticed your comment. Yes it is the

[jira] [Commented] (SPARK-21411) Failed to get new HDFS delegation tokens in AMCredentialRenewer

[ https://issues.apache.org/jira/browse/SPARK-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086777#comment-16086777 ] Saisai Shao commented on SPARK-21411: - This issue is introduced by SPARK-20434. > Failed to get new

[jira] [Created] (SPARK-21411) Failed to get new HDFS delegation tokens in AMCredentialRenewer

Saisai Shao created SPARK-21411: --- Summary: Failed to get new HDFS delegation tokens in AMCredentialRenewer Key: SPARK-21411 URL: https://issues.apache.org/jira/browse/SPARK-21411 Project: Spark

[jira] [Commented] (SPARK-21398) Data on Rest end point is not updating after first fetch

[ https://issues.apache.org/jira/browse/SPARK-21398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086637#comment-16086637 ] Saisai Shao commented on SPARK-21398: - Can you please elaborate more about the issue, which rest api

[jira] [Updated] (SPARK-21407) Support spnego for ThriftServer thrift/http auth

[ https://issues.apache.org/jira/browse/SPARK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21407: Description: Spark ThriftServer doesn't support spnego auth for thrift/http protocol, this mainly

[jira] [Updated] (SPARK-21407) Support spnego for ThriftServer thrift/http auth

[ https://issues.apache.org/jira/browse/SPARK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21407: Summary: Support spnego for ThriftServer thrift/http auth (was: Support spnego for ThriftServer

[jira] [Created] (SPARK-21407) Support spnego for ThriftServer http auth

Saisai Shao created SPARK-21407: --- Summary: Support spnego for ThriftServer http auth Key: SPARK-21407 URL: https://issues.apache.org/jira/browse/SPARK-21407 Project: Spark Issue Type:

[jira] [Updated] (SPARK-21377) Jars specified with --jars or --packages are not added into AM's system classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Summary: Jars specified with --jars or --packages are not added into AM's system classpath (was:

[jira] [Comment Edited] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084460#comment-16084460 ] Saisai Shao edited comment on SPARK-21376 at 7/12/17 6:31 PM: -- I'm

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084460#comment-16084460 ] Saisai Shao commented on SPARK-21376: - I'm referrring to o.a.s.deploy.yarn.Client this class, it will

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084270#comment-16084270 ] Saisai Shao commented on SPARK-21376: - Hi [~tgraves], it is the local yarn launcher process which

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Affects Version/s: (was: 2.1.0) 2.2.0 > Jars pulled from "--packages"

[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083081#comment-16083081 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 10:13 PM: --- My original

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083081#comment-16083081 ] Saisai Shao commented on SPARK-21377: - My original purpose is to add jars uploaded by distributed

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083060#comment-16083060 ] Saisai Shao commented on SPARK-21377: - Thanks [~vanzin] for your comment. Your comment is correct,

[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083060#comment-16083060 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 9:54 PM: -- Thanks

[jira] [Updated] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Description: In this issue we have a long running Spark application with secure HBase, which

[jira] [Updated] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Summary: Add a new configuration to extend AM classpath in yarn client mode (was: Jars pulled

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Description: STR: * Set below config in spark-default.conf {code}

[jira] [Commented] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082864#comment-16082864 ] Saisai Shao commented on SPARK-21377: - [~srowen] this is a separate issue to SPARK-21376. In this

[jira] [Commented] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082873#comment-16082873 ] Saisai Shao commented on SPARK-21377: - SPARK-21376 and here are both security issues, but the root

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Priority: Minor (was: Major) > Jars pulled from "--packages" are not added into AM classpath >

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Priority: Minor (was: Major) > Token is not renewed in yarn client process in cluster mode >

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Component/s: (was: Spark Core) YARN > Jars pulled from "--packages" are not

[jira] [Comment Edited] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082864#comment-16082864 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 7:58 PM: -- [~srowen]

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Affects Version/s: (was: 2.1.0) 2.2.0 2.1.1 >

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082857#comment-16082857 ] Saisai Shao commented on SPARK-21376: - I will work on this, thanks [~yeshavora]. > Token is not

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Component/s: (was: Spark Core) YARN > Token is not renewed in yarn client

[jira] [Comment Edited] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082508#comment-16082508 ] Saisai Shao edited comment on SPARK-14151 at 7/11/17 5:00 PM: -- Thanks

[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082508#comment-16082508 ] Saisai Shao commented on SPARK-14151: - Thanks [~jiangxb] for your response. I'm not sure if it

[jira] [Comment Edited] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079407#comment-16079407 ] Saisai Shao edited comment on SPARK-21346 at 7/9/17 1:29 AM: - Jars, files can

[jira] [Commented] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079407#comment-16079407 ] Saisai Shao commented on SPARK-21346: - Jars, files can be downloaded from HTTPS if you configured

[jira] [Comment Edited] (SPARK-21334) Fix metrics for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079389#comment-16079389 ] Saisai Shao edited comment on SPARK-21334 at 7/9/17 12:43 AM: -- Are you using

[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079389#comment-16079389 ] Saisai Shao commented on SPARK-21334: - Are you using external shuffle service with YARN? If so I

[jira] [Commented] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079385#comment-16079385 ] Saisai Shao commented on SPARK-21346: - SSL is still used for fetch remote resources from HTTPS. If

[jira] [Commented] (SPARK-21348) Spark history Server load too slow when eventlog is large

[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079381#comment-16079381 ] Saisai Shao commented on SPARK-21348: - SPARK-18085 is trying to address this issue. [~duyanghao] are

[jira] [Comment Edited] (SPARK-21080) Workaround for HDFS delegation token expiry broken with some Hadoop versions

[ https://issues.apache.org/jira/browse/SPARK-21080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058843#comment-16058843 ] Saisai Shao edited comment on SPARK-21080 at 6/22/17 6:39 AM: -- That PR

[jira] [Commented] (SPARK-21080) Workaround for HDFS delegation token expiry broken with some Hadoop versions

[ https://issues.apache.org/jira/browse/SPARK-21080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058843#comment-16058843 ] Saisai Shao commented on SPARK-21080: - That PR should be worked, we applied that one to our internal

[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task

[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050016#comment-16050016 ] Saisai Shao commented on SPARK-21082: - That's fine if the storage memory is not enough to cache all

[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task

[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049940#comment-16049940 ] Saisai Shao commented on SPARK-21082: - Fast node actually equals to idle node, since fast node

[jira] [Commented] (SPARK-21080) Workaround for HDFS delegation token expiry broken with some Hadoop versions

[ https://issues.apache.org/jira/browse/SPARK-21080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049920#comment-16049920 ] Saisai Shao commented on SPARK-21080: - Are you getting this issue in HDFS HA mode? If yes, then

[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task

[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049906#comment-16049906 ] Saisai Shao commented on SPARK-21082: - Is it due to fast node and slow node problem? Ideally if all

[jira] [Reopened] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reopened SPARK-19688: - The issue still exists in master branch, so reopen it. > Spark on Yarn Credentials File set to

[jira] [Closed] (SPARK-21008) Streaming applications read stale credentials file when recovering from checkpoint.

[ https://issues.apache.org/jira/browse/SPARK-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-21008. --- Resolution: Duplicate > Streaming applications read stale credentials file when recovering from >

[jira] [Comment Edited] (SPARK-21008) Streaming applications read stale credentials file when recovering from checkpoint.

[ https://issues.apache.org/jira/browse/SPARK-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042341#comment-16042341 ] Saisai Shao edited comment on SPARK-21008 at 6/8/17 7:19 AM: - This is

[jira] [Commented] (SPARK-21008) Streaming applications read stale credentials file when recovering from checkpoint.

[ https://issues.apache.org/jira/browse/SPARK-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042341#comment-16042341 ] Saisai Shao commented on SPARK-21008: - This is duplicated with SPARK-19588, can we please using that

[jira] [Updated] (SPARK-21013) Spark History Server does not show the logs of completed Yarn Jobs

[ https://issues.apache.org/jira/browse/SPARK-21013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21013: Priority: Minor (was: Major) > Spark History Server does not show the logs of completed Yarn Jobs

[jira] [Updated] (SPARK-21012) Support glob path for resources adding to Spark

[ https://issues.apache.org/jira/browse/SPARK-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21012: Description: Current "\-\-jars (spark.jars)", "\-\-files (spark.files)", "\-\-py-files

[jira] [Updated] (SPARK-21012) Support glob path for resources adding to Spark

[ https://issues.apache.org/jira/browse/SPARK-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21012: Issue Type: Improvement (was: Bug) > Support glob path for resources adding to Spark >

[jira] [Updated] (SPARK-21012) Support glob path for resources adding to Spark

[ https://issues.apache.org/jira/browse/SPARK-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21012: Priority: Minor (was: Major) > Support glob path for resources adding to Spark >

[jira] [Created] (SPARK-21012) Support glob path for resources adding to Spark

Saisai Shao created SPARK-21012: --- Summary: Support glob path for resources adding to Spark Key: SPARK-21012 URL: https://issues.apache.org/jira/browse/SPARK-21012 Project: Spark Issue Type:

[jira] [Updated] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13669: Description: Currently we are running into an issue with Yarn work preserving enabled + external

[jira] [Created] (SPARK-20996) Better handling AM reattempt based on exit code in yarn mode

Saisai Shao created SPARK-20996: --- Summary: Better handling AM reattempt based on exit code in yarn mode Key: SPARK-20996 URL: https://issues.apache.org/jira/browse/SPARK-20996 Project: Spark

[jira] [Updated] (SPARK-20996) Better handling AM reattempt based on exit code in yarn mode

[ https://issues.apache.org/jira/browse/SPARK-20996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20996: Description: Yarn provides max attempt configuration for applications running on it, application

[jira] [Commented] (SPARK-20943) Correct BypassMergeSortShuffleWriter's comment

[ https://issues.apache.org/jira/browse/SPARK-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037998#comment-16037998 ] Saisai Shao commented on SPARK-20943: - >From my point, it is not really necessary to just update the

[jira] [Commented] (SPARK-20982) Consider adding SSL support for Spark REST submission server and client

[ https://issues.apache.org/jira/browse/SPARK-20982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037994#comment-16037994 ] Saisai Shao commented on SPARK-20982: - Thanks a lot [~vanzin] for your response. Though this

[jira] [Created] (SPARK-20982) Consider adding SSL support for Spark REST submission server and client

Saisai Shao created SPARK-20982: --- Summary: Consider adding SSL support for Spark REST submission server and client Key: SPARK-20982 URL: https://issues.apache.org/jira/browse/SPARK-20982 Project: Spark

[jira] [Created] (SPARK-20981) Add --repositories equivalent configuration for Spark

Saisai Shao created SPARK-20981: --- Summary: Add --repositories equivalent configuration for Spark Key: SPARK-20981 URL: https://issues.apache.org/jira/browse/SPARK-20981 Project: Spark Issue

[jira] [Commented] (SPARK-20943) Correct BypassMergeSortShuffleWriter's comment

[ https://issues.apache.org/jira/browse/SPARK-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034296#comment-16034296 ] Saisai Shao commented on SPARK-20943: - I think the original purpose of comment is to say

[jira] [Commented] (SPARK-20943) Correct BypassMergeSortShuffleWriter's comment

[ https://issues.apache.org/jira/browse/SPARK-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033016#comment-16033016 ] Saisai Shao commented on SPARK-20943: - I don't think the previous comment is incorrect. In the

[jira] [Commented] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030710#comment-16030710 ] Saisai Shao commented on SPARK-20898: - [~tgraves], I addressed this issue in

[jira] [Commented] (SPARK-19569) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017066#comment-16017066 ] Saisai Shao commented on SPARK-19569: - [~ouyangxc.zte] In you above code you directly call

[jira] [Commented] (SPARK-20772) Add support for query parameters in redirects on Yarn

[ https://issues.apache.org/jira/browse/SPARK-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013431#comment-16013431 ] Saisai Shao commented on SPARK-20772: - I'm guessing if it is an issue of {{AmIpFilter}}, should be a

[jira] [Commented] (SPARK-20658) spark.yarn.am.attemptFailuresValidityInterval doesn't seem to have an effect

[ https://issues.apache.org/jira/browse/SPARK-20658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001917#comment-16001917 ] Saisai Shao commented on SPARK-20658: - It is mainly depends on YARN to measure the failure validity

[jira] [Created] (SPARK-20605) Deprecate not used AM and executor port configuration

Saisai Shao created SPARK-20605: --- Summary: Deprecate not used AM and executor port configuration Key: SPARK-20605 URL: https://issues.apache.org/jira/browse/SPARK-20605 Project: Spark Issue

[jira] [Commented] (SPARK-20421) Mark JobProgressListener (and related classes) as deprecated

[ https://issues.apache.org/jira/browse/SPARK-20421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996301#comment-15996301 ] Saisai Shao commented on SPARK-20421: - [~vanzin], what will you do to {{StorageStatus}}? I saw you

[jira] [Created] (SPARK-20517) Download link in history server UI is not correct

Saisai Shao created SPARK-20517: --- Summary: Download link in history server UI is not correct Key: SPARK-20517 URL: https://issues.apache.org/jira/browse/SPARK-20517 Project: Spark Issue Type:

[jira] [Updated] (SPARK-20517) Download link in history server UI is not correct

[ https://issues.apache.org/jira/browse/SPARK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20517: Description: The download link in history server UI is concatenated with: {code} Download

[jira] [Commented] (SPARK-19688) Spark on Yarn Credentials File set to different application directory

[ https://issues.apache.org/jira/browse/SPARK-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986546#comment-15986546 ] Saisai Shao commented on SPARK-19688: - Does this issue exist in the latest master code, from the PR

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978415#comment-15978415 ] Saisai Shao commented on SPARK-20426: - Currently I don't have a clear fix about this issue, I'm not

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978411#comment-15978411 ] Saisai Shao commented on SPARK-20426: - My simple doubt is that your large cluster / application

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978391#comment-15978391 ] Saisai Shao commented on SPARK-20426: - Is your yarn cluster a large cluster, and there're bunch of

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976319#comment-15976319 ] Saisai Shao commented on SPARK-20391: - I'm in favor of using new REST API to define memory related

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976298#comment-15976298 ] Saisai Shao commented on SPARK-20391: - bq. I assume managed memory here is spark.memory.fraction on

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976145#comment-15976145 ] Saisai Shao commented on SPARK-20391: - Thanks [~irashid] [~tgraves] for your comments. bq. with the

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974685#comment-15974685 ] Saisai Shao commented on SPARK-20391: - [~irashid], would be grateful to hear your suggestion. >

[jira] [Created] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

Saisai Shao created SPARK-20391: --- Summary: Properly rename the memory related fields in ExecutorSummary REST API Key: SPARK-20391 URL: https://issues.apache.org/jira/browse/SPARK-20391 Project: Spark

[jira] [Updated] (SPARK-20365) Not so accurate classpath format for AM and Containers

[ https://issues.apache.org/jira/browse/SPARK-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20365: Summary: Not so accurate classpath format for AM and Containers (was: Inaccurate classpath format

[jira] [Created] (SPARK-20365) Inaccurate classpath format for AM and Containers

Saisai Shao created SPARK-20365: --- Summary: Inaccurate classpath format for AM and Containers Key: SPARK-20365 URL: https://issues.apache.org/jira/browse/SPARK-20365 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos

[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968545#comment-15968545 ] Saisai Shao commented on SPARK-16742: - [~mgummelt], do you have a design doc of the kerberos support

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965411#comment-15965411 ] Saisai Shao commented on SPARK-20286: - bq. Maybe the best approach is to change from

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965306#comment-15965306 ] Saisai Shao commented on SPARK-20286: - So the fix is that if there's no RDD get persisted, executors

[jira] [Commented] (SPARK-20244) Incorrect input size in UI with pyspark

[ https://issues.apache.org/jira/browse/SPARK-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964422#comment-15964422 ] Saisai Shao commented on SPARK-20244: - This actually is not a UI problem, it is FileSystem thread

[jira] [Updated] (SPARK-20275) HistoryServer page shows incorrect complete date of inprogress apps

[ https://issues.apache.org/jira/browse/SPARK-20275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20275: Description: The HistoryServer's incomplete page shows in-progress application's completed date

[jira] [Updated] (SPARK-20275) HistoryServer page shows incorrect complete date of inprogress apps

[ https://issues.apache.org/jira/browse/SPARK-20275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20275: Attachment: screenshot-1.png > HistoryServer page shows incorrect complete date of inprogress apps

[jira] [Created] (SPARK-20275) HistoryServer page shows incorrect complete date of inprogress apps

Saisai Shao created SPARK-20275: --- Summary: HistoryServer page shows incorrect complete date of inprogress apps Key: SPARK-20275 URL: https://issues.apache.org/jira/browse/SPARK-20275 Project: Spark

[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos

[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962455#comment-15962455 ] Saisai Shao commented on SPARK-16742: - Hi [~mgummelt], I'm working on the design of SPARK-19143, by

[jira] [Updated] (SPARK-20239) Improve HistoryServer ACL mechanism

[ https://issues.apache.org/jira/browse/SPARK-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20239: Description: Current SHS (Spark History Server) two different ACLs. * ACL of base URL, it is

[jira] [Created] (SPARK-20239) Improve HistoryServer ACL mechanism

Saisai Shao created SPARK-20239: --- Summary: Improve HistoryServer ACL mechanism Key: SPARK-20239 URL: https://issues.apache.org/jira/browse/SPARK-20239 Project: Spark Issue Type: Bug

<    1   2   3   4   5   6   7   8   9   10   >