[jira] [Commented] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567933#comment-15567933 ] Saisai Shao commented on SPARK-17814: - Which version of YARN do you use? > spark sub

[jira] [Commented] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567931#comment-15567931 ] Saisai Shao commented on SPARK-17814: - This looks like a YARN problem, after tracking

[jira] [Commented] (SPARK-17846) A bad state of Running Applications with spark standalone HA

[ https://issues.apache.org/jira/browse/SPARK-17846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567662#comment-15567662 ] Saisai Shao commented on SPARK-17846: - I think this issue should be the same as SPARK

[jira] [Created] (SPARK-17686) Propose to print Scala version in "spark-submit --version" command

Saisai Shao created SPARK-17686: --- Summary: Propose to print Scala version in "spark-submit --version" command Key: SPARK-17686 URL: https://issues.apache.org/jira/browse/SPARK-17686 Project: Spark

[jira] [Created] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

Saisai Shao created SPARK-17678: --- Summary: Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port" Key: SPARK-17678 URL: https://issues.apache.org/jira/browse/SPARK-17678 Project: Spark

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515836#comment-15515836 ] Saisai Shao commented on SPARK-17637: - [~zhanzhang] would you mind sharing more detai

[jira] [Comment Edited] (SPARK-17624) Flaky test? StateStoreSuite maintenance

[ https://issues.apache.org/jira/browse/SPARK-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512237#comment-15512237 ] Saisai Shao edited comment on SPARK-17624 at 9/22/16 5:36 AM: -

[jira] [Commented] (SPARK-17624) Flaky test? StateStoreSuite maintenance

[ https://issues.apache.org/jira/browse/SPARK-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512237#comment-15512237 ] Saisai Shao commented on SPARK-17624: - I cannot reproduce locally on my > Flaky tes

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17604: Issue Type: Sub-task (was: Improvement) Parent: SPARK-17267 > Support purging aged file en

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17604: Description: Currently with SPARK-15698, FileStreamSource metadata log will be compacted periodica

[jira] [Created] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

Saisai Shao created SPARK-17604: --- Summary: Support purging aged file entry for FileStreamSource metadata log Key: SPARK-17604 URL: https://issues.apache.org/jira/browse/SPARK-17604 Project: Spark

[jira] [Commented] (SPARK-15698) Ability to remove old metadata for structure streaming MetadataLog

[ https://issues.apache.org/jira/browse/SPARK-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505799#comment-15505799 ] Saisai Shao commented on SPARK-15698: - I think [~rxin] set this target version before

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500522#comment-15500522 ] Saisai Shao commented on SPARK-17566: - I've already submitted a PR under SPARK-17512,

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500511#comment-15500511 ] Saisai Shao commented on SPARK-17566: - Shouldn't it be {{!isYarnCluster}}? Since we n

[jira] [Updated] (SPARK-17512) Specifying remote files for Python based Spark jobs in Yarn cluster mode not working

[ https://issues.apache.org/jira/browse/SPARK-17512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17512: Component/s: YARN > Specifying remote files for Python based Spark jobs in Yarn cluster mode not >

[jira] [Commented] (SPARK-17512) Specifying remote files for Python based Spark jobs in Yarn cluster mode not working

[ https://issues.apache.org/jira/browse/SPARK-17512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500178#comment-15500178 ] Saisai Shao commented on SPARK-17512: - This is due to some behavior changes during su

[jira] [Closed] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-17566. --- Resolution: Duplicate > "--master yarn --deploy-mode cluster" gives "Launching Python applications >

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500145#comment-15500145 ] Saisai Shao commented on SPARK-17566: - Sorry I misunderstood your point, looks like i

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500109#comment-15500109 ] Saisai Shao commented on SPARK-17566: - Can you confirm the above command you mentione

[jira] [Comment Edited] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

[ https://issues.apache.org/jira/browse/SPARK-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495243#comment-15495243 ] Saisai Shao edited comment on SPARK-17522 at 9/16/16 3:19 AM: -

[jira] [Commented] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

[ https://issues.apache.org/jira/browse/SPARK-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495243#comment-15495243 ] Saisai Shao commented on SPARK-17522: - [~sunrui] I think the performance is depended

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470553#comment-15470553 ] Saisai Shao commented on SPARK-17340: - I think what [~asukhenko] mentioned in the des

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455080#comment-15455080 ] Saisai Shao commented on SPARK-17340: - yarn-client and yarn-cluster has different way

[jira] [Comment Edited] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454777#comment-15454777 ] Saisai Shao edited comment on SPARK-17340 at 9/1/16 11:02 AM: -

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455076#comment-15455076 ] Saisai Shao commented on SPARK-17340: - You can try not kill local {{yarn#client}} pro

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455055#comment-15455055 ] Saisai Shao commented on SPARK-17340: - I'm saying yarn cluster mode, I think here in

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454777#comment-15454777 ] Saisai Shao commented on SPARK-17340: - I think in your scenario, it is because you ki

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436222#comment-15436222 ] Saisai Shao commented on SPARK-17204: - Yes, I could reproduce this issue, but not con

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436213#comment-15436213 ] Saisai Shao commented on SPARK-17204: - I think to reflect the issue {{sc.range(0, 0)}

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436205#comment-15436205 ] Saisai Shao commented on SPARK-17204: - No, I tested in yarn cluster, not local mode.

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436179#comment-15436179 ] Saisai Shao commented on SPARK-17204: - It works OK in my local test with latest build

[jira] [Updated] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17209: Summary: Support manual credential updating in the run-time for Spark on YARN (was: Support manual

[jira] [Updated] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17209: Description: Current Spark on YARN supports time based credential renewal and updating, this mainl

[jira] [Created] (SPARK-17209) Support manual credential updating in the run-time

Saisai Shao created SPARK-17209: --- Summary: Support manual credential updating in the run-time Key: SPARK-17209 URL: https://issues.apache.org/jira/browse/SPARK-17209 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17148) NodeManager exit because of exception “Executor is not registered”

[ https://issues.apache.org/jira/browse/SPARK-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430402#comment-15430402 ] Saisai Shao commented on SPARK-17148: - I manually verified this by explicitly throwin

[jira] [Created] (SPARK-17019) Expose off-heap memory usage in various places

Saisai Shao created SPARK-17019: --- Summary: Expose off-heap memory usage in various places Key: SPARK-17019 URL: https://issues.apache.org/jira/browse/SPARK-17019 Project: Spark Issue Type: Impr

[jira] [Commented] (AMBARI-18091) Use https url for Spark2 Service check when WireEncryption is enabled

[ https://issues.apache.org/jira/browse/AMBARI-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414751#comment-15414751 ] Saisai Shao commented on AMBARI-18091: -- Please help to review, [~sumitmohanty] [~jl

[jira] [Assigned] (AMBARI-18091) Use https url for Spark2 Service check when WireEncryption is enabled

[ https://issues.apache.org/jira/browse/AMBARI-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned AMBARI-18091: Assignee: Saisai Shao > Use https url for Spark2 Service check when WireEncryption is ena

[jira] [Commented] (SPARK-16966) App Name is a randomUUID even when "spark.app.name" exists

[ https://issues.apache.org/jira/browse/SPARK-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414481#comment-15414481 ] Saisai Shao commented on SPARK-16966: - Here is the code in {{SparkSubmitArguments}} t

[jira] [Commented] (SPARK-16966) App Name is a randomUUID even when "spark.app.name" exists

[ https://issues.apache.org/jira/browse/SPARK-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413201#comment-15413201 ] Saisai Shao commented on SPARK-16966: - Yes, agreed. A better way is to handle this ap

[jira] [Commented] (SPARK-16944) [MESOS] Improve data locality when launching new executors when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411299#comment-15411299 ] Saisai Shao commented on SPARK-16944: - Does Mesos have the similar concept like Yarn

[jira] [Comment Edited] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411159#comment-15411159 ] Saisai Shao edited comment on SPARK-16914 at 8/8/16 1:48 AM: -

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411159#comment-15411159 ] Saisai Shao commented on SPARK-16914: - So from your description, is this exception ma

[jira] [Updated] (SPARK-16871) Support getting HBase tokens from multiple clusters dynamically

[ https://issues.apache.org/jira/browse/SPARK-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16871: Summary: Support getting HBase tokens from multiple clusters dynamically (was: Support getting HBa

[jira] [Created] (SPARK-16871) Support getting HBase tokens from multiple clusters and dynamically

Saisai Shao created SPARK-16871: --- Summary: Support getting HBase tokens from multiple clusters and dynamically Key: SPARK-16871 URL: https://issues.apache.org/jira/browse/SPARK-16871 Project: Spark

[jira] [Commented] (SPARK-16864) Comprehensive version info

[ https://issues.apache.org/jira/browse/SPARK-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405487#comment-15405487 ] Saisai Shao commented on SPARK-16864: - A program way to get spark version is to call

[jira] [Commented] (SPARK-14453) Remove SPARK_JAVA_OPTS environment variable

[ https://issues.apache.org/jira/browse/SPARK-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405112#comment-15405112 ] Saisai Shao commented on SPARK-14453: - If you want to fix this issue, it would be bet

[jira] [Comment Edited] (SPARK-16815) Dataset[List[T]] leads to ArrayStoreException

[ https://issues.apache.org/jira/browse/SPARK-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401566#comment-15401566 ] Saisai Shao edited comment on SPARK-16815 at 8/1/16 6:01 AM: -

[jira] [Commented] (SPARK-16815) Dataset[List[T]] leads to ArrayStoreException

[ https://issues.apache.org/jira/browse/SPARK-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401566#comment-15401566 ] Saisai Shao commented on SPARK-16815: - >From my understanding you can use {code} sp

[jira] [Commented] (SPARK-16817) Enable storing of shuffle data in Alluxio

[ https://issues.apache.org/jira/browse/SPARK-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401432#comment-15401432 ] Saisai Shao commented on SPARK-16817: - What's difference compared to use ramdisk to s

[jira] [Commented] (AMBARI-17954) Fix Spark hdp.version issues in upgrading and fresh install

[ https://issues.apache.org/jira/browse/AMBARI-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398740#comment-15398740 ] Saisai Shao commented on AMBARI-17954: -- CC [~sumitmohanty] [~jluniya], please help

[jira] [Created] (AMBARI-17954) Fix Spark hdp.version issues in upgrading and fresh install

Saisai Shao created AMBARI-17954: Summary: Fix Spark hdp.version issues in upgrading and fresh install Key: AMBARI-17954 URL: https://issues.apache.org/jira/browse/AMBARI-17954 Project: Ambari

[jira] [Commented] (SPARK-16085) Spark stand-alone ui redirects to RM application master UI for yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395705#comment-15395705 ] Saisai Shao commented on SPARK-16085: - Unfortunately, there's no such configuration f

[jira] [Commented] (SPARK-16708) ExecutorAllocationManager.numRunningTasks can be negative when stage retry

[ https://issues.apache.org/jira/browse/SPARK-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393377#comment-15393377 ] Saisai Shao commented on SPARK-16708: - Looks similar to SPARK-11334, and I have a pat

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393033#comment-15393033 ] Saisai Shao commented on SPARK-16723: - So maybe this application is not yet started i

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393003#comment-15393003 ] Saisai Shao commented on SPARK-16723: - Did you enable log aggregation in YARN, if not

[jira] [Comment Edited] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393003#comment-15393003 ] Saisai Shao edited comment on SPARK-16723 at 7/26/16 1:36 AM: -

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392980#comment-15392980 ] Saisai Shao commented on SPARK-16723: - {{yarn logs -applicationId application_1467990

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392965#comment-15392965 ] Saisai Shao commented on SPARK-16723: - I think you should check the AM and executor l

[jira] [Commented] (AMBARI-16864) Add unit tests for Spark2 service definition

[ https://issues.apache.org/jira/browse/AMBARI-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383435#comment-15383435 ] Saisai Shao commented on AMBARI-16864: -- Done, patch updated. > Add unit tests for

[jira] [Updated] (SPARK-16540) Jars specified with --jars will added twice when running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16540: Description: Currently when running spark on yarn, jars specified with \--jars, \--packages will b

[jira] [Created] (SPARK-16540) Jars specified with --jars will added twice when running on YARN

Saisai Shao created SPARK-16540: --- Summary: Jars specified with --jars will added twice when running on YARN Key: SPARK-16540 URL: https://issues.apache.org/jira/browse/SPARK-16540 Project: Spark

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376449#comment-15376449 ] Saisai Shao commented on SPARK-16534: - Maybe I can take a try if no one is working on

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376432#comment-15376432 ] Saisai Shao commented on SPARK-16534: - Is there anyone working on this? > Kafka 0.10

[jira] [Commented] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374612#comment-15374612 ] Saisai Shao commented on SPARK-16521: - I see, sorry about the duplication. > Add sup

[jira] [Closed] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16521. --- Resolution: Duplicate > Add support of parameterized configuration for SparkConf > --

[jira] [Commented] (SPARK-16522) [MESOS] Spark application throws exception on exit

[ https://issues.apache.org/jira/browse/SPARK-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374583#comment-15374583 ] Saisai Shao commented on SPARK-16522: - Perhaps there's race condition when exiting th

[jira] [Updated] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16521: Priority: Minor (was: Major) > Add support of parameterized configuration for SparkConf >

[jira] [Created] (SPARK-16521) Add support of parameterized configuration for SparkConf

Saisai Shao created SPARK-16521: --- Summary: Add support of parameterized configuration for SparkConf Key: SPARK-16521 URL: https://issues.apache.org/jira/browse/SPARK-16521 Project: Spark Issue

[jira] [Commented] (SPARK-16428) Spark file system watcher not working on Windows

[ https://issues.apache.org/jira/browse/SPARK-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372885#comment-15372885 ] Saisai Shao commented on SPARK-16428: - bq. Spark detected those files with the above

[jira] [Commented] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371975#comment-15371975 ] Saisai Shao commented on SPARK-16435: - OK, I will file a small patch to add the warni

[jira] [Created] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

Saisai Shao created SPARK-16435: --- Summary: Behavior changes if initialExecutor is less than minExecutor for dynamic allocation Key: SPARK-16435 URL: https://issues.apache.org/jira/browse/SPARK-16435 Pro

[jira] [Updated] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14743: Component/s: YARN > Improve delegation token handling in secure clusters >

[jira] [Closed] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16342. --- Resolution: Duplicate > Add a new Configurable Token Manager for Spark Running on YARN > ---

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365535#comment-15365535 ] Saisai Shao commented on SPARK-16342: - Close as JIRA as duplicated and move to SPARK-

[jira] [Commented] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365534#comment-15365534 ] Saisai Shao commented on SPARK-14743: - Post design doc here and move SPARK-16342 to h

[jira] [Comment Edited] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365534#comment-15365534 ] Saisai Shao edited comment on SPARK-14743 at 7/7/16 3:18 AM: -

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365483#comment-15365483 ] Saisai Shao commented on SPARK-16342: - OK, I see. Sorry I didn't notice your JIRA, le

[jira] [Issue Comment Deleted] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16342: Comment: was deleted (was: Thanks [~vanzin] for pointing out the jira, looks like most part of the

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365477#comment-15365477 ] Saisai Shao commented on SPARK-16342: - Thanks [~vanzin] for pointing out the jira, lo

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365478#comment-15365478 ] Saisai Shao commented on SPARK-16342: - Thanks [~vanzin] for pointing out the jira, lo

[jira] [Closed] (SPARK-16393) Move and implement token obtaining logics for built-in token providers

[ https://issues.apache.org/jira/browse/SPARK-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16393. --- Resolution: Duplicate > Move and implement token obtaining logics for built-in token providers >

[jira] [Closed] (SPARK-16392) Build ConfigurableTokenManager framework

[ https://issues.apache.org/jira/browse/SPARK-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16392. --- Resolution: Duplicate > Build ConfigurableTokenManager framework > --

[jira] [Created] (SPARK-16393) Move and implement token obtaining logics for built-in token providers

Saisai Shao created SPARK-16393: --- Summary: Move and implement token obtaining logics for built-in token providers Key: SPARK-16393 URL: https://issues.apache.org/jira/browse/SPARK-16393 Project: Spark

[jira] [Created] (SPARK-16392) Build ConfigurableTokenManager framework

Saisai Shao created SPARK-16392: --- Summary: Build ConfigurableTokenManager framework Key: SPARK-16392 URL: https://issues.apache.org/jira/browse/SPARK-16392 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-16382) YARN - Dynamic allocation with spark.executor.instances should increase max executors.

[ https://issues.apache.org/jira/browse/SPARK-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15363592#comment-15363592 ] Saisai Shao commented on SPARK-16382: - I would suggest to fail and complain. Max usua

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361917#comment-15361917 ] Saisai Shao commented on SPARK-16342: - [~tgraves] [~vanzin] [~ste...@apache.org] woul

[jira] [Updated] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16342: Description: Current Spark on YARN token management has some problems: 1. Supported service is har

[jira] [Commented] (SPARK-15923) Spark Application rest api returns "no such app: "

[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361323#comment-15361323 ] Saisai Shao commented on SPARK-15923: - [~WeiqingYang] This is the correct behavior. I

[jira] [Commented] (AMBARI-17545) Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading

[ https://issues.apache.org/jira/browse/AMBARI-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360862#comment-15360862 ] Saisai Shao commented on AMBARI-17545: -- [~sumitmohanty] [~afernandez] please help t

[jira] [Created] (AMBARI-17545) Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading

Saisai Shao created AMBARI-17545: Summary: Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading Key: AMBARI-17545 URL: https://issues.apache.org/jira/browse/AMBARI-17545

[jira] [Created] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

Saisai Shao created SPARK-16342: --- Summary: Add a new Configurable Token Manager for Spark Running on YARN Key: SPARK-16342 URL: https://issues.apache.org/jira/browse/SPARK-16342 Project: Spark

[jira] [Commented] (AMBARI-17489) Remove spark.yarn.max.executor.failures configuration in Spark Ambari definition

[ https://issues.apache.org/jira/browse/AMBARI-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358123#comment-15358123 ] Saisai Shao commented on AMBARI-17489: -- CC [~sumitmohanty] [~jluniya] please help t

[jira] [Created] (AMBARI-17489) Remove spark.yarn.max.executor.failures configuration in Spark Ambari definition

Saisai Shao created AMBARI-17489: Summary: Remove spark.yarn.max.executor.failures configuration in Spark Ambari definition Key: AMBARI-17489 URL: https://issues.apache.org/jira/browse/AMBARI-17489 Pr

[jira] [Commented] (SPARK-16230) Executors self-killing after being assigned tasks while still in init

[ https://issues.apache.org/jira/browse/SPARK-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354280#comment-15354280 ] Saisai Shao commented on SPARK-16230: - I'm not sure which version Spark are you using

[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354207#comment-15354207 ] Saisai Shao commented on SPARK-16265: - If you want to run Spark on different JVM othe

[jira] [Commented] (SPARK-16246) Too many block-manager-slave-async-thread opened (TIMED_WAITING) for spark Kafka streaming

[ https://issues.apache.org/jira/browse/SPARK-16246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352481#comment-15352481 ] Saisai Shao commented on SPARK-16246: - It would be better to have a thread dump about

[jira] [Created] (AMBARI-17459) remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in HDP2.5 definition

Saisai Shao created AMBARI-17459: Summary: remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in HDP2.5 definition Key: AMBARI-17459 URL: https://issues.apache.org/jira/browse/AMBARI-

[jira] [Updated] (AMBARI-17328) Spark2 history server quick link still point to spark1 url

[ https://issues.apache.org/jira/browse/AMBARI-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated AMBARI-17328: - Issue Type: Sub-task (was: Technical task) > Spark2 history server quick link still point to sp

[jira] [Commented] (SPARK-16146) Spark application failed by Yarn preempting

[ https://issues.apache.org/jira/browse/SPARK-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345984#comment-15345984 ] Saisai Shao commented on SPARK-16146: - I see, that could explain why executor get los

<    5   6   7   8   9   10   11   12   13   14   >