[jira] [Commented] (AMBARI-17489) Remove spark.yarn.max.executor.failures configuration in Spark Ambari definition

[ https://issues.apache.org/jira/browse/AMBARI-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590615#comment-15590615 ] Saisai Shao commented on AMBARI-17489: -- [~sumitmohanty], I've rebased this patch to branch-2.4 and

[jira] [Created] (SPARK-17999) Add getPreferredLocations for KafkaSourceRDD

Saisai Shao created SPARK-17999: --- Summary: Add getPreferredLocations for KafkaSourceRDD Key: SPARK-17999 URL: https://issues.apache.org/jira/browse/SPARK-17999 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17984) Add support for numa aware feature

[ https://issues.apache.org/jira/browse/SPARK-17984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15587459#comment-15587459 ] Saisai Shao commented on SPARK-17984: - NUMA should be supported by most commodity servers as well as

[jira] [Commented] (SPARK-17787) spark submit throws error while using kafka Appender log4j:ERROR Could not instantiate class [kafka.producer.KafkaLog4jAppender]. java.lang.ClassNotFoundException: kaf

[ https://issues.apache.org/jira/browse/SPARK-17787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568020#comment-15568020 ] Saisai Shao commented on SPARK-17787: - I guess you don't have jars on the node where AM/Executors

[jira] [Closed] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-17814. --- Resolution: Won't Fix > spark submit arguments are truncated in yarn-cluster mode >

[jira] [Comment Edited] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567964#comment-15567964 ] Saisai Shao edited comment on SPARK-17814 at 10/12/16 7:57 AM: --- By checking

[jira] [Comment Edited] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567964#comment-15567964 ] Saisai Shao edited comment on SPARK-17814 at 10/12/16 7:56 AM: --- By checking

[jira] [Commented] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567964#comment-15567964 ] Saisai Shao commented on SPARK-17814: - By checking the yarn code, I think you might run into YARN's

[jira] [Commented] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567933#comment-15567933 ] Saisai Shao commented on SPARK-17814: - Which version of YARN do you use? > spark submit arguments

[jira] [Commented] (SPARK-17814) spark submit arguments are truncated in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567931#comment-15567931 ] Saisai Shao commented on SPARK-17814: - This looks like a YARN problem, after tracking the code path,

[jira] [Commented] (SPARK-17846) A bad state of Running Applications with spark standalone HA

[ https://issues.apache.org/jira/browse/SPARK-17846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567662#comment-15567662 ] Saisai Shao commented on SPARK-17846: - I think this issue should be the same as SPARK-14262. > A bad

[jira] [Created] (SPARK-17686) Propose to print Scala version in "spark-submit --version" command

Saisai Shao created SPARK-17686: --- Summary: Propose to print Scala version in "spark-submit --version" command Key: SPARK-17686 URL: https://issues.apache.org/jira/browse/SPARK-17686 Project: Spark

[jira] [Created] (SPARK-17678) Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

Saisai Shao created SPARK-17678: --- Summary: Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port" Key: SPARK-17678 URL: https://issues.apache.org/jira/browse/SPARK-17678 Project: Spark

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515836#comment-15515836 ] Saisai Shao commented on SPARK-17637: - [~zhanzhang] would you mind sharing more details about your

[jira] [Comment Edited] (SPARK-17624) Flaky test? StateStoreSuite maintenance

[ https://issues.apache.org/jira/browse/SPARK-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512237#comment-15512237 ] Saisai Shao edited comment on SPARK-17624 at 9/22/16 5:36 AM: -- I cannot

[jira] [Commented] (SPARK-17624) Flaky test? StateStoreSuite maintenance

[ https://issues.apache.org/jira/browse/SPARK-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512237#comment-15512237 ] Saisai Shao commented on SPARK-17624: - I cannot reproduce locally on my > Flaky test?

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17604: Issue Type: Sub-task (was: Improvement) Parent: SPARK-17267 > Support purging aged file

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17604: Description: Currently with SPARK-15698, FileStreamSource metadata log will be compacted

[jira] [Created] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

Saisai Shao created SPARK-17604: --- Summary: Support purging aged file entry for FileStreamSource metadata log Key: SPARK-17604 URL: https://issues.apache.org/jira/browse/SPARK-17604 Project: Spark

[jira] [Commented] (SPARK-15698) Ability to remove old metadata for structure streaming MetadataLog

[ https://issues.apache.org/jira/browse/SPARK-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505799#comment-15505799 ] Saisai Shao commented on SPARK-15698: - I think [~rxin] set this target version before. I'm OK to

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500522#comment-15500522 ] Saisai Shao commented on SPARK-17566: - I've already submitted a PR under SPARK-17512, since this JIRA

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500511#comment-15500511 ] Saisai Shao commented on SPARK-17566: - Shouldn't it be {{!isYarnCluster}}? Since we need to avoid

[jira] [Updated] (SPARK-17512) Specifying remote files for Python based Spark jobs in Yarn cluster mode not working

[ https://issues.apache.org/jira/browse/SPARK-17512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17512: Component/s: YARN > Specifying remote files for Python based Spark jobs in Yarn cluster mode not

[jira] [Commented] (SPARK-17512) Specifying remote files for Python based Spark jobs in Yarn cluster mode not working

[ https://issues.apache.org/jira/browse/SPARK-17512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500178#comment-15500178 ] Saisai Shao commented on SPARK-17512: - This is due to some behavior changes during submitting spark

[jira] [Closed] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-17566. --- Resolution: Duplicate > "--master yarn --deploy-mode cluster" gives "Launching Python applications

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500145#comment-15500145 ] Saisai Shao commented on SPARK-17566: - Sorry I misunderstood your point, looks like it should be

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500109#comment-15500109 ] Saisai Shao commented on SPARK-17566: - Can you confirm the above command you mentioned can be run on

[jira] [Comment Edited] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

[ https://issues.apache.org/jira/browse/SPARK-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495243#comment-15495243 ] Saisai Shao edited comment on SPARK-17522 at 9/16/16 3:19 AM: -- [~sunrui] I

[jira] [Commented] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

[ https://issues.apache.org/jira/browse/SPARK-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495243#comment-15495243 ] Saisai Shao commented on SPARK-17522: - [~sunrui] I think the performance is depended on different

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470553#comment-15470553 ] Saisai Shao commented on SPARK-17340: - I think what [~asukhenko] mentioned in the description is one

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455080#comment-15455080 ] Saisai Shao commented on SPARK-17340: - yarn-client and yarn-cluster has different way to handle

[jira] [Comment Edited] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454777#comment-15454777 ] Saisai Shao edited comment on SPARK-17340 at 9/1/16 11:02 AM: -- I think in

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455076#comment-15455076 ] Saisai Shao commented on SPARK-17340: - You can try not kill local {{yarn#client}} process after

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455055#comment-15455055 ] Saisai Shao commented on SPARK-17340: - I'm saying yarn cluster mode, I think here in my comment

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454777#comment-15454777 ] Saisai Shao commented on SPARK-17340: - I think in your scenario, it is because you killed local

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436222#comment-15436222 ] Saisai Shao commented on SPARK-17204: - Yes, I could reproduce this issue, but not constantly,

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436213#comment-15436213 ] Saisai Shao commented on SPARK-17204: - I think to reflect the issue {{sc.range(0, 0)}} should be

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436205#comment-15436205 ] Saisai Shao commented on SPARK-17204: - No, I tested in yarn cluster, not local mode. > Spark 2.0 off

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436179#comment-15436179 ] Saisai Shao commented on SPARK-17204: - It works OK in my local test with latest build: {code} val

[jira] [Updated] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17209: Summary: Support manual credential updating in the run-time for Spark on YARN (was: Support

[jira] [Updated] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17209: Description: Current Spark on YARN supports time based credential renewal and updating, this

[jira] [Created] (SPARK-17209) Support manual credential updating in the run-time

Saisai Shao created SPARK-17209: --- Summary: Support manual credential updating in the run-time Key: SPARK-17209 URL: https://issues.apache.org/jira/browse/SPARK-17209 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17148) NodeManager exit because of exception “Executor is not registered”

[ https://issues.apache.org/jira/browse/SPARK-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430402#comment-15430402 ] Saisai Shao commented on SPARK-17148: - I manually verified this by explicitly throwing the

[jira] [Created] (SPARK-17019) Expose off-heap memory usage in various places

Saisai Shao created SPARK-17019: --- Summary: Expose off-heap memory usage in various places Key: SPARK-17019 URL: https://issues.apache.org/jira/browse/SPARK-17019 Project: Spark Issue Type:

[jira] [Commented] (AMBARI-18091) Use https url for Spark2 Service check when WireEncryption is enabled

[ https://issues.apache.org/jira/browse/AMBARI-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414751#comment-15414751 ] Saisai Shao commented on AMBARI-18091: -- Please help to review, [~sumitmohanty] [~jluniya], thanks a

[jira] [Assigned] (AMBARI-18091) Use https url for Spark2 Service check when WireEncryption is enabled

[ https://issues.apache.org/jira/browse/AMBARI-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned AMBARI-18091: Assignee: Saisai Shao > Use https url for Spark2 Service check when WireEncryption is

[jira] [Commented] (SPARK-16966) App Name is a randomUUID even when "spark.app.name" exists

[ https://issues.apache.org/jira/browse/SPARK-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414481#comment-15414481 ] Saisai Shao commented on SPARK-16966: - Here is the code in {{SparkSubmitArguments}} to handle the

[jira] [Commented] (SPARK-16966) App Name is a randomUUID even when "spark.app.name" exists

[ https://issues.apache.org/jira/browse/SPARK-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413201#comment-15413201 ] Saisai Shao commented on SPARK-16966: - Yes, agreed. A better way is to handle this app name thing in

[jira] [Commented] (SPARK-16944) [MESOS] Improve data locality when launching new executors when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411299#comment-15411299 ] Saisai Shao commented on SPARK-16944: - Does Mesos have the similar concept like Yarn container, also

[jira] [Comment Edited] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411159#comment-15411159 ] Saisai Shao edited comment on SPARK-16914 at 8/8/16 1:48 AM: - So from your

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411159#comment-15411159 ] Saisai Shao commented on SPARK-16914: - So from your description, is this exception mainly due to the

[jira] [Updated] (SPARK-16871) Support getting HBase tokens from multiple clusters dynamically

[ https://issues.apache.org/jira/browse/SPARK-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16871: Summary: Support getting HBase tokens from multiple clusters dynamically (was: Support getting

[jira] [Created] (SPARK-16871) Support getting HBase tokens from multiple clusters and dynamically

Saisai Shao created SPARK-16871: --- Summary: Support getting HBase tokens from multiple clusters and dynamically Key: SPARK-16871 URL: https://issues.apache.org/jira/browse/SPARK-16871 Project: Spark

[jira] [Commented] (SPARK-16864) Comprehensive version info

[ https://issues.apache.org/jira/browse/SPARK-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405487#comment-15405487 ] Saisai Shao commented on SPARK-16864: - A program way to get spark version is to call

[jira] [Commented] (SPARK-14453) Remove SPARK_JAVA_OPTS environment variable

[ https://issues.apache.org/jira/browse/SPARK-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405112#comment-15405112 ] Saisai Shao commented on SPARK-14453: - If you want to fix this issue, it would be better target to

[jira] [Comment Edited] (SPARK-16815) Dataset[List[T]] leads to ArrayStoreException

[ https://issues.apache.org/jira/browse/SPARK-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401566#comment-15401566 ] Saisai Shao edited comment on SPARK-16815 at 8/1/16 6:01 AM: - >From my

[jira] [Commented] (SPARK-16815) Dataset[List[T]] leads to ArrayStoreException

[ https://issues.apache.org/jira/browse/SPARK-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401566#comment-15401566 ] Saisai Shao commented on SPARK-16815: - >From my understanding you can use {code}

[jira] [Commented] (SPARK-16817) Enable storing of shuffle data in Alluxio

[ https://issues.apache.org/jira/browse/SPARK-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401432#comment-15401432 ] Saisai Shao commented on SPARK-16817: - What's difference compared to use ramdisk to store shuffle

[jira] [Commented] (AMBARI-17954) Fix Spark hdp.version issues in upgrading and fresh install

[ https://issues.apache.org/jira/browse/AMBARI-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398740#comment-15398740 ] Saisai Shao commented on AMBARI-17954: -- CC [~sumitmohanty] [~jluniya], please help to review,

[jira] [Created] (AMBARI-17954) Fix Spark hdp.version issues in upgrading and fresh install

Saisai Shao created AMBARI-17954: Summary: Fix Spark hdp.version issues in upgrading and fresh install Key: AMBARI-17954 URL: https://issues.apache.org/jira/browse/AMBARI-17954 Project: Ambari

[jira] [Commented] (SPARK-16085) Spark stand-alone ui redirects to RM application master UI for yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395705#comment-15395705 ] Saisai Shao commented on SPARK-16085: - Unfortunately, there's no such configuration for Spark to

[jira] [Commented] (SPARK-16708) ExecutorAllocationManager.numRunningTasks can be negative when stage retry

[ https://issues.apache.org/jira/browse/SPARK-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393377#comment-15393377 ] Saisai Shao commented on SPARK-16708: - Looks similar to SPARK-11334, and I have a patch on it, though

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393033#comment-15393033 ] Saisai Shao commented on SPARK-16723: - So maybe this application is not yet started in the yarn side,

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393003#comment-15393003 ] Saisai Shao commented on SPARK-16723: - Did you enable log aggregation in YARN, if not this command is

[jira] [Comment Edited] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393003#comment-15393003 ] Saisai Shao edited comment on SPARK-16723 at 7/26/16 1:36 AM: -- Did you

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392980#comment-15392980 ] Saisai Shao commented on SPARK-16723: - {{yarn logs -applicationId application_1467990031555_0089}} >

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392965#comment-15392965 ] Saisai Shao commented on SPARK-16723: - I think you should check the AM and executor logs to see the

[jira] [Commented] (AMBARI-16864) Add unit tests for Spark2 service definition

[ https://issues.apache.org/jira/browse/AMBARI-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383435#comment-15383435 ] Saisai Shao commented on AMBARI-16864: -- Done, patch updated. > Add unit tests for Spark2 service

[jira] [Updated] (SPARK-16540) Jars specified with --jars will added twice when running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16540: Description: Currently when running spark on yarn, jars specified with \--jars, \--packages will

[jira] [Created] (SPARK-16540) Jars specified with --jars will added twice when running on YARN

Saisai Shao created SPARK-16540: --- Summary: Jars specified with --jars will added twice when running on YARN Key: SPARK-16540 URL: https://issues.apache.org/jira/browse/SPARK-16540 Project: Spark

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376449#comment-15376449 ] Saisai Shao commented on SPARK-16534: - Maybe I can take a try if no one is working on this :). BTW do

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376432#comment-15376432 ] Saisai Shao commented on SPARK-16534: - Is there anyone working on this? > Kafka 0.10 Python support

[jira] [Commented] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374612#comment-15374612 ] Saisai Shao commented on SPARK-16521: - I see, sorry about the duplication. > Add support of

[jira] [Closed] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16521. --- Resolution: Duplicate > Add support of parameterized configuration for SparkConf >

[jira] [Commented] (SPARK-16522) [MESOS] Spark application throws exception on exit

[ https://issues.apache.org/jira/browse/SPARK-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374583#comment-15374583 ] Saisai Shao commented on SPARK-16522: - Perhaps there's race condition when exiting the Spark

[jira] [Updated] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16521: Priority: Minor (was: Major) > Add support of parameterized configuration for SparkConf >

[jira] [Created] (SPARK-16521) Add support of parameterized configuration for SparkConf

Saisai Shao created SPARK-16521: --- Summary: Add support of parameterized configuration for SparkConf Key: SPARK-16521 URL: https://issues.apache.org/jira/browse/SPARK-16521 Project: Spark Issue

[jira] [Commented] (SPARK-16428) Spark file system watcher not working on Windows

[ https://issues.apache.org/jira/browse/SPARK-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372885#comment-15372885 ] Saisai Shao commented on SPARK-16428: - bq. Spark detected those files with the above terminal output

[jira] [Commented] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371975#comment-15371975 ] Saisai Shao commented on SPARK-16435: - OK, I will file a small patch to add the warning log about

[jira] [Created] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

Saisai Shao created SPARK-16435: --- Summary: Behavior changes if initialExecutor is less than minExecutor for dynamic allocation Key: SPARK-16435 URL: https://issues.apache.org/jira/browse/SPARK-16435

[jira] [Updated] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14743: Component/s: YARN > Improve delegation token handling in secure clusters >

[jira] [Closed] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16342. --- Resolution: Duplicate > Add a new Configurable Token Manager for Spark Running on YARN >

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365535#comment-15365535 ] Saisai Shao commented on SPARK-16342: - Close as JIRA as duplicated and move to SPARK-14743. > Add a

[jira] [Commented] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365534#comment-15365534 ] Saisai Shao commented on SPARK-14743: - Post design doc here and move SPARK-16342 to here. > Improve

[jira] [Comment Edited] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365534#comment-15365534 ] Saisai Shao edited comment on SPARK-14743 at 7/7/16 3:18 AM: - Post design doc

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365483#comment-15365483 ] Saisai Shao commented on SPARK-16342: - OK, I see. Sorry I didn't notice your JIRA, let me consolidate

[jira] [Issue Comment Deleted] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16342: Comment: was deleted (was: Thanks [~vanzin] for pointing out the jira, looks like most part of the

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365477#comment-15365477 ] Saisai Shao commented on SPARK-16342: - Thanks [~vanzin] for pointing out the jira, looks like most

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365478#comment-15365478 ] Saisai Shao commented on SPARK-16342: - Thanks [~vanzin] for pointing out the jira, looks like most

[jira] [Closed] (SPARK-16393) Move and implement token obtaining logics for built-in token providers

[ https://issues.apache.org/jira/browse/SPARK-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16393. --- Resolution: Duplicate > Move and implement token obtaining logics for built-in token providers >

[jira] [Closed] (SPARK-16392) Build ConfigurableTokenManager framework

[ https://issues.apache.org/jira/browse/SPARK-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16392. --- Resolution: Duplicate > Build ConfigurableTokenManager framework >

[jira] [Created] (SPARK-16393) Move and implement token obtaining logics for built-in token providers

Saisai Shao created SPARK-16393: --- Summary: Move and implement token obtaining logics for built-in token providers Key: SPARK-16393 URL: https://issues.apache.org/jira/browse/SPARK-16393 Project: Spark

[jira] [Created] (SPARK-16392) Build ConfigurableTokenManager framework

Saisai Shao created SPARK-16392: --- Summary: Build ConfigurableTokenManager framework Key: SPARK-16392 URL: https://issues.apache.org/jira/browse/SPARK-16392 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-16382) YARN - Dynamic allocation with spark.executor.instances should increase max executors.

[ https://issues.apache.org/jira/browse/SPARK-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363592#comment-15363592 ] Saisai Shao commented on SPARK-16382: - I would suggest to fail and complain. Max usually specifies

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361917#comment-15361917 ] Saisai Shao commented on SPARK-16342: - [~tgraves] [~vanzin] [~ste...@apache.org] would be great to

[jira] [Updated] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16342: Description: Current Spark on YARN token management has some problems: 1. Supported service is

[jira] [Commented] (SPARK-15923) Spark Application rest api returns "no such app: "

[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361323#comment-15361323 ] Saisai Shao commented on SPARK-15923: - [~WeiqingYang] This is the correct behavior. In the client

[jira] [Commented] (AMBARI-17545) Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading

[ https://issues.apache.org/jira/browse/AMBARI-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360862#comment-15360862 ] Saisai Shao commented on AMBARI-17545: -- [~sumitmohanty] [~afernandez] please help to review, not

[jira] [Created] (AMBARI-17545) Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading

Saisai Shao created AMBARI-17545: Summary: Remove spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property in upgrading Key: AMBARI-17545 URL: https://issues.apache.org/jira/browse/AMBARI-17545

[jira] [Created] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

Saisai Shao created SPARK-16342: --- Summary: Add a new Configurable Token Manager for Spark Running on YARN Key: SPARK-16342 URL: https://issues.apache.org/jira/browse/SPARK-16342 Project: Spark

<    4   5   6   7   8   9   10   11   12   13   >