[jira] [Resolved] (SPARK-24014) Add onStreamingStarted method to StreamingListener

[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24014. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull req

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443475#comment-16443475 ] Saisai Shao commented on SPARK-24008: - Why do you need to broadcast {{SQLContext}} or

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443495#comment-16443495 ] Saisai Shao commented on SPARK-24008: - The case you provided seems not so valid. You'

[jira] [Commented] (SPARK-24016) Yarn does not update node blacklist in static allocation

[ https://issues.apache.org/jira/browse/SPARK-24016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445252#comment-16445252 ] Saisai Shao commented on SPARK-24016: - I think this can be useful if we enabled "spa

[jira] [Assigned] (SPARK-22372) Make YARN client extend SparkApplication

[ https://issues.apache.org/jira/browse/SPARK-22372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22372: --- Assignee: Marcelo Vanzin > Make YARN client extend SparkApplication > --

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309363#comment-16309363 ] Saisai Shao commented on SPARK-22876: - This looks like a bug, ideally we should get n

[jira] [Assigned] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22587: --- Assignee: Mingjie Tang > Spark job fails if fs.defaultFS and application jar are different u

[jira] [Updated] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22587: Fix Version/s: 2.3.0 > Spark job fails if fs.defaultFS and application jar are different url >

[jira] [Resolved] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22587. - Resolution: Fixed > Spark job fails if fs.defaultFS and application jar are different url > -

[jira] [Commented] (SPARK-22921) Merge script should prompt for assigning jiras

[ https://issues.apache.org/jira/browse/SPARK-22921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321669#comment-16321669 ] Saisai Shao commented on SPARK-22921: - Hi [~irashid], looks like the changes will thr

[jira] [Resolved] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23029. - Resolution: Not A Problem > Setting spark.shuffle.file.buffer will make the shuffle fail > --

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323496#comment-16323496 ] Saisai Shao commented on SPARK-23029: - Please see the comments in the code in {{Bypas

[jira] [Commented] (SPARK-22958) Spark is stuck when the only one executor fails to register with driver

[ https://issues.apache.org/jira/browse/SPARK-22958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323569#comment-16323569 ] Saisai Shao commented on SPARK-22958: - If executor is failed to register itself to dr

[jira] [Commented] (SPARK-21475) Change to use NIO's Files API for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323590#comment-16323590 ] Saisai Shao commented on SPARK-21475: - [~zsxwing] is 3.0.0 the valid fix version? >

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323871#comment-16323871 ] Saisai Shao commented on SPARK-23029: - Total buffer size for one executor is {{32MB *

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323872#comment-16323872 ] Saisai Shao commented on SPARK-23029: - There should be no issue unless you want to fi

[jira] [Commented] (SPARK-22958) Spark is stuck when the only one executor fails to register with driver

[ https://issues.apache.org/jira/browse/SPARK-22958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16325986#comment-16325986 ] Saisai Shao commented on SPARK-22958: - Please see this line: https://github.com/apac

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328297#comment-16328297 ] Saisai Shao commented on SPARK-23123: - It looks like a Hadoop issue rather than a Spa

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328361#comment-16328361 ] Saisai Shao commented on SPARK-23123: - >From the stack, looks like the issue is from

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328364#comment-16328364 ] Saisai Shao commented on SPARK-23123: - [~ste...@apache.org], any thought on this issu

[jira] [Created] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

Saisai Shao created SPARK-23140: --- Summary: DataSourceV2Strategy is missing in HiveSessionStateBuilder Key: SPARK-23140 URL: https://issues.apache.org/jira/browse/SPARK-23140 Project: Spark Issu

[jira] [Updated] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-23140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23140: Environment: (was: DataSourceV2Strategy is not added into HiveSessionStateBuilder's planner, wh

[jira] [Updated] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-23140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23140: Description: DataSourceV2Strategy is not added into HiveSessionStateBuilder's planner, which will

[jira] [Updated] (SPARK-23147) Stage page will throw exception when there's no complete tasks

[ https://issues.apache.org/jira/browse/SPARK-23147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23147: Attachment: Screen Shot 2018-01-18 at 8.50.08 PM.png > Stage page will throw exception when there's

[jira] [Created] (SPARK-23147) Stage page will throw exception when there's no complete tasks

Saisai Shao created SPARK-23147: --- Summary: Stage page will throw exception when there's no complete tasks Key: SPARK-23147 URL: https://issues.apache.org/jira/browse/SPARK-23147 Project: Spark

[jira] [Updated] (SPARK-23147) Stage page will throw exception when there's no complete tasks

[ https://issues.apache.org/jira/browse/SPARK-23147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23147: Description: Current Stage page's task table will throw an exception when there's no complete task

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331542#comment-16331542 ] Saisai Shao commented on SPARK-23128: - I'm wondering if we need a SPIP for such chang

[jira] [Commented] (SPARK-23097) Migrate text socket source

[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331715#comment-16331715 ] Saisai Shao commented on SPARK-23097: - Hi [~joseph.torres], would you mind if I take

[jira] [Commented] (SPARK-23097) Migrate text socket source

[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331735#comment-16331735 ] Saisai Shao commented on SPARK-23097: - Thanks, will do. > Migrate text socket source

[jira] [Resolved] (SPARK-22976) Worker cleanup can remove running driver directories

[ https://issues.apache.org/jira/browse/SPARK-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22976. - Resolution: Fixed Fix Version/s: 2.3.0 > Worker cleanup can remove running driver director

[jira] [Assigned] (SPARK-22976) Worker cleanup can remove running driver directories

[ https://issues.apache.org/jira/browse/SPARK-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22976: --- Assignee: Russell Spitzer > Worker cleanup can remove running driver directories > -

[jira] [Resolved] (SPARK-22838) Avoid unnecessary copying of data

[ https://issues.apache.org/jira/browse/SPARK-22838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22838. - Resolution: Invalid > Avoid unnecessary copying of data > - > >

[jira] [Updated] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23187: Affects Version/s: (was: 2.3.1) 2.3.0 > Accumulator object can not be se

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335428#comment-16335428 ] Saisai Shao commented on SPARK-23187: - I will try to investigate on it. > Accumulato

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338553#comment-16338553 ] Saisai Shao commented on SPARK-23206: - I think this Jira duplicates SPARK-9103. Also

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338737#comment-16338737 ] Saisai Shao commented on SPARK-23187: - Actually heartbeat report is OK according to m

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340521#comment-16340521 ] Saisai Shao commented on SPARK-23187: - I'm going to close this JIRA, as there's no is

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340522#comment-16340522 ] Saisai Shao commented on SPARK-23187: - I'm going to close this JIRA, as there's no is

[jira] [Resolved] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23187. - Resolution: Not A Problem > Accumulator object can not be sent from Executor to Driver >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340536#comment-16340536 ] Saisai Shao commented on SPARK-23206: - Would you please summarize what kind of metric

[jira] [Resolved] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23200. - Resolution: Fixed Fix Version/s: 2.4.0 > Reset configuration when restarting from checkpoi

[jira] [Commented] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340648#comment-16340648 ] Saisai Shao commented on SPARK-23200: - Issue resolved by pull request 20383 https://

[jira] [Assigned] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23200: --- Assignee: Anirudh Ramanathan > Reset configuration when restarting from checkpoints > --

[jira] [Created] (SPARK-23228) Able to track Python create SparkSession in JVM

Saisai Shao created SPARK-23228: --- Summary: Able to track Python create SparkSession in JVM Key: SPARK-23228 URL: https://issues.apache.org/jira/browse/SPARK-23228 Project: Spark Issue Type: Imp

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342824#comment-16342824 ] Saisai Shao commented on SPARK-23187: - I think I tried, {{accumUpdates}} can be recei

[jira] [Resolved] (SPARK-23088) History server not showing incomplete/running applications

[ https://issues.apache.org/jira/browse/SPARK-23088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23088. - Resolution: Fixed Assignee: paul mackles Fix Version/s: 2.4.0 Issue resolved by p

[jira] [Created] (SPARK-23279) Avoid triggering distributed job for Console sink

Saisai Shao created SPARK-23279: --- Summary: Avoid triggering distributed job for Console sink Key: SPARK-23279 URL: https://issues.apache.org/jira/browse/SPARK-23279 Project: Spark Issue Type: I

[jira] [Commented] (SPARK-23279) Avoid triggering distributed job for Console sink

[ https://issues.apache.org/jira/browse/SPARK-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346301#comment-16346301 ] Saisai Shao commented on SPARK-23279: - Issue resolved by pull request 20447 https://g

[jira] [Resolved] (SPARK-23279) Avoid triggering distributed job for Console sink

[ https://issues.apache.org/jira/browse/SPARK-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23279. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.3.0 > Avoid triggering d

[jira] [Reopened] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reopened SPARK-23200: - Assignee: (was: Santiago Saavedra) > Reset configuration when restarting from checkpoints >

[jira] [Updated] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23200: Fix Version/s: (was: 2.4.0) > Reset configuration when restarting from checkpoints > --

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349604#comment-16349604 ] Saisai Shao commented on SPARK-23235: - There's a new similar Jira (SPARK-23206) about

[jira] [Commented] (SPARK-12343) Remove YARN Client / ClientArguments

[ https://issues.apache.org/jira/browse/SPARK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153910#comment-15153910 ] Saisai Shao commented on SPARK-12343: - Hi guys, Do we still want to support users w

[jira] [Created] (SPARK-13426) Remove the support of SIMR cluster manager

Saisai Shao created SPARK-13426: --- Summary: Remove the support of SIMR cluster manager Key: SPARK-13426 URL: https://issues.apache.org/jira/browse/SPARK-13426 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-13426) Remove the support of SIMR cluster manager

[ https://issues.apache.org/jira/browse/SPARK-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13426: Issue Type: Sub-task (was: Bug) Parent: SPARK-11806 > Remove the support of SIMR cluster m

[jira] [Updated] (SPARK-13426) Remove the support of SIMR cluster manager

[ https://issues.apache.org/jira/browse/SPARK-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13426: Description: Currently there's not so many users who will use SIMR to run Spark, especially in a i

[jira] [Created] (SPARK-13447) Fix AM failure situation for dynamic allocation diabled sitation

Saisai Shao created SPARK-13447: --- Summary: Fix AM failure situation for dynamic allocation diabled sitation Key: SPARK-13447 URL: https://issues.apache.org/jira/browse/SPARK-13447 Project: Spark

[jira] [Updated] (SPARK-13447) Fix AM failure situation for dynamic allocation diabled situation

[ https://issues.apache.org/jira/browse/SPARK-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13447: Summary: Fix AM failure situation for dynamic allocation diabled situation (was: Fix AM failure si

[jira] [Updated] (SPARK-13447) Fix AM failure situation for dynamic allocation disabled situation

[ https://issues.apache.org/jira/browse/SPARK-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13447: Summary: Fix AM failure situation for dynamic allocation disabled situation (was: Fix AM failure s

[jira] [Commented] (SPARK-13608) py4j.Py4JException: Method createDirectStream([class org.apache.spark.streaming.api.java.JavaStreamingContext, class java.util.HashMap, class java.util.HashSet, class

[ https://issues.apache.org/jira/browse/SPARK-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175144#comment-15175144 ] Saisai Shao commented on SPARK-13608: - Hi [~avatarzhang] , would you please elaborate

[jira] [Closed] (SPARK-13608) py4j.Py4JException: Method createDirectStream([class org.apache.spark.streaming.api.java.JavaStreamingContext, class java.util.HashMap, class java.util.HashSet, class jav

[ https://issues.apache.org/jira/browse/SPARK-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-13608. --- Resolution: Not A Problem > py4j.Py4JException: Method createDirectStream([class > org.apache.spark.

[jira] [Commented] (SPARK-11157) Allow Spark to be built without assemblies

[ https://issues.apache.org/jira/browse/SPARK-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175276#comment-15175276 ] Saisai Shao commented on SPARK-11157: - Hi [~vanzin], for Spark Streaming PySpark rela

[jira] [Created] (SPARK-13642) Inconsistent finishing state between driver and AM

Saisai Shao created SPARK-13642: --- Summary: Inconsistent finishing state between driver and AM Key: SPARK-13642 URL: https://issues.apache.org/jira/browse/SPARK-13642 Project: Spark Issue Type:

[jira] [Updated] (SPARK-13642) Inconsistent finishing state between driver and AM

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Description: Currently when running Spark on Yarn with yarn cluster mode, the default application

[jira] [Commented] (SPARK-13642) Inconsistent finishing state between driver and AM

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177347#comment-15177347 ] Saisai Shao commented on SPARK-13642: - [~tgraves] [~vanzin], would you please comment

[jira] [Updated] (SPARK-13642) Inconsistent finishing state between driver and AM

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Description: Currently when running Spark on Yarn with yarn cluster mode, the default application

[jira] [Updated] (SPARK-13642) Inconsistent finishing state between driver and AM

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Description: Currently when running Spark on Yarn with yarn cluster mode, the default application

[jira] [Commented] (SPARK-13642) Inconsistent finishing state between driver and AM

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179102#comment-15179102 ] Saisai Shao commented on SPARK-13642: - Hi [~tgraves], thanks a lot for your comments.

[jira] [Updated] (SPARK-13642) Different finishing state between Spark and YARN

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Summary: Different finishing state between Spark and YARN (was: Inconsistent finishing state betwe

[jira] [Comment Edited] (SPARK-13642) Different finishing state between Spark and YARN

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179102#comment-15179102 ] Saisai Shao edited comment on SPARK-13642 at 3/4/16 1:45 AM: -

[jira] [Updated] (SPARK-13642) Properly handle signal kill of ApplicationMaster

[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Summary: Properly handle signal kill of ApplicationMaster (was: Different finishing state between

[jira] [Created] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

Saisai Shao created SPARK-13669: --- Summary: Job will always fail in the external shuffle service unavailable situation Key: SPARK-13669 URL: https://issues.apache.org/jira/browse/SPARK-13669 Project: Spa

[jira] [Commented] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179485#comment-15179485 ] Saisai Shao commented on SPARK-13669: - Hi [~imranr], I think you fixed this issue (SP

[jira] [Created] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

Saisai Shao created SPARK-13675: --- Summary: The url link in historypage is not correct for application running in yarn cluster mode Key: SPARK-13675 URL: https://issues.apache.org/jira/browse/SPARK-13675

[jira] [Updated] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

[ https://issues.apache.org/jira/browse/SPARK-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13675: Description: Current URL for each application to access history UI is like: http://localhost:18080

[jira] [Updated] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

[ https://issues.apache.org/jira/browse/SPARK-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13675: Description: Current URL for each application to access history UI is like: http://localhost:18080

[jira] [Updated] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

[ https://issues.apache.org/jira/browse/SPARK-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13675: Attachment: Screen Shot 2016-02-29 at 3.57.32 PM.png > The url link in historypage is not correct f

[jira] [Commented] (SPARK-13723) YARN - Change behavior of --num-executors when spark.dynamicAllocation.enabled true

[ https://issues.apache.org/jira/browse/SPARK-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184177#comment-15184177 ] Saisai Shao commented on SPARK-13723: - Yes, I agree with what [~tgraves] described ab

[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194653#comment-15194653 ] Saisai Shao commented on SPARK-13877: - I agreed to move this out, so it could easily

[jira] [Created] (SPARK-13885) Spark On Yarn attempt id representation regression

Saisai Shao created SPARK-13885: --- Summary: Spark On Yarn attempt id representation regression Key: SPARK-13885 URL: https://issues.apache.org/jira/browse/SPARK-13885 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13914) Add functionality to back up spark event logs

[ https://issues.apache.org/jira/browse/SPARK-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196580#comment-15196580 ] Saisai Shao commented on SPARK-13914: - Agree with [~srowen], this feature is quite us

[jira] [Commented] (SPARK-13915) Allow bin/spark-submit to be called via symbolic link

[ https://issues.apache.org/jira/browse/SPARK-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196590#comment-15196590 ] Saisai Shao commented on SPARK-13915: - I suppose it should be worked if you set {{SPA

[jira] [Commented] (SPARK-13955) Spark in yarn mode fails

[ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203774#comment-15203774 ] Saisai Shao commented on SPARK-13955: - [~zjffdu], I think way to fix this is to speci

[jira] [Created] (SPARK-14062) Put metrics.properties to distributed cache for Spark running on Yarn

Saisai Shao created SPARK-14062: --- Summary: Put metrics.properties to distributed cache for Spark running on Yarn Key: SPARK-14062 URL: https://issues.apache.org/jira/browse/SPARK-14062 Project: Spark

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119128#comment-14119128 ] Saisai Shao commented on SPARK-3146: Hi [~tdas], Sorry for late response, thanks a lo

[jira] [Commented] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120834#comment-14120834 ] Saisai Shao commented on SPARK-3292: Hi [~guowei], did you test the scenario with wind

[jira] [Commented] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120839#comment-14120839 ] Saisai Shao commented on SPARK-3292: I think these two tickets address the same proble

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121160#comment-14121160 ] Saisai Shao commented on SPARK-3129: Hi [~hshreedharan], one more question: Is your d

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122274#comment-14122274 ] Saisai Shao commented on SPARK-3129: Hi [~hshreedharan]], thanks for your reply, is th

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122292#comment-14122292 ] Saisai Shao commented on SPARK-2926: Hi Matei, sorry for late response, I will test mo

[jira] [Commented] (SPARK-2122) Move aggregation into shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126471#comment-14126471 ] Saisai Shao commented on SPARK-2122: Yes, this is a duplicated ticket, it is fixed in

[jira] [Closed] (SPARK-2122) Move aggregation into shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-2122. -- Resolution: Duplicate > Move aggregation into shuffle implementation > -

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131032#comment-14131032 ] Saisai Shao commented on SPARK-2926: Hi [~matei], according to your comments, I just d

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Attachment: Spark Shuffle Test Report(contd).pdf > Add MR-style (merge-sort) SortShuffleReader for sor

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131314#comment-14131314 ] Saisai Shao commented on SPARK-2926: Hi Reynold, thanks a lot for your watching this,

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132598#comment-14132598 ] Saisai Shao commented on SPARK-2926: Ok, I will take a try and let you know then it is

[jira] [Comment Edited] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132598#comment-14132598 ] Saisai Shao edited comment on SPARK-2926 at 9/13/14 8:09 AM: -

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133694#comment-14133694 ] Saisai Shao commented on SPARK-2926: Hey [~rxin], here is the branch rebased on your c

[jira] [Commented] (SPARK-3563) Shuffle data not always be cleaned

[ https://issues.apache.org/jira/browse/SPARK-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137246#comment-14137246 ] Saisai Shao commented on SPARK-3563: In my thought, I think it relies on JVM's GC stra

[jira] [Commented] (SPARK-3563) Shuffle data not always be cleaned

[ https://issues.apache.org/jira/browse/SPARK-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138389#comment-14138389 ] Saisai Shao commented on SPARK-3563: I think it relies on JVM's GC strategy to treat r

<    1   2   3   4   5   6   7   8   9   10   >