[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183793745 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #20512: [SPARK-23182][CORE] Allow enabling TCP keep alive on the...

2018-04-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20512 this is just far enough outside my expertise I don't have an opinion -- but @zsxwing might have some thoughts

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183781695 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183779968 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183778637 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183777930 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -689,6 +689,20 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183776603 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183775539 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21131: [SPARK-23433][CORE] Late zombie task completions update ...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21131 @markhamstra @zsxwing @jiangxb1987 @Ngone51 would appreciate a review, thanks --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21132: [SPARK-24029][core] Follow up: set SO_REUSEADDR on the s...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21132 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21096: [SPARK-24011][CORE][WIP] cache rdd's immediate parent Sh...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21096 its not a bad idea, but as @markhamstra mentions we can't have an `rddToImmediateShuffleDependency` data structure which keeps growing. You could keep it local to one job submission, which would

[GitHub] spark issue #20998: [SPARK-23888][CORE] correct the comment of hasAttemptOnH...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20998 merged to master, thanks @Ngone51 . I also updated the commit msg some before committing, I thought it best to focus on the eventual change, figured it wasn't worth bugging you for another update

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-23 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/21131 [SPARK-23433][CORE] Late zombie task completions update all tasksets Fetch failure lead to multiple tasksets which are active for a given stage. A late completion from an earlier attempt

[GitHub] spark issue #21085: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21085 whoops, thanks for the reminder @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21085: [SPARK-23948] Trigger mapstage's job listener in ...

2018-04-19 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/21085 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20640: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20640 Hi Everyone, just wanted to let you know that SPARK-16630 is progressing now https://github.com/apache/spark/pull/21068 and after some discussion, the implementation will actually need to live

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21068 @tgravescs on the blacklist ratio for task-based blacklisting -- there is nothing, but there are some related jiras: [SPARK-22148](https://issues.apache.org/jira/browse/SPARK-22148) & [SPARK-1

[GitHub] spark issue #21104: [SPARK-24021][CORE] fix bug in BlacklistTracker's update...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21104 merged to master & 2.3.1, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional command

[GitHub] spark issue #21104: [SPARK-24021][CORE] fix bug in BlacklistTracker's update...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21104 lgtm, good catch --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-18 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21068 I think Tom makes a good case for why this should live in the YarnAllocator as you have it. I also don't think you need to worry about creating an abstract class yet, that refactoring can

[GitHub] spark issue #21085: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21085 known flaky test https://issues.apache.org/jira/browse/SPARK-23894 merging to branch 2.3 --- - To unsubscribe, e

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21068 > actually the only other thing I need to make sure is there aren't any delays if we now send the information from yarn allocator back to scheduler and then I assume it would need to get it b

[GitHub] spark issue #21019: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21019 I guess I rely entirely on the merge script, but in these simple cases I should just do the push directly

[GitHub] spark issue #21085: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21085 clean cherry-pick of https://github.com/apache/spark/pull/21019, I just forgot to merge back to 2.3 --- - To unsubscribe, e-mail

[GitHub] spark issue #21019: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21019 a few minutes after merging this I realized I should have also merged to branch 2.3. I don't see a way to do that without another PR. oops. I opened this, its a clean cherry-pick https

[GitHub] spark pull request #21085: [SPARK-23948] Trigger mapstage's job listener in ...

2018-04-17 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/21085 [SPARK-23948] Trigger mapstage's job listener in submitMissingTasks ## What changes were proposed in this pull request? SparkContext submitted a map stage from `submitMapStage

[GitHub] spark issue #20998: [SPARK-23888][CORE] speculative task should not run on a...

2018-04-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20998 @Ngone51 can you instead leave the behavior as is, and just update the comment? Sorry that its going to be a small change in the end, and all the extra work the bad comments led you to do

[GitHub] spark issue #21019: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21019 merged to master, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r181896603 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -234,8 +244,22 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #21019: [SPARK-23948] Trigger mapstage's job listener in ...

2018-04-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21019#discussion_r181889300 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1433,6 +1426,16 @@ class DAGScheduler

[GitHub] spark pull request #21019: [SPARK-23948] Trigger mapstage's job listener in ...

2018-04-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21019#discussion_r181891158 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2146,6 +2146,57 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #21019: [SPARK-23948] Trigger mapstage's job listener in ...

2018-04-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21019#discussion_r181892493 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2146,6 +2146,57 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r181513236 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/FailureWithinTimeIntervalTracker.scala --- @@ -0,0 +1,80

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r181515465 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,155

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21068 Jenkins, add to whitelist --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20998: [SPARK-23888][CORE] speculative task should not run on a...

2018-04-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20998 I'm not even really concerned about the case for two hosts -- I agree its fine if we do something sub-optimal. I'm more concerned about code-clarity and the behavior in general. It seems cleaner

[GitHub] spark issue #21041: [SPARK-23962][SQL][TEST] Fix race in currentExecutionIds...

2018-04-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21041 @vanzin @cloud-fan @gatorsmile please take a look, thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21019: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21019 I need to look more closely at the change, but your description of the problem makes sense. Can you also add a test case

[GitHub] spark pull request #21041: [SPARK-23962][SQL][TEST] Fix race in currentExecu...

2018-04-11 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/21041 [SPARK-23962][SQL][TEST] Fix race in currentExecutionIds(). SQLMetricsTestUtils.currentExecutionIds() was racing with the listener bus, which lead to some flaky tests. Wait till the listener

[GitHub] spark issue #20925: [SPARK-22941][core] Do not exit JVM when submit fails wi...

2018-04-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20925 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20925: [SPARK-22941][core] Do not exit JVM when submit fails wi...

2018-04-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20925 another unrelated flaky test, I filed https://issues.apache.org/jira/browse/SPARK-23962 --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #20998: [SPARK-23888][CORE] speculative task should not r...

2018-04-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20998#discussion_r180784912 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -880,8 +880,8 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark pull request #20998: [SPARK-23888][CORE] speculative task should not r...

2018-04-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20998#discussion_r180784374 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -880,6 +880,59 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark issue #20952: [SPARK-6951][core] Speed up parsing of event logs during...

2018-04-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20952 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20998: [SPARK-23888][CORE] speculative task should not run on a...

2018-04-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20998 @mridulm more thoughts? I think this is the right change but I will leave open for a bit to get more input --- - To unsubscribe

[GitHub] spark pull request #20998: [SPARK-23888][CORE] speculative task should not r...

2018-04-10 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20998#discussion_r180443917 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -880,6 +880,59 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark pull request #20998: [SPARK-23888][CORE] speculative task should not r...

2018-04-10 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20998#discussion_r180439612 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -880,6 +880,59 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark pull request #20998: [SPARK-23888][CORE] speculative task should not r...

2018-04-10 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20998#discussion_r180439559 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -880,6 +880,59 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark issue #20925: [SPARK-22941][core] Do not exit JVM when submit fails wi...

2018-04-10 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20925 Flaky test I've seen before: https://issues.apache.org/jira/browse/SPARK-23894 Jenkins, retest this please

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-10 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20940 btw you mentioned that some of the issues were fixed, but I haven't seen any more changes, maybe you forgot to push the changes

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-10 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r180426692 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -234,8 +244,22 @@ private[spark] class EventLoggingListener

[GitHub] spark issue #20952: [SPARK-6951][core] Speed up parsing of event logs during...

2018-04-09 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20952 lgtm assuming tests pass --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20925: [SPARK-22941][core] Do not exit JVM when submit fails wi...

2018-04-09 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20925 lgtm assuming tests pass --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20997: [SPARK-19185] [DSTREAMS] Avoid concurrent use of ...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20997#discussion_r180220465 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala --- @@ -0,0 +1,381 @@ +/* + * Licensed

[GitHub] spark pull request #20997: [SPARK-19185] [DSTREAMS] Avoid concurrent use of ...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20997#discussion_r180212631 --- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala --- @@ -0,0 +1,381 @@ +/* + * Licensed

[GitHub] spark pull request #20997: [SPARK-19185] [DSTREAMS] Avoid concurrent use of ...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20997#discussion_r180222555 --- Diff: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumerSuite.scala --- @@ -0,0 +1,111

[GitHub] spark pull request #20997: [SPARK-19185] [DSTREAMS] Avoid concurrent use of ...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20997#discussion_r180221087 --- Diff: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumerSuite.scala --- @@ -0,0 +1,111

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r180199472 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -772,6 +772,12 @@ private[spark] class Executor( val accumUpdates

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r180197576 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -234,8 +244,22 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #20952: [SPARK-6951][core] Speed up parsing of event logs...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20952#discussion_r180187517 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -125,6 +126,7 @@ private[history] class FsHistoryProvider(conf

[GitHub] spark pull request #20952: [SPARK-6951][core] Speed up parsing of event logs...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20952#discussion_r180197210 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -417,15 +419,23 @@ private[history] class FsHistoryProvider

[GitHub] spark issue #20952: [SPARK-6951][core] Speed up parsing of event logs during...

2018-04-09 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20952 typo in summary: "On top of data" --> "On top of that" (I think) --- - To unsubscribe, e

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r180117149 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -234,8 +244,22 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r180114405 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -234,8 +244,22 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r180110683 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -772,6 +772,12 @@ private[spark] class Executor( val accumUpdates

[GitHub] spark pull request #20925: [SPARK-22941][core] Do not exit JVM when submit f...

2018-04-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20925#discussion_r179987224 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -154,9 +165,17 @@ List buildSparkSubmitArgs

[GitHub] spark pull request #20925: [SPARK-22941][core] Do not exit JVM when submit f...

2018-04-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20925#discussion_r179987409 --- Diff: launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java --- @@ -154,9 +165,17 @@ List buildSparkSubmitArgs

[GitHub] spark pull request #20925: [SPARK-22941][core] Do not exit JVM when submit f...

2018-04-08 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20925#discussion_r179952361 --- Diff: core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala --- @@ -775,17 +781,17 @@ class SparkSubmitSuite } test

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-07 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20987 I filed https://issues.apache.org/jira/browse/SPARK-23894 for the test failure -- appears to be a flaky test

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-07 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20987#discussion_r179931481 --- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala --- @@ -257,19 +281,32 @@ class ExecutorSuite extends SparkFunSuite

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179802392 --- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala --- @@ -654,6 +681,25 @@ private[spark] object JsonProtocolSuite extends

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179795171 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -234,8 +244,22 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179797803 --- Diff: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala --- @@ -251,6 +260,163 @@ class EventLoggingListenerSuite extends

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179796802 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -268,6 +268,9 @@ private class LiveExecutor(val executorId: String, _addTime

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179796239 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -93,6 +94,9 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179803848 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -772,6 +772,12 @@ private[spark] class Executor( val accumUpdates

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179797706 --- Diff: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala --- @@ -251,6 +260,163 @@ class EventLoggingListenerSuite extends

[GitHub] spark pull request #20940: [SPARK-23429][CORE] Add executor memory metrics t...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20940#discussion_r179801207 --- Diff: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala --- @@ -251,6 +260,163 @@ class EventLoggingListenerSuite extends

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-06 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20987 > Things I'm concerned about is that does there exists a situation like 'a task gets killed after it gets a FetchFailure, but re-run later and gets a FetchFailure too without TaskKilledExcept

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20987#discussion_r179776800 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -494,19 +507,6 @@ private[spark] class Executor

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20987#discussion_r179776639 --- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala --- @@ -173,8 +173,26 @@ class ExecutorSuite extends SparkFunSuite

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20987#discussion_r179776576 --- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala --- @@ -330,6 +362,15 @@ class FetchFailureHidingRDD( case t

[GitHub] spark issue #20987: [SPARK-23816][CORE] Killed tasks should ignore FetchFail...

2018-04-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20987 pinging some potential reviewers: @tgravescs @kayousterhout @zsxwing @mridulm --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20987: [SPARK-23816][CORE] Killed tasks should ignore Fe...

2018-04-05 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20987 [SPARK-23816][CORE] Killed tasks should ignore FetchFailures. SPARK-19276 ensured that FetchFailures do not get swallowed by other layers of exception handling, but it also meant that a killed

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179585326 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint

[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data

2018-04-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19041 hey @brad-kaiser lemme temper what I said in my previous comments a bit -- I understand what you're doing here now and I think it makes sense, i don't see any serious design issues

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179508180 --- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data

2018-04-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19041 thanks for the updates @brad-kaiser. I think I understand and don't have any major concerns. It doesn't seem easy to use the LRU from MemoryStore, so can set that aside for now btw as you

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179266769 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -252,6 +257,44 @@ class BlockManagerMasterEndpoint

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179201817 --- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179200666 --- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179206819 --- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179206187 --- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179196510 --- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179180337 --- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala --- @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179171981 --- Diff: core/src/test/scala/org/apache/spark/scheduler/CacheRecoveryIntegrationSuite.scala --- @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179182498 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -420,63 +432,53 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-04-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r179183962 --- Diff: core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala --- @@ -449,18 +457,26 @@ class ExecutorAllocationManagerSuite

<    2   3   4   5   6   7   8   9   10   11   >