[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.

2017-02-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16978 lgtm assuming tests pass --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-02-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16867 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-02-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r102292628 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -137,6 +137,12 @@ private[spark] class TaskSetManager( // Task

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-02-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r102292835 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -911,14 +916,14 @@ private[spark] class TaskSetManager

[GitHub] spark pull request #16867: [SPARK-16929] Improve performance when check spec...

2017-02-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16867#discussion_r102293021 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -911,14 +916,14 @@ private[spark] class TaskSetManager

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-02-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16989 Jenkins, add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-02-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16989 Hi @jinxing64 I posted a comment on jira about the design -- I think this is a big enough change that its worth discussing the design first. Its fine to keep working on the code as a demonstration

[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

2017-02-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16892#discussion_r102282650 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

2017-02-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16892#discussion_r102281985 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -1569,24 +1569,44 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark issue #16901: [SPARK-19565] Improve DAGScheduler tests.

2017-02-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16901 sorry responding late to this, but your analysis sounds fine --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.

2017-02-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16978 code changes look fine, but is it possible to add a test case to HistoryServerSuite? I worry nobody is going to think about security / know how to test it so it may easily get broken. --- If your

[GitHub] spark issue #16639: [SPARK-19276][CORE] Fetch Failure handling robust to use...

2017-02-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16639 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16930: [SPARK-19597][CORE] test case for task deserialization e...

2017-02-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16930 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16639: [SPARK-19276][CORE] Fetch Failure handling robust...

2017-02-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16639#discussion_r101091688 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -400,8 +410,16 @@ private[spark] class Executor

[GitHub] spark pull request #16639: [SPARK-19276][CORE] Fetch Failure handling robust...

2017-02-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16639#discussion_r101090765 --- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala --- @@ -133,6 +123,153 @@ class ExecutorSuite extends SparkFunSuite

[GitHub] spark pull request #16930: [SPARK-19597][CORE] test case for task deserializ...

2017-02-14 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/16930 [SPARK-19597][CORE] test case for task deserialization errors Adds a task case that ensures that Executors gracefully handle a task that fails to deserialize, by sending back a reasonable failure

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r101070245 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1181,15 +1181,31 @@ class DAGScheduler( case smt

[GitHub] spark pull request #16901: [SPARK-19565] Improve DAGScheduler tests.

2017-02-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16901#discussion_r101064021 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2161,6 +2161,63 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #16901: [SPARK-19565] Improve DAGScheduler tests.

2017-02-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16901#discussion_r101061681 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2161,6 +2161,63 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #16901: [SPARK-19565] Improve DAGScheduler tests.

2017-02-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16901#discussion_r101059074 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2161,6 +2161,63 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark issue #16876: [SPARK-19537] Move pendingPartitions to ShuffleMapStage.

2017-02-09 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16876 lgtm when tests pass --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16813: [SPARK-19466][CORE][SCHEDULER] Improve Fair Scheduler Lo...

2017-02-09 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16813 @shaneknapp the build failed to trigger for me again, I did it manually via spark-prs.appspot.com --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16813: [SPARK-19466][CORE][SCHEDULER] Improve Fair Scheduler Lo...

2017-02-09 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16813 failure is probably unrelated Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #16876: [SPARK-19537] Move pendingPartitions to ShuffleMa...

2017-02-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16876#discussion_r100468844 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1038,9 +1038,8 @@ class DAGScheduler

[GitHub] spark pull request #16862: [SPARK-19520][streaming] Do not encrypt data writ...

2017-02-09 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16862#discussion_r100414780 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -758,9 +761,33 @@ private[spark] class BlockManager( blockId

[GitHub] spark issue #16650: [SPARK-16554][CORE] Automatically Kill Executors and Nod...

2017-02-09 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16650 merged to master, thanks @jsoltren --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16831: [SPARK-19263] Fix race in SchedulerIntegrationSuite.

2017-02-08 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16831 @jinxing64 that way of testing is fine, but I find its much faster to use sbt. http://www.scala-sbt.org/0.13/docs/Testing.html ``` build/sbt -Pyarn -Phadoop-2.6 -Phive

[GitHub] spark pull request #16639: [SPARK-19276][CORE] Fetch Failure handling robust...

2017-02-07 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16639#discussion_r99950321 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -405,6 +415,13 @@ private[spark] class Executor

[GitHub] spark issue #16639: [SPARK-19276][CORE] Fetch Failure handling robust to use...

2017-02-07 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16639 > (1) Instead of this approach, did you consider walking through the exceptions (with getCause()) to see if there's a nested FetchFailure in there? That seems simpler, with the con of miss

[GitHub] spark issue #16620: [SPARK-19263] DAGScheduler should avoid sending conflict...

2017-02-07 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16620 @kayousterhout I think your fix is correct, but I actually think its a bigger change in behavior, one that has been explicitly argued *against* in the past. I think the idea is that if you've got

[GitHub] spark issue #16831: [SPARK-19263] Fix race in SchedulerIntegrationSuite.

2017-02-07 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16831 lgtm sorry my fault in not breaking this out in the first place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16376: [SPARK-18967][SCHEDULER] compute locality levels even if...

2017-02-06 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16376 yes, I think this is ready (I just noticed a couple of minor nits with a fresh read but no real changes) --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r99664044 --- Diff: core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala --- @@ -467,6 +469,52 @@ class StandaloneDynamicAllocationSuite

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99618027 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2161,6 +2161,96 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99616688 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1191,8 +1191,29 @@ class DAGScheduler( } else

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99615449 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2161,6 +2161,96 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99615666 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2161,6 +2161,96 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99618127 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -648,4 +661,70 @@ class BasicSchedulerIntegrationSuite

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99615906 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2161,6 +2161,96 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99615141 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -14,7 +14,7 @@ * See the License for the specific language

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r99597739 --- Diff: core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala --- @@ -467,6 +469,51 @@ class StandaloneDynamicAllocationSuite

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r99595765 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -600,6 +603,16 @@ class

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r99596922 --- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala --- @@ -456,4 +461,69 @@ class BlacklistTrackerSuite extends

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-02-06 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r99596040 --- Diff: core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala --- @@ -467,6 +469,51 @@ class StandaloneDynamicAllocationSuite

[GitHub] spark issue #15237: [SPARK-17663] [CORE] SchedulableBuilder should handle in...

2017-02-06 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/15237 merged to master, thanks @erenavsarogullari --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-02-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r99474371 --- Diff: core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala --- @@ -489,6 +491,29 @@ class StandaloneDynamicAllocationSuite

[GitHub] spark issue #16620: [SPARK-19263] DAGScheduler should avoid sending conflict...

2017-02-03 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16620 Hi @jinxing64 I'm sorry I haven't had time to look again. So the one big concern I had was still that test case -- I know you fixed up some of the things I complained about, but I still

[GitHub] spark pull request #16781: [SPARK-12297][SQL][POC] Hive compatibility for Pa...

2017-02-02 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/16781 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #16781: [SPARK-12297][SQL][POC] Hive compatibility for Pa...

2017-02-02 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/16781 [SPARK-12297][SQL][POC] Hive compatibility for Parquet Timestamps ## What changes were proposed in this pull request? Hive has very strange behavior when writing timestamps to parquet data

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r98699067 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1212,8 +1223,9 @@ class DAGScheduler

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r98703486 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -68,6 +68,12 @@ private[scheduler] abstract class Stage( /** Set of jobs

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r98703683 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -648,4 +660,70 @@ class BasicSchedulerIntegrationSuite

[GitHub] spark issue #16620: [SPARK-19263] DAGScheduler should avoid sending conflict...

2017-01-31 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16620 Hi @jinxing64 sorry to go back and forth on this numerous times -- I think I have another alternative, see https://github.com/squito/spark/tree/SPARK-19263_alternate Its most of your

[GitHub] spark issue #15237: [SPARK-17663] [CORE] SchedulableBuilder should handle in...

2017-01-30 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/15237 @erenavsarogullari sorry to push back, but if you're willing to do the filename thing now, why not just tackle it in this same pr? seems pretty minor to separate into its own issue

[GitHub] spark pull request #15505: [SPARK-18890][CORE] Move task serialization from ...

2017-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r98591359 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -51,8 +54,39 @@ private[spark] class TaskDescription( val

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r98472355 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -718,6 +703,21 @@ private[spark] class TaskSetManager

[GitHub] spark pull request #15237: [SPARK-17663] [CORE] SchedulableBuilder should ha...

2017-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15237#discussion_r98464559 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala --- @@ -102,38 +103,51 @@ private[spark] class FairSchedulableBuilder(val

[GitHub] spark pull request #15237: [SPARK-17663] [CORE] SchedulableBuilder should ha...

2017-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15237#discussion_r98464730 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala --- @@ -102,38 +103,51 @@ private[spark] class FairSchedulableBuilder(val

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r98281403 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -173,6 +174,17 @@ private[scheduler] class BlacklistTracker

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r98283147 --- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala --- @@ -456,4 +459,61 @@ class BlacklistTrackerSuite extends

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r98281375 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -173,6 +174,17 @@ private[scheduler] class BlacklistTracker

[GitHub] spark issue #16650: [SPARK-16554][CORE] Automatically Kill Executors and Nod...

2017-01-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16650 I think this is basically ready, perhaps one more test case. @tgravescs any other comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r98281415 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -187,6 +199,17 @@ private[scheduler] class BlacklistTracker

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r98281426 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -187,6 +199,17 @@ private[scheduler] class BlacklistTracker

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r98283223 --- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala --- @@ -17,7 +17,8 @@ package org.apache.spark.scheduler

[GitHub] spark issue #16650: [SPARK-16554][CORE] Automatically Kill Executors and Nod...

2017-01-25 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16650 one more thought on tests -- your mock `ExecutorAllocationClient` could make sure that when `killExecutorsOnHost` is called, the host is already in `nodeBlacklist`, to avoid that last race you found

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97875223 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -139,6 +139,11 @@ package object config { .timeConf

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97875600 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97875716 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -187,6 +206,22 @@ private[scheduler] class BlacklistTracker

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97881401 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -148,6 +153,12 @@ class

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97878659 --- Diff: core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala --- @@ -528,6 +548,16 @@ class StandaloneDynamicAllocationSuite

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97877918 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -148,6 +153,12 @@ class

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97874677 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala --- @@ -59,6 +59,19 @@ private[spark] trait ExecutorAllocationClient

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97877539 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97881356 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97881374 --- Diff: core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala --- @@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker

[GitHub] spark issue #16690: [SPARK-19347] ReceiverSupervisorImpl can add block to Re...

2017-01-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16690 cc @vanzin @zsxwing (I was going to make a longer comment about removing `askWithRetry` and whether we need another method, but then saw the comments on the referenced PR -- I'll defer

[GitHub] spark issue #16690: [SPARK-19347] ReceiverSupervisorImpl can add block to Re...

2017-01-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16690 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r97406162 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1218,7 +1225,9 @@ class DAGScheduler( logInfo

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r97586026 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1193,7 +1193,14 @@ class DAGScheduler

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r97417513 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -648,4 +648,69 @@ class BasicSchedulerIntegrationSuite

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-01-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r97417399 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -648,4 +648,69 @@ class BasicSchedulerIntegrationSuite

[GitHub] spark issue #11867: [SPARK-14049] [CORE] Add functionality in spark history ...

2017-01-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/11867 thanks @paragpc merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #11867: [SPARK-14049] [CORE] Add functionality in spark history ...

2017-01-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/11867 lgtm, will merge when tests pass --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16620: [SPARK-19263] DAGScheduler should avoid sending conflict...

2017-01-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16620 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #15505: [SPARK-18890][CORE] Move task serialization from ...

2017-01-23 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r97395812 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -602,6 +619,21 @@ class

[GitHub] spark pull request #11867: [SPARK-14049] [CORE] Add functionality in spark h...

2017-01-23 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11867#discussion_r97391985 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala --- @@ -43,11 +45,26 @@ private[v1] class ApplicationListResource

[GitHub] spark pull request #11867: [SPARK-14049] [CORE] Add functionality in spark h...

2017-01-23 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11867#discussion_r97392343 --- Diff: docs/monitoring.md --- @@ -267,12 +267,23 @@ can be identified by their `[attempt-id]`. In the API listed below, when running

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97162877 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -719,7 +719,11 @@ private[spark] object TaskSchedulerImpl

[GitHub] spark issue #16650: [SPARK-16554][CORE] Automatically Kill Executors and Nod...

2017-01-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16650 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97162721 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -429,7 +429,7 @@ class TaskSetManagerSuite extends SparkFunSuite

[GitHub] spark pull request #16650: [SPARK-16554][CORE] Automatically Kill Executors ...

2017-01-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16650#discussion_r97162706 --- Diff: core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala --- @@ -43,7 +43,7 @@ class BlacklistTrackerSuite extends SparkFunSuite

[GitHub] spark issue #16473: [SPARK-19069] [CORE] Expose task 'status' and 'duration'...

2017-01-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16473 merged to master, thanks @paragpc ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16473: [SPARK-19069] [CORE] Expose task 'status' and 'duration'...

2017-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16473 ah, sorry I didn't notice that before my comment. well looks good now, I'll merge once tests pass. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #11867: [SPARK-14049] [CORE] Add functionality in spark h...

2017-01-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11867#discussion_r96939526 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala --- @@ -43,11 +45,24 @@ private[v1] class ApplicationListResource

[GitHub] spark issue #16473: [SPARK-19069] [CORE] Expose task 'status' and 'duration'...

2017-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16473 thanks @paragpc . lgtm, but now there are merge conflicts. Can you merge with the latest master? You might need to regenerate those test expectation files. --- If your project is set up

[GitHub] spark issue #16639: [SPARK-19276][CORE] Fetch Failure handling robust to use...

2017-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16639 thanks for the feedback @mridulm , all good points. I pushed an update to address some of the points, also have some follow up discussion --- If your project is set up for it, you can reply

[GitHub] spark pull request #16639: [SPARK-19276][CORE] Fetch Failure handling robust...

2017-01-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16639#discussion_r96910697 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -405,6 +415,13 @@ private[spark] class Executor

[GitHub] spark pull request #16639: [SPARK-19276][CORE] Fetch Failure handling robust...

2017-01-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16639#discussion_r96910555 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -405,6 +415,13 @@ private[spark] class Executor

[GitHub] spark pull request #16639: [SPARK-19276][CORE] Fetch Failure handling robust...

2017-01-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16639#discussion_r96909723 --- Diff: core/src/main/scala/org/apache/spark/shuffle/FetchFailedException.scala --- @@ -45,6 +45,12 @@ private[spark] class FetchFailedException

<    10   11   12   13   14   15   16   17   18   19   >