Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/1297#discussion_r17790219
--- Diff: core/src/main/scala/org/apache/spark/rdd/IndexedRDDLike.scala ---
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/1297#discussion_r17791303
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ImmutableLongOpenHashSet.scala
---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/1297#issuecomment-56199798
This looks great! my comments are minor.
I know its early to be discussing example docs, but I just wanted to
mention that I can see caching being an area of
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16650
merged to master, thanks @jsoltren
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16862#discussion_r100414780
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -758,9 +761,33 @@ private[spark] class BlockManager(
blockId
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16876#discussion_r100468844
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1038,9 +1038,8 @@ class DAGScheduler(
}
if
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16813
failure is probably unrelated
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16813
@shaneknapp the build failed to trigger for me again, I did it manually via
spark-prs.appspot.com
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16876
lgtm when tests pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16901#discussion_r101059074
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2161,6 +2161,63 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16901#discussion_r101061681
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2161,6 +2161,63 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16901#discussion_r101064021
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2161,6 +2161,63 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16620#discussion_r101070245
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1181,15 +1181,31 @@ class DAGScheduler(
case smt
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16930
[SPARK-19597][CORE] test case for task deserialization errors
Adds a task case that ensures that Executors gracefully handle a task that
fails to deserialize, by sending back a reasonable failure
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16639#discussion_r101090765
--- Diff: core/src/test/scala/org/apache/spark/executor/ExecutorSuite.scala
---
@@ -133,6 +123,153 @@ class ExecutorSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16639#discussion_r101091688
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -400,8 +410,16 @@ private[spark] class Executor
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16930
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16639
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16978
code changes look fine, but is it possible to add a test case to
HistoryServerSuite? I worry nobody is going to think about security / know how
to test it so it may easily get broken.
---
If your
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16901
sorry responding late to this, but your analysis sounds fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16892#discussion_r102281985
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -1569,24 +1569,44 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16892#discussion_r102282650
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16989
Hi @jinxing64 I posted a comment on jira about the design -- I think this
is a big enough change that its worth discussing the design first. Its fine to
keep working on the code as a demonstration
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16989
Jenkins, add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16867#discussion_r102293021
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -911,14 +916,14 @@ private[spark] class TaskSetManager
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16867#discussion_r102292835
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -911,14 +916,14 @@ private[spark] class TaskSetManager
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16867
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16867#discussion_r102292628
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -137,6 +137,12 @@ private[spark] class TaskSetManager(
// Task
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16978
lgtm assuming tests pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16946#discussion_r102546890
--- Diff: docs/running-on-yarn.md ---
@@ -604,3 +604,17 @@ spark.yarn.am.extraJavaOptions
-Dsun.security.krb5.debug=true -Dsun.security.spn
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16946#discussion_r102539475
--- Diff:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnProxyRedirectFilterSuite.scala
---
@@ -0,0 +1,53
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16892#discussion_r102996593
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16930
thanks @kayousterhout !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15604
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94497574
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite with
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94499675
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite with
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94499385
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite with
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r94501620
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +103,52 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r94501723
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +103,52 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r94500060
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +176,39 @@ class PoolSuite extends SparkFunSuite with
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15604
hmm, so I'm thinking about this again, along with your other pr
https://github.com/apache/spark/pull/15237
and it got me thinking -- there also aren't any tests for a mix of FIFO
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94670256
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite with
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16376
Thanks for the feedback. I've updated the comment on `myLocalityLevels`.
Also I updated the tests slightly. To ensure that we're really testing no
delay, I updated the test
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r94835320
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -159,7 +159,12 @@ private[spark] class TaskSetManager
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16053
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16053
lgtm, assuming tests pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16053
merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16489
[SPARK-18836][MINOR] remove obsolete param doc for metrics
## What changes were proposed in this pull request?
4cb49412d1d7d10ffcc738475928c7de2bc59fd4 removed the `metrics` parameter
from
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16489
@shivaram @kayousterhout super minor follow up to
https://github.com/apache/spark/pull/16261 I just noticed.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16489
oh thanks @srowen , I didn't know it could do that. I'll update with a
larger list of fixes then
---
If your project is set up for it, you can reply to this email and have your
reply
Github user squito closed the pull request at:
https://github.com/apache/spark/pull/16489
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16376
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95198208
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95198044
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95198725
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -149,7 +149,12 @@ private[spark] object Utils extends Logging
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95199529
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -592,47 +579,6 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r95419719
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -54,7 +54,7 @@ import org.apache.spark.util.{AccumulatorV2
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
@witgo -- right, I'm saying that we should make local mode work to make
debugging & development as easy as possible, not to maximize performance. That
means we *should* do the extra seri
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95423240
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -149,7 +149,12 @@ private[spark] object Utils extends Logging
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r95424970
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -54,7 +54,7 @@ import org.apache.spark.util.{AccumulatorV2
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95433750
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r95446452
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -819,4 +819,84 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95448131
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16376
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95625631
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
---
@@ -0,0 +1,158 @@
+Spark Auth Protocol and AES Encryption
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95625434
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
---
@@ -0,0 +1,158 @@
+Spark Auth Protocol and AES Encryption
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95624898
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
---
@@ -0,0 +1,158 @@
+Spark Auth Protocol and AES Encryption
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95661186
--- Diff: docs/configuration.md ---
@@ -1625,40 +1625,40 @@ Apart from these, the following properties are also
available, and may be useful
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95725456
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +88,86 @@ class BlacklistTrackerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95724900
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +88,86 @@ class BlacklistTrackerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95724311
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +158,42 @@ class ExecutorsListener(storageStatusListener
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
cc @ajbozarth as you were interested in this earlier
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
@ajbozarth whoops, I hadn't noticed the extra "Blacklisted" column in the
summary table at the top -- I was thinking we'd add another row for blacklisted
executors. But I actual
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96080614
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,43 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082369
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -17,11 +17,37 @@
package
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082959
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,32 +245,45 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96083494
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,32 +245,45 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082685
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -517,6 +518,32 @@ class DAGSchedulerSuite extends SparkFunSuite with
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082178
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -38,4 +64,17 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082398
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -602,6 +616,20 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96105066
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,43 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81859662
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81861857
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81864312
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -98,6 +84,14 @@ private[spark] class TaskSetManager(
var
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81864626
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -478,16 +473,18 @@ private[spark] class TaskSetManager
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81865871
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -766,9 +782,11 @@ private[spark] class TaskSetManager
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81867147
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -766,9 +782,11 @@ private[spark] class TaskSetManager
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81867460
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistIntegrationSuite.scala
---
@@ -51,37 +54,67 @@ class BlacklistIntegrationSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81868412
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81893799
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -809,32 +816,65 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81895140
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81896656
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15249
@mridulm on the questions about expiry from blacklists, you are not missing
anything -- this explictly does not do any timeouts at the taskset level (this
is mentioned in the design doc). The
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81898588
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15249
@kayousterhout @mridulm thanks for the feedback. obviously still need to
figure out the timeout thing but otherwise think I've addressed things. will
do another pass in the morning.
---
If
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r81999277
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -592,34 +589,54 @@ private[spark] class TaskSetManager
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15249#discussion_r82007301
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -809,32 +816,65 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15249
I forgot to add that I had turned off blacklisting by default, I agree with
your suggestion Kay. I pushed another commit which updated the docs as well.
There are some other small style things and
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15335#discussion_r82021050
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1255,27 +1255,46 @@ class DAGScheduler(
s"l
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15335#discussion_r82010161
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1255,27 +1255,46 @@ class DAGScheduler(
s"l
1 - 100 of 3535 matches
Mail list logo
