Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
This might be a really dumb question, but please humor me -- can you
explain to me the threads involved, both before and after this change? I must
be missing something, because it seems to me
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96889501
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -602,6 +619,20 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96883996
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,32 +245,45 @@ class
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
merged to master, thanks @jsoltren!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16639
cc @kayousterhout @markhamstra @mateiz
This isn't just protecting against crazy user code -- I've seen users hit
this with spark sql (because of
https://github.com/apache/spark/blob
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16639
[SPARK-19276][CORE] Fetch Failure handling robust to user error handling
## What changes were proposed in this pull request?
Fault-tolerance in spark requires special handling of shuffle
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
the test failures look real and from the changes you've made, can you take
a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96668414
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +91,15 @@ class BlacklistTrackerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96667288
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +162,47 @@ class ExecutorsListener(storageStatusListener
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96667888
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +162,47 @@ class ExecutorsListener(storageStatusListener
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16620
Thanks for pointing out this issue, and the nice description. Still
looking but sounds like a legitimate issue. I think
`SchedulerIntegrationSuite` should be able to replicate the exact scenario
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16473#discussion_r96459828
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -127,6 +127,14 @@ private[spark] object UIData {
def updateTaskMetrics
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96105066
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,43 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082178
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -38,4 +64,17 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082398
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -602,6 +616,20 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082685
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -517,6 +518,32 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96083494
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,32 +245,45 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082959
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,32 +245,45 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96082369
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
---
@@ -17,11 +17,37 @@
package
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96080614
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,43 @@ private[spark] class TaskDescription(
val
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
@ajbozarth whoops, I hadn't noticed the extra "Blacklisted" column in the
summary table at the top -- I was thinking we'd add another row for blacklisted
executors. But I actually think t
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
cc @ajbozarth as you were interested in this earlier
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95724900
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +88,86 @@ class BlacklistTrackerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95724311
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +158,42 @@ class ExecutorsListener(storageStatusListener
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95725456
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +88,86 @@ class BlacklistTrackerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95661186
--- Diff: docs/configuration.md ---
@@ -1625,40 +1625,40 @@ Apart from these, the following properties are also
available, and may be useful
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95624898
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
---
@@ -0,0 +1,158 @@
+Spark Auth Protocol and AES Encryption
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95625434
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
---
@@ -0,0 +1,158 @@
+Spark Auth Protocol and AES Encryption
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r95625631
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
---
@@ -0,0 +1,158 @@
+Spark Auth Protocol and AES Encryption
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16376
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95448131
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r95446452
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -819,4 +819,84 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95433750
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r95424970
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -54,7 +54,7 @@ import org.apache.spark.util.{AccumulatorV2
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95423240
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -149,7 +149,12 @@ private[spark] object Utils extends Logging
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
@witgo -- right, I'm saying that we should make local mode work to make
debugging & development as easy as possible, not to maximize performance. That
means we *should* do the extra serializa
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r95419719
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -54,7 +54,7 @@ import org.apache.spark.util.{AccumulatorV2
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95199529
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -592,47 +579,6 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95198725
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -149,7 +149,12 @@ private[spark] object Utils extends Logging
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95198044
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r95198208
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -52,7 +55,36 @@ private[spark] class TaskDescription(
val
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16376
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito closed the pull request at:
https://github.com/apache/spark/pull/16489
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16489
oh thanks @srowen , I didn't know it could do that. I'll update with a
larger list of fixes then
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16489
@shivaram @kayousterhout super minor follow up to
https://github.com/apache/spark/pull/16261 I just noticed.
---
If your project is set up for it, you can reply to this email and have your
reply
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16489
[SPARK-18836][MINOR] remove obsolete param doc for metrics
## What changes were proposed in this pull request?
4cb49412d1d7d10ffcc738475928c7de2bc59fd4 removed the `metrics` parameter
from
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16053
merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16053
lgtm, assuming tests pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16053
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16376#discussion_r94835320
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -159,7 +159,12 @@ private[spark] class TaskSetManager
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16376
Thanks for the feedback. I've updated the comment on `myLocalityLevels`.
Also I updated the tests slightly. To ensure that we're really testing no
delay, I updated the tests to use
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94670256
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15604
hmm, so I'm thinking about this again, along with your other pr
https://github.com/apache/spark/pull/15237
and it got me thinking -- there also aren't any tests for a mix of FIFO and
FAIR
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r94500060
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +176,39 @@ class PoolSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r94501620
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +103,52 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r94501723
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +103,52 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94499385
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94497574
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15604#discussion_r94499675
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +180,97 @@ class PoolSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15604
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
Looks like @kayousterhout posted some comments addressing my concerns on
https://github.com/apache/spark/pull/16053 at the same time as my last set of
comments. But essentially it sounds like Kay
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16376
[SPARK-18967][SCHEDULER] compute locality levels even if delay = 0
## What changes were proposed in this pull request?
Before this change, with delay scheduling off, spark would effectively
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16354#discussion_r93292503
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -549,11 +546,15 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16354
@mridulm @markhamstra @kayousterhout
This is *not* ready to merge -- it needs some cleanup and more tests -- but
I thought that seeing an implementation might help think through the design. I
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16354
[SPARK-18886][Scheduler][WIP] Adjust Delay scheduling to prevent
under-utilization of cluster
## What changes were proposed in this pull request?
This is a significant change to delay
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
@witgo @kayousterhout where do we stand on this and
https://github.com/apache/spark/pull/16053? Both still viable alternatives?
https://github.com/apache/spark/pull/16053 is still missing
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16053#discussion_r93080712
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -17,27 +17,119 @@
package org.apache.spark.scheduler
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r93079839
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -17,27 +17,179 @@
package org.apache.spark.scheduler
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r93078511
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -139,29 +139,6 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r93077759
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
---
@@ -59,6 +62,12 @@ private[spark] class LocalEndpoint
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
oops, thanks for letting me know @zsxwing , I just submitted
https://github.com/apache/spark/pull/16298
---
If your project is set up for it, you can reply to this email and have your
reply appear
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16298
[SPARK-8425][Scheduler][HOTFIX] fix scala 2.10 compile error
## What changes were proposed in this pull request?
https://github.com/apache/spark/commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
thanks @kayousterhout ! appreciate all the time you've spent helping out
on this issue.
merged to master
---
If your project is set up for it, you can reply to this email and have your
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16270
merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16270#discussion_r92284372
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -27,6 +27,8 @@ import scala.language.existentials
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
thanks for the review @kayousterhout. I also added a testcase to
BlacklistTrackerSuite, "task failure timeout works as expected for long-running
tasksets" to cover your point about the lo
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r92275871
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -408,6 +411,96 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r92271406
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -157,8 +160,16 @@ abstract class SchedulerIntegrationSuite[T
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16270
[SPARK-18846][Scheduler] Fix flakiness in SchedulerIntegrationSuite
## What changes were proposed in this pull request?
There is a small race in SchedulerIntegrationSuite.
The test
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r92267462
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -678,4 +716,13 @@ private[spark] object TaskSchedulerImpl
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r92254218
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -17,10 +17,254 @@
package org.apache.spark.scheduler
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r92229579
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -157,8 +160,16 @@ abstract class SchedulerIntegrationSuite[T
Github user squito commented on the issue:
https://github.com/apache/spark/pull/11105
I'm thinking about the binary compatability issue specifically with
`DoubleAccumulator` etc. Whether we intended it or not, `DoubleAccumulator` is
not `final`, so a user could have subclassed
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r90543844
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -20,15 +20,21 @@ package org.apache.spark.scheduler
import
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r90548446
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +105,55 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r90548133
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +105,55 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r90544307
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -20,15 +20,21 @@ package org.apache.spark.scheduler
import
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r90547665
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +105,55 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r90548835
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -178,4 +177,36 @@ class PoolSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r90543989
--- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala ---
@@ -74,30 +79,24 @@ class PoolSuite extends SparkFunSuite with
LocalSparkContext
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15237
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
ping @kayousterhout. I've merged in the other changes so this is up to
date now.
I also did another pass and did a bit of minor cleanup and commenting, I
think in line with stuff we've
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15982
I think the comment on `#mergeSpillsWithFileStreams` needs to be updated
slightly to include encryption, but other than that lgtm.
---
If your project is set up for it, you can reply to this email
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90121766
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java ---
@@ -337,42 +340,47 @@ void forceSorterToSpill() throws IOException
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90127615
--- Diff:
core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java
---
@@ -86,14 +88,7 @@ public int compare
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90127514
--- Diff:
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
---
@@ -75,13 +75,6 @@
@Mock(answer = RETURNS_SMART_NULLS
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90126726
--- Diff:
core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java
---
@@ -40,9 +41,11 @@
import org.mockito.stubbing.Answer
1501 - 1600 of 3521 matches
Mail list logo
