Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15249
As I mentioned before, this is definitely a huge step in the right
direction !
Having said that, I want to ensure we dont aggressively blacklist executors
and nodes - at scale, I have seen
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15218
@zhzhan I am curious why this is the case for the jobs being mentioned.
This pr should have an impact if the locality preference of the taskset
being run is fairly suboptimal to begin with, no
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15408#discussion_r82664592
--- Diff:
core/src/main/java/org/apache/spark/io/NioBasedBufferedFileInputStream.java ---
@@ -0,0 +1,120 @@
+/*
+ * Licensed under the Apache
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15408#discussion_r82665139
--- Diff:
core/src/main/java/org/apache/spark/io/NioBasedBufferedFileInputStream.java ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed under the Apache
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15408#discussion_r82665543
--- Diff:
core/src/main/java/org/apache/spark/io/NioBasedBufferedFileInputStream.java ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed under the Apache
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15408#discussion_r82665886
--- Diff:
core/src/main/java/org/apache/spark/io/NioBasedBufferedFileInputStream.java ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed under the Apache
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15408#discussion_r8279
--- Diff:
core/src/main/java/org/apache/spark/io/NioBasedBufferedFileInputStream.java ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed under the Apache
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15408
There is a behavioral change with this PR, which I am not sure is relevant.
BufferedInputStream supports mark/reset, while we are not doing so here -
does deserialization and other codepaths
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15371#discussion_r82670700
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -444,7 +444,9 @@ class CollectionAccumulator[T] extends AccumulatorV2[T
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15371#discussion_r82683704
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -444,7 +444,9 @@ class CollectionAccumulator[T] extends AccumulatorV2[T
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15408
Barring query to @rxin (regarding buffer pooling), I am fine with the
change - pretty neat, thanks @sitalkedia !
Would be good if more eyeballs look at it though given how fundamental it
is
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15408#discussion_r82816160
--- Diff:
core/src/main/java/org/apache/spark/io/NioBufferedFileInputStream.java ---
@@ -0,0 +1,129 @@
+/*
+ * Licensed under the Apache License
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15422
Why would corrupt record cause EOFException to be thrown ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15422
@srowen The tuples already returned would have been valid, it is the
subsequent block decompression which has failed. For example, in a 1gb file,
the last few bytes missing (or corrupt) will cause
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15422
@zsxwing You are right, NewHadoopRDD is not handling this case.
Probably would be good to add exception handling there when nextKeyValue
throws exception ?
Context is, for large jobs
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15422
@marmbrus +1 on logging, that is definitely something which was probably
missed here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15422
@zsxwing The map task is run by
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15422
@srowen Since this is happening 'below' the user code (in the hadoop rdd),
is there a way around how to handle this ?
I agree that for a lot of usecases where it is critical to wo
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/12524
@seayi any progress on this ? Would be good to add this in if consistently
reproducible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15422#discussion_r82932992
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -179,7 +183,16 @@ class NewHadoopRDD[K, V](
override def
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15422#discussion_r82932645
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -253,8 +256,12 @@ class HadoopRDD[K, V](
try {
finished
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15422#discussion_r82932947
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -588,6 +588,12 @@ object SQLConf {
.doubleConf
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15422#discussion_r82933077
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -170,4 +170,9 @@ package object config {
.doc("Port to us
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15422
Merged - had issue with pip (new laptop, sigh), and so jira and pr did not
get closed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15454#discussion_r83155108
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -245,8 +248,7 @@ class HadoopRDD[K, V](
try {
finished
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15454#discussion_r83155316
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -171,7 +175,11 @@ class NewHadoopRDD[K, V](
override def
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
Would be cleaner to simply copy executorDataMap.keys and works off that to
ensure there is no coupling between actor thread and invoker.
---
If your project is set up for it, you can reply to this
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15218
I am assuming @kayousterhout does not have comments on this.
Can you please fix the conflict @zhzhan ? I will merge it in after that to
master.
---
If your project is set up for it, you can
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15218
Merged to master, thanks @zhzhan !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15512#discussion_r83752469
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala ---
@@ -84,6 +90,7 @@ private[spark] class TaskResultGetter(sparkEnv
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15531
@srowen Unfortunately the 'Some' in the == is usually missed out, resulting
in bugs (and we have had a fair share of them in the past - some more severe
than others).
Given that t
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15382#discussion_r83928417
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -741,7 +741,7 @@ private[sql] class SQLConf extends Serializable with
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
@scwf I think the initial fix with a small change might be sufficient.
What I meant was something like this :
```
protected def reset(): Unit = {
val executors
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
@zsxwing Ah, then simply making it send() instead of askWithRetry() should
do, no ?
That was actually in the initial PR - I was not sure if we want to change
the behavior from askWithRetry to
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
LGTM, @zsxwing any comments ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
BTW, it was interesting that the earlier change did not trigger a test
failure (the issue @viirya pointed out - about needing to move RemoveExecutor
to receive)
---
If your project is set up for
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15550#discussion_r84004989
--- Diff: core/src/main/scala/org/apache/spark/rdd/ZippedWithIndexRDD.scala
---
@@ -64,8 +64,14 @@ class ZippedWithIndexRDD[T: ClassTag](prev: RDD[T
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15553
+CC @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
GitHub user mridulm opened a pull request:
https://github.com/apache/spark/pull/15553
[SPARK-18008] [build] Add support for -Dmaven.test.skip=true and
-Dmaven.javadoc.skip=true
## What changes were proposed in this pull request?
Add support for -Dmaven.test.skip=true and
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15553#discussion_r84047441
--- Diff: common/network-common/pom.xml ---
@@ -77,27 +77,40 @@
compile
-
-
- log4j
- log4j
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15553#discussion_r84047904
--- Diff: pom.xml ---
@@ -2415,6 +2389,67 @@
+
+ docBuild
+
+
+ maven.javadoc.skip
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15553#discussion_r84049757
--- Diff: sql/hive-thriftserver/pom.xml ---
@@ -41,11 +41,8 @@
${project.version}
- org.apache.spark
- spark
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15553
@srowen You are right; for normal build, this should be a no-op.
Currently, there is no way to suppress compilation of tests (we can
suppress running test via -DskipTests but it will still
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15553
Thanks for the link to 'skip' - will test it out !
About test compilation :
For example, in common/network-shuffle/pom.xm
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15553
Please note that this applies only the spark artifacts - not any of the
others (which, as you mentioned, will be a simple disk lookup).
Since I was moving it for spark generated artifacts anyway
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15553#discussion_r84057228
--- Diff: sql/hive-thriftserver/pom.xml ---
@@ -41,11 +41,8 @@
${project.version}
- org.apache.spark
- spark
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15553#discussion_r84081078
--- Diff: sql/hive-thriftserver/pom.xml ---
@@ -41,11 +41,8 @@
${project.version}
- org.apache.spark
- spark
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84154979
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala
---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84160226
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala
---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
@zsxwing To minimize scope of synchronized block.
The way @scwf has now, the synchronized block is limited to duplicating key
and setting some state.
Remaining can happen outside of the lock
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
@zsxwing I think the issue is that case RemoveExecutor() is not identical
to what exists in receiveAndReply
Any reason
'executorDataMap.get(executorId).foreach(_.executorEndpoint
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15481
Ah ! Apologies, I got confused. Yes, I agree, that is a better approach.
It also means we can get rid of the RemoveExecutor pattern match from
receive right ? As it stands now, that looks
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/15541#discussion_r84225583
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala
---
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106779213
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -17,48 +17,61 @@
package org.apache.spark.storage
-import
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106779546
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -73,55 +86,219 @@ private[spark] class DiskStore(conf: SparkConf,
diskManager
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106779004
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -34,6 +34,8 @@ import org.apache.spark.util.{ShutdownHookManager
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106269093
--- Diff:
core/src/main/scala/org/apache/spark/security/CryptoStreamUtils.scala ---
@@ -102,4 +150,34 @@ private[spark] object CryptoStreamUtils extends
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r10677
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -94,7 +101,11 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106778932
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -73,55 +86,219 @@ private[spark] class DiskStore(conf: SparkConf,
diskManager
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106779457
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -73,55 +86,219 @@ private[spark] class DiskStore(conf: SparkConf,
diskManager
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106264689
--- Diff:
core/src/main/scala/org/apache/spark/security/CryptoStreamUtils.scala ---
@@ -63,12 +83,40 @@ private[spark] object CryptoStreamUtils extends
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106778914
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -17,48 +17,61 @@
package org.apache.spark.storage
-import
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106778813
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -73,55 +86,219 @@ private[spark] class DiskStore(conf: SparkConf,
diskManager
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106268712
--- Diff:
core/src/main/scala/org/apache/spark/security/CryptoStreamUtils.scala ---
@@ -63,12 +83,40 @@ private[spark] object CryptoStreamUtils extends
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106063310
--- Diff:
core/src/main/scala/org/apache/spark/security/CryptoStreamUtils.scala ---
@@ -48,12 +50,30 @@ private[spark] object CryptoStreamUtils extends
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106779317
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -73,55 +86,219 @@ private[spark] class DiskStore(conf: SparkConf,
diskManager
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106778005
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -56,6 +57,43 @@ private[spark] class BlockResult(
val bytes: Long
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106778650
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -79,6 +81,11 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106778688
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -17,48 +17,61 @@
package org.apache.spark.storage
-import
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106778760
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala ---
@@ -73,55 +86,219 @@ private[spark] class DiskStore(conf: SparkConf,
diskManager
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/16867
LGTM @kayousterhout , @squito.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17290#discussion_r106788142
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala ---
@@ -340,7 +340,7 @@ private[storage] class BlockInfoManager extends
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r106788877
--- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala ---
@@ -212,8 +212,8 @@ case object TaskResultLost extends TaskFailedReason
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r106789380
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -540,6 +540,39 @@ class SparkContextSuite extends SparkFunSuite with
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r106788727
--- Diff: core/src/main/scala/org/apache/spark/ui/UIUtils.scala ---
@@ -354,7 +354,7 @@ private[spark] object UIUtils extends Logging
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r106789297
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -540,6 +540,39 @@ class SparkContextSuite extends SparkFunSuite with
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r106789004
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -467,7 +474,7 @@ private[spark] class TaskSchedulerImpl
private
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17343
If we make flush() noop, then buffered (uncommitted) data wont be written
to the stream; am I missing something here, or is this change broken ?
---
If your project is set up for it, you can reply
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17343
Background - you need to do a flush() to ensure the indices generated are
valid.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17343
Ah, looks like I missed that CountingOutputStream was introduced after BOS
and not before.
Looks good to me.
---
If your project is set up for it, you can reply to this email and have your
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107233146
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -302,12 +298,12 @@ private[spark] class Executor(
// If
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107239694
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -467,7 +474,7 @@ private[spark] class TaskSchedulerImpl
private
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107235395
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -569,8 +575,10 @@ class SparkContextSuite extends SparkFunSuite with
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107234055
--- Diff: core/src/main/scala/org/apache/spark/TaskContextImpl.scala ---
@@ -59,8 +59,8 @@ private[spark] class TaskContextImpl(
/** List of callback
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107234445
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -160,15 +160,20 @@ private[spark] abstract class Task[T](
// A flag
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107237325
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -215,7 +215,8 @@ private[spark] class PythonRunner
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107235703
--- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala ---
@@ -212,8 +212,8 @@ case object TaskResultLost extends TaskFailedReason
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107232894
--- Diff: core/src/main/scala/org/apache/spark/TaskContextImpl.scala ---
@@ -140,16 +140,22 @@ private[spark] class TaskContextImpl
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17290
I agree with @srowen I dont see how this change affects the test.
`blocksWithReleasedLocks` should be unchanged w.r.t this test.
---
If your project is set up for it, you can reply to this email
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17166
Hi @kayousterhout,
Can you take over reviewing this PR ? I might be tied up with other
things for next couple of weeks, and I dont want @ericl's work to be blocked on
me.
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107345386
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -467,7 +474,7 @@ private[spark] class TaskSchedulerImpl
private
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17300#discussion_r107352668
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -555,12 +555,15 @@ private[spark] class BlockManager
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17300#discussion_r107352378
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -500,6 +500,30 @@ class BlockManagerSuite extends SparkFunSuite with
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107631487
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -467,7 +474,7 @@ private[spark] class TaskSchedulerImpl
private
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107773629
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -467,7 +474,7 @@ private[spark] class TaskSchedulerImpl
private
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107797382
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -215,7 +215,7 @@ private[spark] class PythonRunner
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17166#discussion_r107797887
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -467,7 +479,7 @@ private[spark] class TaskSchedulerImpl
private
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17343
LGTM will wait a bit to allow for others to comment.
@zsxwing can you also take a look ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17276#discussion_r108052747
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
---
@@ -169,6 +173,36 @@ public void write(Iterator> reco
Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17276#discussion_r108052753
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
---
@@ -169,6 +173,36 @@ public void write(Iterator> reco
301 - 400 of 1358 matches
Mail list logo
