Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/23223
Hi @tgravescs , I tried it, but found it's difficult to produce
KILLED_BY_RESOURCEMANAGER exit status. I followed
[YARN-73](https://issues.apache.org/jira/browse/YARN-73)
[YARN-495](https
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/23223
> it would be interesting to test it further to see if it does.
@tgravescs Yeah, I have the same thought. I'd like to try it, but I can
not guarantee that I can achieve it... Becaus
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23223#discussion_r239341126
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -612,11 +612,14 @@ private[yarn] class
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23223#discussion_r239316608
--- Diff:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
---
@@ -417,4 +426,59 @@ class YarnAllocatorSuite
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23223#discussion_r239316424
--- Diff:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
---
@@ -114,13 +116,20 @@ class YarnAllocatorSuite
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/23223
> it looks like its only going to blacklist the node for the AM, not other
nodes for general containers.
@squito Yarn have blacklist for AM when config
`am-scheduling.node-blacklist
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/23223
> Are you seeing actual issues with this blacklisting when it shouldn't?
Unfortunately, no. @tgravescs @squ
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/23223
> I mean if node blacklisting in Spark would be perfectly aligned to YARN
then it would be just redundant to have it in Spark in the first place.
This change seems result in *perfec
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/23223
ping @attilapiros @vanzin @jerryshao for kindly review.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/23223
Yarnallocator should have same blacklist behaviour with yarn to maxmize use
of cluster resource
## What changes were proposed in this pull request?
As I mentioned in jira
[SPARK-26269
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20269#discussion_r238135789
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -38,10 +38,13 @@ package object config {
ConfigBuilder
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/22288
(I'm on a outside trip these days, so I have to use my mobile phone to type
these words. Sorry for the format.)
> Is this the same as the current pr, but just killing only if i
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/22288
As I mentioned at
https://github.com/apache/spark/pull/22288#discussion_r216874530, I'm quite
worry about this killing behaviour. I thik we should kill a executor iff it is
idle
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22288#discussion_r216874530
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -414,9 +425,48 @@ private[spark] class TaskSchedulerImpl
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18492#discussion_r216597619
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -373,8 +373,14 @@ private[spark] class ExecutorAllocationManager
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22288#discussion_r214720097
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -414,9 +425,54 @@ private[spark] class TaskSchedulerImpl
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22288#discussion_r214719743
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -414,9 +425,54 @@ private[spark] class TaskSchedulerImpl
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/22202
Since `stage 1` is only a `ShuffleMapStage`, so, why there're no other
child stages to be submitted ?
---
-
To unsubscribe, e
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r212167438
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r212163785
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r212160161
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r211954019
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22001#discussion_r209662081
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -402,6 +422,19 @@ class DAGScheduler
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22001#discussion_r209658945
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -929,6 +955,28 @@ class DAGScheduler(
// HadoopRDD whose
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21565#discussion_r200019612
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -488,9 +488,16 @@ private[spark] class ExecutorAllocationManager
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21565#discussion_r195927590
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -488,9 +488,16 @@ private[spark] class ExecutorAllocationManager
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21486#discussion_r194606075
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -197,14 +197,14 @@ private[spark] class HeartbeatReceiver(sc:
SparkContext
Github user Ngone51 closed the pull request at:
https://github.com/apache/spark/pull/21096
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user Ngone51 closed the pull request at:
https://github.com/apache/spark/pull/20996
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21494
Hi, @jiangxb1987 , can you explain more for what is `barrier scheduling` in
spark and elaborate an example which would only works with `barrier
scheduling`( but could not work under current spark
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21486#discussion_r192592142
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala
---
@@ -207,6 +210,55 @@ class HeartbeatReceiverSuite
assert
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21486#discussion_r192592170
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala
---
@@ -207,6 +210,55 @@ class HeartbeatReceiverSuite
assert
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21486#discussion_r192591845
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala
---
@@ -207,6 +210,55 @@ class HeartbeatReceiverSuite
assert
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191623277
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191178697
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191182696
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191176828
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191175242
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala
---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191175890
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191117686
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala
---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191175960
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferFileRegionSuite.scala
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r191104760
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala
---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189939603
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -267,7 +273,7 @@ class ExternalAppendOnlyMap[K, V, C
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189894423
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -414,6 +415,99 @@ class ExternalAppendOnlyMapSuite
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189892444
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -267,7 +273,7 @@ class ExternalAppendOnlyMap[K, V, C
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189892547
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +591,25 @@ class ExternalAppendOnlyMap[K, V, C
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21369
cc @JerryLead
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21369#discussion_r189438190
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -585,17 +592,15 @@ class ExternalAppendOnlyMap[K, V, C
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187825561
--- Diff: core/src/main/scala/org/apache/spark/Heartbeater.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187824094
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1753,9 +1766,21 @@ class DAGScheduler
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187823298
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -169,6 +179,27 @@ private[spark] class EventLoggingListener
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187248156
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -93,6 +94,10 @@ private[spark] class EventLoggingListener
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187236701
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1753,9 +1766,21 @@ class DAGScheduler
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187239219
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -169,6 +179,27 @@ private[spark] class EventLoggingListener
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187244792
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -169,6 +179,27 @@ private[spark] class EventLoggingListener
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187238940
--- Diff: core/src/main/scala/org/apache/spark/Heartbeater.scala ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r187247534
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/PeakExecutorMetrics.scala ---
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21209
Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20604#discussion_r186425765
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1643,7 +1646,10 @@ class SparkContext(config: SparkConf) extends
Logging
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21096
Thanks for your opinions @squito @markhamstra .
Maybe, I should leave it for now.
---
-
To unsubscribe, e-mail
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21209
ping @jiangxb1987
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21212#discussion_r186261650
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -267,28 +269,30 @@ final class ShuffleBlockFetcherIterator
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21209
ping @squito @vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/21209
[SPARK-24141][CORE] Fix bug in CoarseGrainedSchedulerBackend.killExecutors
## What changes were proposed in this pull request?
In method *CoarseGrainedSchedulerBackend.killExecutors
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20604#discussion_r185159109
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1643,7 +1646,10 @@ class SparkContext(config: SparkConf) extends
Logging
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20604
ping @squito
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21175
cc @kiszk @maropu @cloud-fan @jiangxb1987
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184607965
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -56,6 +56,12 @@ class ChunkedByteBufferSuite extends SparkFunSuite
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184597197
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -56,6 +56,12 @@ class ChunkedByteBufferSuite extends SparkFunSuite
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184596199
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -56,6 +56,12 @@ class ChunkedByteBufferSuite extends SparkFunSuite
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21175#discussion_r184590989
--- Diff:
core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala ---
@@ -56,6 +56,12 @@ class ChunkedByteBufferSuite extends SparkFunSuite
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21131
LGTM, and nice UT.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21175
@manbuyun you need to add the unit test into `ChunkedByteBufferSuite.scala`
and push a new commit.
---
-
To unsubscribe, e-mail
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20930
No wonder I can't understand the issue for a long time since I've thought
it happened on Spark2.3 . And now it makes sense. Thanks @jiangxb1987
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20604
ping @squito
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21131#discussion_r183797532
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21131#discussion_r183790463
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21131#discussion_r183789814
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21131#discussion_r183701269
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21131#discussion_r183690133
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21131#discussion_r183619646
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -689,6 +689,20 @@ private[spark] class TaskSchedulerImpl
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21131#discussion_r183619704
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -689,6 +689,20 @@ private[spark] class TaskSchedulerImpl
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20998
Agree and thank you @squito .
And thanks for all of you. @felixcheung @mridulm @jiangxb1987 @srowen
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20998#discussion_r183408481
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -287,7 +287,7 @@ private[spark] class TaskSetManager(
None
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20930
> because we can get the MapStatus, but get a 'null'. If I'm not mistaken,
this also because the ExecutorLost trigger removeOutputsOnExecutor
If there's a `null` MapStatus for stage 2,
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20930
Hi, @xuanyuanking , thank for your patient explanation, sincerely.
With regard to your latest explanation:
> stage 2's shuffleID is 1, but stage 3 failed by missing an out
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21096
ping @jiangxb1987 @squito
Would you please have a look at this PR? What's your opinions on the cache
strategy
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21104
ping @jerryshao
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/21104
[SPARK-24021][CORE] fix bug in BlacklistTracker's
updateBlacklistForFetchFailure
## What changes were proposed in this pull request?
Thereâs a miswrite in BlacklistTracker's
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/21096
Thank you for your comments @markhamstra .
Yeah, I'm considering adding a UT to support this change. And thank for
reminding me of DAGScheduler's basic principle
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/21096
cache rdd's immediate parent ShuffleDependencies to accelerate
getShuffleDependencies
## What changes were proposed in this pull request?
When creating stages for jobs, we need to find
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20930
Hi, @xuanyuanking , I'm still confused (smile & cry).
> Stage 2 retry 4 times triggered by Stage 3's fetch failed event. Actually
in this scenario, stage 3 will always failed by fet
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r182308871
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2399,6 +2399,50 @@ class DAGSchedulerSuite extends SparkFunSuite
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20998
Will do, and it's okay.
My view limited in the source code yet, but you guys have more practical
experience. So I learned from your points. It's beneficial
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20604
> I'd go even further and suggest that with this fix in, we can actually
remove SPARK-21834, as its no longer necessary.
Yes, otherwise, this PR's work is meaningl
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20604#discussion_r182086337
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1643,7 +1646,10 @@ class SparkContext(config: SparkConf) extends
Logging
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21079#discussion_r181938225
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
@@ -113,3 +118,24 @@ private[spark] class ShuffleMapTask
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r181729645
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -750,6 +752,10 @@ private[spark] class TaskSetManager
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r181732788
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -750,6 +752,10 @@ private[spark] class TaskSetManager
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20998#discussion_r180937626
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -880,8 +880,8 @@ class TaskSetManagerSuite extends SparkFunSuite
1 - 100 of 182 matches
Mail list logo
