tgravescs commented on a change in pull request #27636:
[SPARK-30873][CORE][YARN]Handling Node Decommissioning for Yarn cluster manger
in Spark
URL: https://github.com/apache/spark/pull/27636#discussion_r389673367
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -1542,4 +1542,50 @@ package object config {
.bytesConf(ByteUnit.BYTE)
.createOptional
+ private[spark] val GRACEFUL_DECOMMISSION_ENABLE =
+ ConfigBuilder("spark.graceful.decommission.enable")
+ .doc("Whether to enable the node graceful decommissioning handling")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val GRACEFUL_DECOMMISSION_FETCHFAILED_IGNORE_THRESHOLD =
Review comment:
I assume this is a total number of fetch failures, or is it per stage? I'm
sure I'll see it more in the code later, but if I don't understand from reading
config text then the user won't either.
Without having read the rest of the code, this seems like kind of a weird
setting to me. If we are handling decomissioned nodes then why do we need this.
Is it in case the entire cluster is being decommissioned?
If we keep this we may want to combined the names here, there was thread on
dev about making to many .component. type names.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
