Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/12994#discussion_r62500518
--- Diff:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -222,4 +219,43 @@ protected void serviceStop() {
public ByteBuffer getMetaData() {
return ByteBuffer.allocate(0);
}
+
+ /**
+ * Set the recovery path for shuffle service recovery when NM is
restarted. The method will be
+ * overrode and called when Hadoop version is 2.5+ and NM recovery is
enabled, otherwise we
+ * have to manually call this to set our own recovery path.
+ */
+ public void setRecoveryPath(Path recoveryPath) {
+ _recoveryPath = recoveryPath;
+ }
+
+ /**
+ * Get the recovery path, this will override the default one to get the
our own maintained
+ * recovery path.
+ */
+ protected Path getRecoveryPath() {
+ String[] localDirs =
_conf.getTrimmedStrings("yarn.nodemanager.local-dirs");
+ for (String dir : localDirs) {
+ File f = new File(new Path(dir).toUri().getPath(),
"registeredExecutors.ldb");
+ if (f.exists()) {
+ if (_recoveryPath == null) {
+ // If NM recovery is not enabled, we should specify the recovery
path using NM local
+ // dirs, which is compatible with the old code.
+ _recoveryPath = new Path(dir);
+ } else {
+ // If NM recovery is enabled and recovery file is existed in old
NM local dirs, which
--- End diff --
nit change "and recovery file is existed in old" to "the recovery file
exists in the old"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
