Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18905#discussion_r132901109
--- Diff:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -333,33 +333,63 @@ protected Path getRecoveryPath(String fileName) {
}
/**
+ * Check the chosen DB file available or not.
+ */
+ protected Boolean checkFileAvailable(File file) {
+ if (file.canWrite()){
+ return true;
+ }
+
+ return false;
+ }
+
+ /**
* Figure out the recovery path and handle moving the DB if YARN NM
recovery gets enabled
* when it previously was not. If YARN NM recovery is enabled it uses
that path, otherwise
* it will uses a YARN local dir.
*/
protected File initRecoveryDb(String dbName) {
+ Boolean bolRecoveryPathAvailable = true;
+
if (_recoveryPath != null) {
File recoveryFile = new File(_recoveryPath.toUri().getPath(),
dbName);
- if (recoveryFile.exists()) {
+
+ bolRecoveryPathAvailable = checkFileAvailable(recoveryFile);
+ logger.info("Recovery path {} ldb available: {}.", _recoveryPath,
bolRecoveryPathAvailable);
+ if (recoveryFile.exists() && bolRecoveryPathAvailable) {
return recoveryFile;
}
}
+
+ // If recovery path unavailable, no use it any more.
+ if (!bolRecoveryPathAvailable) {
--- End diff --
I think recovery path is set by user or use yarn default, user should make
sure the availability of this directory, and yarn internally relies on it. It
doesn't make sense to change to another disk if recovery path is unavailable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
