sercanCyberVision commented on code in PR #4293:
URL: https://github.com/apache/hive/pull/4293#discussion_r1218129590
##########
ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java:
##########
@@ -236,4 +243,29 @@ static Options createOptions() {
return result;
}
+
+ /**
+ * While deleting dangling scratch dirs from hdfs, we can clean
corresponding local files as well
+ * @param sessionName prefix to determine removable tmp files
+ * @param localTmpdir local tmp file location
+ */
+ public void removeLocalTmpFiles(String sessionName, String localTmpdir) {
+ File[] files = new File(localTmpdir).listFiles();
+ boolean success;
+ if (files != null) {
+ for (File file : files) {
+ success = false;
+ if (file.getName().startsWith(sessionName)) {
Review Comment:
I have moved the constructor to the first place where `localTmpdir` is
created (out of the loops). It will be invoked once per service run.
##########
ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java:
##########
@@ -236,4 +243,29 @@ static Options createOptions() {
return result;
}
+
+ /**
+ * While deleting dangling scratch dirs from hdfs, we can clean
corresponding local files as well
+ * @param sessionName prefix to determine removable tmp files
+ * @param localTmpdir local tmp file location
+ */
+ public void removeLocalTmpFiles(String sessionName, String localTmpdir) {
+ File[] files = new File(localTmpdir).listFiles();
+ boolean success;
+ if (files != null) {
+ for (File file : files) {
+ success = false;
+ if (file.getName().startsWith(sessionName)) {
+ success = file.delete();
Review Comment:
I have added `File.canWrite()` check before attempting to delete.
##########
ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java:
##########
@@ -236,4 +243,29 @@ static Options createOptions() {
return result;
}
+
+ /**
+ * While deleting dangling scratch dirs from hdfs, we can clean
corresponding local files as well
+ * @param sessionName prefix to determine removable tmp files
+ * @param localTmpdir local tmp file location
+ */
+ public void removeLocalTmpFiles(String sessionName, String localTmpdir) {
Review Comment:
@veghlaci05 this is visible for testing. I have added `@VisibleForTesting`
annotation. Initially the method was private and in unit test I have executed
whole service, but service failed with below exception;
```
java.lang.UnsupportedOperationException: Append is not supported by
ChecksumFileSystem
```
at below
[point](https://github.com/apache/hive/blob/badba5e8a3b98f8c7be4903a6d9b62f8ea5e142e/ql/src/java/org/apache/hadoop/hive/ql/session/ClearDanglingScratchDir.java#L142);
```
IOUtils.closeStream(fs.append(lockFilePath));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
