stiga-huang commented on code in PR #3910:
URL: https://github.com/apache/hive/pull/3910#discussion_r1073054442
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java:
##########
@@ -245,16 +243,32 @@ private static void disableLlapCaching(HiveConf conf) {
* @throws IOException the directory cannot be deleted
* @throws HiveException the table is not found
*/
- static void cleanupEmptyDir(HiveConf conf, String tmpTableName) throws
IOException, HiveException {
+ static void cleanupEmptyTableDir(HiveConf conf, String tmpTableName)
+ throws IOException, HiveException {
org.apache.hadoop.hive.ql.metadata.Table tmpTable =
Hive.get().getTable(tmpTableName);
if (tmpTable != null) {
- Path path = new Path(tmpTable.getSd().getLocation());
- FileSystem fs = path.getFileSystem(conf);
+ cleanupEmptyDir(conf, new Path(tmpTable.getSd().getLocation()));
+ }
+ }
+
+ /**
+ * Remove the directory if it's empty.
+ * @param conf the Hive configuration
+ * @param path path of the directory
+ * @throws IOException if any IO error occurs
+ */
+ static void cleanupEmptyDir(HiveConf conf, Path path) throws IOException {
+ FileSystem fs = path.getFileSystem(conf);
+ try {
if (!fs.listFiles(path, false).hasNext()) {
fs.delete(path, true);
}
+ } catch (FileNotFoundException e) {
+ // Ignore the case when the dir was already removed
+ LOG.warn("Ignored exception during cleanup {}", path, e);
Review Comment:
FWIW, the following log shows the stacktrace of where the
`FileNotFoundException` is thrown:
```
2023-01-02T02:12:55,849 ERROR
[impala-ec2-centos79-m6i-4xlarge-ondemand-1428.vpc.cloudera.com-48_executor]
compactor.Worker: Caught exception while trying to compact
id:15,dbname:partial_catalog_info_test,tableName:insert_only_partitioned,partName:part=1,state:^@,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:jenkins,tooManyAborts:false,hasOldAbort:false,highestWriteId:3,errorMessage:null,workerId:
null,initiatorId: null,retryRetention0. Marking failed to avoid repeated
failures
java.io.FileNotFoundException: File
hdfs://localhost:20500/tmp/hive/jenkins/092b533a-81c8-4b95-88e4-9472cf6f365d/_tmp_space.db/62ec04fb-e2d2-4a99-a454-ae709a3cccfe
does not exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1275)
~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1249)
~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
~[hadoop-hdfs-client-3.1.1.7.2.15.4-6.jar:?]
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2144)
~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.fs.FileSystem$5.<init>(FileSystem.java:2302)
~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:2299)
~[hadoop-common-3.1.1.7.2.15.4-6.jar:?]
at
org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor$Util.cleanupEmptyDir(QueryCompactor.java:261)
~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at
org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.commitCompaction(MmMinorQueryCompactor.java:72)
~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at
org.apache.hadoop.hive.ql.txn.compactor.QueryCompactor.runCompactionQueries(QueryCompactor.java:146)
~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at
org.apache.hadoop.hive.ql.txn.compactor.MmMinorQueryCompactor.runCompaction(MmMinorQueryCompactor.java:63)
~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at
org.apache.hadoop.hive.ql.txn.compactor.Worker.findNextCompactionAndExecute(Worker.java:435)
~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at
org.apache.hadoop.hive.ql.txn.compactor.Worker.lambda$run$0(Worker.java:115)
~[hive-exec-3.1.3000.2022.0.13.0-60.jar:3.1.3000.2022.0.13.0-60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_261]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_261]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
