Hao Zhong created HIVE-24457: -------------------------------- Summary: WorkloadManager.scheduleWork should use FileSystem.getLocal(conf) to delete files Key: HIVE-24457 URL: https://issues.apache.org/jira/browse/HIVE-24457 Project: Hive Issue Type: Bug Reporter: Hao Zhong
WorkloadManager.scheduleWork deletes files as follows: {code:java} private void scheduleWork(WmThreadSyncWork context) { ... // 4. Delete unneeded directories that were replaced by other ones via reopen. for (final Path path : context.pathsToDelete) { LOG.info("Deleting {}", path); workPool.submit(() -> { try { path.getFileSystem(conf).delete(path, true); } catch (Exception ex) { LOG.error("Failed to delete an old path; ignoring " + ex.getMessage()); } }); } context.pathsToDelete.clear(); }{code} It should call FileSystem.getLocal(conf). A previous bug report complains a similar problem: https://issues.apache.org/jira/browse/HIVE-8056 Its buggy code is: {code:java} private void dropSessionPaths(Configuration conf) throws IOException { if (hdfsSessionPath != null) { hdfsSessionPath.getFileSystem(conf).delete(hdfsSessionPath, true); } if (localSessionPath != null) { localSessionPath.getFileSystem(conf).delete(localSessionPath, true); } {code} Its fixed code is: {code:java} private void dropSessionPaths(Configuration conf) throws IOException { if (hdfsSessionPath != null) { hdfsSessionPath.getFileSystem(conf).delete(hdfsSessionPath, true); } if (localSessionPath != null) { FileSystem.getLocal(conf).delete(localSessionPath, true); } } {code} BTW, HIVE-8056 warns that the bug is flaky. This bug can also be flaky. -- This message was sent by Atlassian Jira (v8.3.4#803005)