Hao Zhong created HIVE-24457:
--------------------------------
Summary: WorkloadManager.scheduleWork should use
FileSystem.getLocal(conf) to delete files
Key: HIVE-24457
URL: https://issues.apache.org/jira/browse/HIVE-24457
Project: Hive
Issue Type: Bug
Reporter: Hao Zhong
WorkloadManager.scheduleWork deletes files as follows:
{code:java}
private void scheduleWork(WmThreadSyncWork context) {
...
// 4. Delete unneeded directories that were replaced by other ones via reopen.
for (final Path path : context.pathsToDelete) {
LOG.info("Deleting {}", path);
workPool.submit(() -> {
try {
path.getFileSystem(conf).delete(path, true);
} catch (Exception ex) {
LOG.error("Failed to delete an old path; ignoring " +
ex.getMessage());
}
});
}
context.pathsToDelete.clear();
}{code}
It should call FileSystem.getLocal(conf).
A previous bug report complains a similar problem:
https://issues.apache.org/jira/browse/HIVE-8056
Its buggy code is:
{code:java}
private void dropSessionPaths(Configuration conf) throws IOException {
if (hdfsSessionPath != null) {
hdfsSessionPath.getFileSystem(conf).delete(hdfsSessionPath, true);
}
if (localSessionPath != null) {
localSessionPath.getFileSystem(conf).delete(localSessionPath, true);
}
{code}
Its fixed code is:
{code:java}
private void dropSessionPaths(Configuration conf) throws IOException {
if (hdfsSessionPath != null) {
hdfsSessionPath.getFileSystem(conf).delete(hdfsSessionPath, true);
}
if (localSessionPath != null) {
FileSystem.getLocal(conf).delete(localSessionPath, true);
}
}
{code}
BTW, HIVE-8056 warns that the bug is flaky. This bug can also be flaky.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
