Rajesh Balamohan created HIVE-24774: ---------------------------------------
Summary: Reduce FS listing during dynamic partition loading Key: HIVE-24774 URL: https://issues.apache.org/jira/browse/HIVE-24774 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Rajesh Balamohan When loading large number of partitions in cloud storage, notification log takes lot longer time to list newly added files. It would be good to explore if FileStatus can be reused from Hive::listFilesCreatedByQuery or from copyFiles {noformat} at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031) at org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4171) at org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566) at org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3519) at org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3504) at org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:2984) at org.apache.hadoop.hive.ql.exec.MoveTask.handleDynParts(MoveTask.java:562) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:440) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:730) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:490) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:484) {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)