[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964180#comment-16964180 ]
Attila Magyar commented on HIVE-22411: -------------------------------------- [~ste...@apache.org], this has been there for a while, I don't know the full back story of this implementation. I'll try to find one of the original authors and find out if it's possible to avoid full listing. > Performance degradation on single row inserts > --------------------------------------------- > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive > Reporter: Attila Magyar > Assignee: Attila Magyar > Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separateĀ > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to useĀ fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)