[
https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Yousufi updated HIVE-14165:
------------------------------------
Description:
The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's
FileInputFormat.java will list the files during split computation anyway to
determine their size. One way to remove this is to catch the InvalidInputFormat
exception FileInputFormat#getSplits() on the Hive side instead of doing the
file listing beforehand.
For S3 select queries on partitioned tables, this results in a 2x speedup.
was:Split size computation be may improved by the optimizations for
listFiles() in HADOOP-13208
> Remove Hive file listing during split computation
> -------------------------------------------------
>
> Key: HIVE-14165
> URL: https://issues.apache.org/jira/browse/HIVE-14165
> Project: Hive
> Issue Type: Sub-task
> Affects Versions: 2.1.0
> Reporter: Abdullah Yousufi
> Assignee: Abdullah Yousufi
>
> The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's
> FileInputFormat.java will list the files during split computation anyway to
> determine their size. One way to remove this is to catch the
> InvalidInputFormat exception FileInputFormat#getSplits() on the Hive side
> instead of doing the file listing beforehand.
> For S3 select queries on partitioned tables, this results in a 2x speedup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
