[
https://issues.apache.org/jira/browse/HIVE-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069641#comment-16069641
]
Chaozhong Yang commented on HIVE-16972:
---------------------------------------
[~wzheng] the root cause is "java.lang.ClassNotFoundException:
org.apache.commons.collections4.Predicate" (weird error), I will update and
submit a new patch.
> FetchOperator: filter out inputSplits which length is zero
> ----------------------------------------------------------
>
> Key: HIVE-16972
> URL: https://issues.apache.org/jira/browse/HIVE-16972
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
> Affects Versions: 2.1.0, 2.1.1
> Reporter: Chaozhong Yang
> Assignee: Chaozhong Yang
> Attachments: HIVE-16972.2.patch, HIVE-16972.3.patch,
> HIVE-16972.4.patch, HIVE-16972.patch
>
>
> * Background
> We can describe the basic work flow of common HQL query as follows:
> 1. compile and execute
> 2. fetch results
> In many cases, we don't need to worry about the issues fetching results
> from HDFS(iff there are mapreduce jobs generated in planning step). However,
> the number of results files on HDFS and data distribution will affect the
> final status of HQL query, especially for HiveServer2. We have some map-only
> queries, e.g:
> {code:sql}
> select * from myTable where date > '20170101' and date <= '20170301' and id =
> 88;
> {code}
> This query will generate more than 20,000 files(look at screenshot image
> uploaded) on HDFS and most of those files are empty. Of course, they are very
> sparse. If we send TFetchResultsRequest from HiveServer2 client with some
> parameters(timeout:90s, maxRows:1024) , FetchOperator can not fetch 1024 rows
> in 90 seconds and our HiveServer2 client will mark this TFetchResultsRequest
> as timed out failure. Why? In fact, It's expensive to fetch results from
> empty file. In our HDFS cluster( 5000+ DataNodes) , reading data from an
> empty file will cost almost 100 ms (100ms * 1000 ==> 100s > 90s timeout).
> Obviously, we can filter out those empty files or splits to speed up the
> process of FetchResults.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
