[
https://issues.apache.org/jira/browse/HIVE-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069894#comment-16069894
]
Hive QA commented on HIVE-16972:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875213/HIVE-16972.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 10830 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt1]
(batchId=74)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_binary_binary]
(batchId=94)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_binary_external_table_queries]
(batchId=95)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_binary_storage_queries]
(batchId=95)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key2]
(batchId=93)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key3]
(batchId=92)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_custom_key]
(batchId=92)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_handler_snapshot]
(batchId=95)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_join]
(batchId=94)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_key_range]
(batchId=93)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_pushdown]
(batchId=93)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries]
(batchId=94)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_scan_params]
(batchId=96)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_timestamp]
(batchId=95)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_timestamp_format]
(batchId=96)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[ppd_key_ranges]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnMR (batchId=213)
org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketizedInputFormat
(batchId=269)
org.apache.hadoop.hive.ql.TestTxnCommands2.testSimpleRead (batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication
(batchId=216)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication
(batchId=216)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=216)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=220)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.testPigPopulation
(batchId=230)
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
(batchId=189)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5849/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5849/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5849/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875213 - PreCommit-HIVE-Build
> FetchOperator: filter out inputSplits which length is zero
> ----------------------------------------------------------
>
> Key: HIVE-16972
> URL: https://issues.apache.org/jira/browse/HIVE-16972
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
> Affects Versions: 2.1.0, 2.1.1
> Reporter: Chaozhong Yang
> Assignee: Chaozhong Yang
> Attachments: HIVE-16972.2.patch, HIVE-16972.3.patch,
> HIVE-16972.4.patch, HIVE-16972.5.patch, HIVE-16972.patch
>
>
> * Background
> We can describe the basic work flow of common HQL query as follows:
> 1. compile and execute
> 2. fetch results
> In many cases, we don't need to worry about the issues fetching results
> from HDFS(iff there are mapreduce jobs generated in planning step). However,
> the number of results files on HDFS and data distribution will affect the
> final status of HQL query, especially for HiveServer2. We have some map-only
> queries, e.g:
> {code:sql}
> select * from myTable where date > '20170101' and date <= '20170301' and id =
> 88;
> {code}
> This query will generate more than 20,000 files(look at screenshot image
> uploaded) on HDFS and most of those files are empty. Of course, they are very
> sparse. If we send TFetchResultsRequest from HiveServer2 client with some
> parameters(timeout:90s, maxRows:1024) , FetchOperator can not fetch 1024 rows
> in 90 seconds and our HiveServer2 client will mark this TFetchResultsRequest
> as timed out failure. Why? In fact, It's expensive to fetch results from
> empty file. In our HDFS cluster( 5000+ DataNodes) , reading data from an
> empty file will cost almost 100 ms (100ms * 1000 ==> 100s > 90s timeout).
> Obviously, we can filter out those empty files or splits to speed up the
> process of FetchResults.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
