[
https://issues.apache.org/jira/browse/HIVE-23541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128702#comment-17128702
]
Hive QA commented on HIVE-23541:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13005132/HIVE-23541.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 17108 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_ptf_window_unbounded]
(batchId=19)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_windowing]
(batchId=69)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_windowing_2]
(batchId=57)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_windowing]
(batchId=85)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[distinct_windowing]
(batchId=40)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[distinct_windowing_no_cbo]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[estimate_pkfk_filtered_fk]
(batchId=95)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[estimate_pkfk_nocond]
(batchId=109)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[estimate_pkfk_push]
(batchId=71)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_window]
(batchId=108)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[outer_reference_windowed]
(batchId=71)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ptf]
(batchId=46)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_quotes_1]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[temp_table_windowing_expressions]
(batchId=95)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union36]
(batchId=121)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_reference_windowed]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_ptf_part_simple]
(batchId=63)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing]
(batchId=121)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_expressions]
(batchId=111)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_gby2]
(batchId=98)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_gby]
(batchId=111)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_multipartitioning]
(batchId=75)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_navfn]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_order_null]
(batchId=68)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_range_multiorder]
(batchId=44)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_rank]
(batchId=89)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_row_number]
(batchId=87)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_windowspec]
(batchId=63)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_ptf]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing]
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_decimal]
(batchId=99)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_expressions]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_gby2]
(batchId=65)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_multipartitioning]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_navfn]
(batchId=99)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_order_null]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_range_multiorder]
(batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_rank]
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_udaf]
(batchId=97)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22787/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22787/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22787/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 39 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13005132 - PreCommit-HIVE-Build
> Vectorization: Unbounded following window function start producing results
> too early
> ------------------------------------------------------------------------------------
>
> Key: HIVE-23541
> URL: https://issues.apache.org/jira/browse/HIVE-23541
> Project: Hive
> Issue Type: Bug
> Components: PTF-Windowing, Vectorization
> Affects Versions: 4.0.0, 3.1.2
> Reporter: Gopal Vijayaraghavan
> Assignee: Ramesh Kumar Thangarajan
> Priority: Major
> Attachments: HIVE-23541.1.patch
>
>
> ReduceRecordSource indicates the end of group for a reducer input, whenever
> the entire key changes.
> ReduceRecordSource::processVectorGroup calls
> reducer.setNextVectorBatchGroupStatus(/* isLastGroupBatch */ true); when the
> last group is being processed.
> However for PTF window functions with unbounded following, this is triggered
> by the key changing and not the partition changing.
> This results in the VectorPTFOperator detect a change in the sort key as a
> switch of the partition key and start producing results too early.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java#L399
> {code}
> create temporary table test2(id STRING,name STRING,event_dt date) stored as
> orc;
> insert into test2 values ('100','A','2019-08-15'), ('100','A','2019-10-12');
> SELECT name, event_dt, first_value(event_dt) over (PARTITION BY name ORDER BY
> event_dt desc ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT_ROW) last_event_dt
> FROM test2; -- streaming FIRST_VALUE with DESCENDING
> SELECT name, event_dt, last_value(event_dt) over (PARTITION BY name ORDER BY
> event_dt asc ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING )
> last_event_dt FROM test2; -- non-streaming LAST_VALUE with ASCENDING
> {code}
> These two queries should return identical results, with the streaming version
> being significantly faster than the non-streaming one, due to the lack of
> buffered/spilled rows with streaming.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
