[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612693#comment-17612693
]
John Sherman commented on HIVE-26584:
-------------------------------------
Thanks [~ayushtkn] and [~zabetak] for helping to make the patch better.
I don't see any indication that the tests require external tables. But I agree
that I could be overlooking a detail and I don't functionally need to change
the test so significantly (even if I think the newer version is cleaner). I
went with just rmr the created directories to clean up the directories at the
end.
I did not add DROP IF EXISTS or rmr(s) before the creation since I find that
practice typically hides problems and sometimes causes hidden dependencies
between tests.
As for clearTablesCreatedDuringTests - it doesn't clean these files up because
it only cleans up tables under the configured warehouse directory. This test
case manually creates a location not under the warehouse directory so it
doesn't end up cleaning up the files. I could modify the
clearTablesCreatedDuringTests to clean up all directories mentioned in CREATE
EXTERNAL TABLE location clauses, but that could be risky since it could lead to
user files being accidentally removed with a misconfigured location clause and
I am not sure I would be able to add all the possible checks to prevent that.
In the future I think tests like this should load the data via LOAD DATA if
possible and not use a custom LOCATION clause so it gets cleaned up normally.
> compressed_skip_header_footer_aggr.q is flaky
> ---------------------------------------------
>
> Key: HIVE-26584
> URL: https://issues.apache.org/jira/browse/HIVE-26584
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 4.0.0-alpha-2
> Reporter: John Sherman
> Assignee: John Sherman
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> One of my PRs compressed_skip_header_footer_aggr.q was failing with
> unexpected diff. Such as:
> {code:java}
> TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
> contained differences (error code = 1) after executing
> compressed_skip_header_footer_aggr.q
> 69,71c69,70
> < 1 2019-12-31
> < 2 2018-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 89d87
> < NULL NULL
> 91c89
> < 2 2018-12-31
> ---
> > 2 2019-12-31
> 100c98
> < 1
> ---
> > 2
> 109c107
> < 1 2019-12-31
> ---
> > 2 2019-12-31
> 127,128c125,126
> < 1 2019-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 146a145
> > 2 2019-12-31
> 155c154
> < 1
> ---
> > 2 {code}
> Investigating it, it did not seem to fail when executed locally. Since I
> suspected test interference I searched for the tablenames/directories used
> and discovered empty_skip_header_footer_aggr.q which uses the same table
> names AND external directories.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
