Stamatis Zampetakis created HIVE-26609:
------------------------------------------
Summary: Cleanup external table directories created in qtests
after test run
Key: HIVE-26609
URL: https://issues.apache.org/jira/browse/HIVE-26609
Project: Hive
Issue Type: Improvement
Components: Tests
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
In many qtests we are creating external tables by setting explicitly the
location of the table.
[https://github.com/apache/hive/blob/566f48d3d3fc740ef958bdf963e511e0853da402/ql/src/test/queries/clientnegative/authorization_uri_create_table_ext.q#L7]
If the test does not remove explicitly the directory (as it happens above) then
it remains there and may cause flakiness and unrelated test failures if other
tests happen to use the same directory somehow.
A recent case where this problem appeared (directory conflict between tests) is
logged under HIVE-26584. There the solution was to add explicit rm commands in
the qfiles.
A more general solution would be to handle the cleanup of such directories
inside
[QTestUtil.clearTablesCreatedDuringTests|https://github.com/apache/hive/blob/566f48d3d3fc740ef958bdf963e511e0853da402/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L342].
The idea is to get the location for an external table from the metastore and
remove the respective directory if it is under a known "safe" directory such as
{{{}$\{system:test.tmp.dir{}}}}.
As discussed under HIVE-26584 it might be risky to forcefully delete any kind
of directory coming from an external table at the risk of corrupting the
development environment. If we restrict the cleanup to known directories it
should be fine though.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
