[
https://issues.apache.org/jira/browse/HIVE-29754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh updated HIVE-29754:
--------------------------------------------
Description:
When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
behind in the final partition directory after the query completes successfully.
This does not occur with MapReduce execution engine.
Steps to Reproduce
1. Create two tables and a partitioned output table:
{code:java}
CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE output (id INT, val STRING)
PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
2. Load data such that ALL rows in src_prev also exist in src_today
{code:java}
INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
3. Run INSERT OVERWRITE with UNION ALL using Tez:
{code:java}
INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
SELECT p.id, p.val
FROM src_prev p
LEFT OUTER JOIN src_today t ON p.id = t.id
WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
UNION ALL
SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
4. Inspect the output partition on HDFS:
{code:java}
hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
Actual Result:
{code:java}
drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
Expected Result:
{code:java}
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
No temporary directories in the final partition location.
*– Query on this table from spark engine fails with FileNotFountException*
was:
When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
behind in the final partition directory after the query completes successfully.
This does not occur with MapReduce execution engine.
Steps to Reproduce
1. Create two tables and a partitioned output table:
{code:java}
CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE output (id INT, val STRING)
PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
2. Load data such that ALL rows in src_prev also exist in src_today
{code:java}
INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
3. Run INSERT OVERWRITE with UNION ALL using Tez:
{code:java}
INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
SELECT p.id, p.val
FROM src_prev p
LEFT OUTER JOIN src_today t ON p.id = t.id
WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
UNION ALL
SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
4. Inspect the output partition on HDFS:
{code:java}
hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
Actual Result:
{code:java}
drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
Expected Result:
{code:java}
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
No temporary directories in the final partition location.
-- Query on this table from spark engine fails with FileNotFountException
> Tez leaves empty -tmp.HIVE_UNION_SUBDIR_N.moved directory in partition after
> INSERT OVERWRITE ... UNION ALL when one branch produces 0 rows
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29754
> URL: https://issues.apache.org/jira/browse/HIVE-29754
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Reporter: Indhumathi Muthumurugesh
> Assignee: Indhumathi Muthumurugesh
> Priority: Major
>
> When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
> an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
> behind in the final partition directory after the query completes
> successfully.
> This does not occur with MapReduce execution engine.
> Steps to Reproduce
> 1. Create two tables and a partitioned output table:
> {code:java}
> CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
> CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
> CREATE TABLE output (id INT, val STRING)
> PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
> 2. Load data such that ALL rows in src_prev also exist in src_today
> {code:java}
> INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
> INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
> 3. Run INSERT OVERWRITE with UNION ALL using Tez:
> {code:java}
> INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
> SELECT p.id, p.val
> FROM src_prev p
> LEFT OUTER JOIN src_today t ON p.id = t.id
> WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
> UNION ALL
> SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
> 4. Inspect the output partition on HDFS:
> {code:java}
> hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
> Actual Result:
> {code:java}
> drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
> -rw-r--r-- 1_000001_0
> -rw-r--r-- 2_000001_0 {code}
> Expected Result:
> {code:java}
> -rw-r--r-- 1_000001_0
> -rw-r--r-- 2_000001_0 {code}
> No temporary directories in the final partition location.
>
> *– Query on this table from spark engine fails with FileNotFountException*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
