[
https://issues.apache.org/jira/browse/HIVE-29754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh updated HIVE-29754:
--------------------------------------------
Description:
When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
behind in the final partition directory after the query completes successfully.
This does not occur with MapReduce execution engine.
Steps to Reproduce
1. Create two tables and a partitioned output table:
{code:java}
CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE output (id INT, val STRING)
PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
2. Load data such that ALL rows in src_prev also exist in src_today
{code:java}
INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
3. Run INSERT OVERWRITE with UNION ALL using Tez:
{code:java}
INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
SELECT p.id, p.val
FROM src_prev p
LEFT OUTER JOIN src_today t ON p.id = t.id
WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
UNION ALL
SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
4. Inspect the output partition on HDFS:
{code:java}
hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
Actual Result:
{code:java}
drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
Expected Result:
{code:java}
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
No temporary directories in the final partition location.
*– Query on this table from spark engine fails with FileNotFountException*
In \{{Utilities.mvFileToFinalPath()}}
(ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java)
*Line 1474* — staging dir renamed to .moved unconditionally on success:
{code:java}
tmpPath = new Path(tmpPath.getParent(), tmpPath.getName() + ".moved");
Utilities.rename(fs, tmpPathOriginal, tmpPath);
// tmpPath is now: .../-tmp.HIVE_UNION_SUBDIR_1.moved {code}
*Line 1485* — files checked, but empty branch has no files:
{code:java}
FileStatus[] statuses = HiveStatsUtils.getFileStatusRecurse(tmpPath,
...).toArray(...);
if (statuses != null && statuses.length > 0) {
// move files → renameOrMoveFilesInParallel(hconf, fs, tmpPath, specPath);
}
// ← NO else: when statuses.length == 0 (empty UNION branch),
// .moved directory is never deleted ← BUG at line 1531
{code}
*Line 1532* — the else at this level is only for success=false:
{code:java}
} else {
fs.delete(tmpPath, true); // only reached when success=false
} {code}
When a UNION ALL branch produces 0 rows, Tez still creates the
HIVE_UNION_SUBDIR_N staging directory but writes no files. mvFileToFinalPath
renames it to .moved but the statuses.length == 0 check causes the entire block
to be skipped — the .moved directory is never deleted. MoveTask subsequently
moves it to the final partition.
was:
When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
behind in the final partition directory after the query completes successfully.
This does not occur with MapReduce execution engine.
Steps to Reproduce
1. Create two tables and a partitioned output table:
{code:java}
CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
CREATE TABLE output (id INT, val STRING)
PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
2. Load data such that ALL rows in src_prev also exist in src_today
{code:java}
INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
3. Run INSERT OVERWRITE with UNION ALL using Tez:
{code:java}
INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
SELECT p.id, p.val
FROM src_prev p
LEFT OUTER JOIN src_today t ON p.id = t.id
WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
UNION ALL
SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
4. Inspect the output partition on HDFS:
{code:java}
hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
Actual Result:
{code:java}
drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
Expected Result:
{code:java}
-rw-r--r-- 1_000001_0
-rw-r--r-- 2_000001_0 {code}
No temporary directories in the final partition location.
*– Query on this table from spark engine fails with FileNotFountException*
> Tez leaves empty -tmp.HIVE_UNION_SUBDIR_N.moved directory in partition after
> INSERT OVERWRITE ... UNION ALL when one branch produces 0 rows
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29754
> URL: https://issues.apache.org/jira/browse/HIVE-29754
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Reporter: Indhumathi Muthumurugesh
> Assignee: Indhumathi Muthumurugesh
> Priority: Major
>
> When executing INSERT OVERWRITE ... UNION ALL using the Tez execution engine,
> an empty temporary directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
> behind in the final partition directory after the query completes
> successfully.
> This does not occur with MapReduce execution engine.
> Steps to Reproduce
> 1. Create two tables and a partitioned output table:
> {code:java}
> CREATE TABLE src_prev (id INT, val STRING) STORED AS TEXTFILE;
> CREATE TABLE src_today (id INT, val STRING) STORED AS TEXTFILE;
> CREATE TABLE output (id INT, val STRING)
> PARTITIONED BY (run_dt STRING) STORED AS TEXTFILE; {code}
> 2. Load data such that ALL rows in src_prev also exist in src_today
> {code:java}
> INSERT INTO src_prev VALUES (1,'aaa'), (2,'bbb');
> INSERT INTO src_today VALUES (1,'aaa'), (2,'bbb'), (3,'ccc'); {code}
> 3. Run INSERT OVERWRITE with UNION ALL using Tez:
> {code:java}
> INSERT OVERWRITE TABLE output PARTITION (run_dt='2024-01-01')
> SELECT p.id, p.val
> FROM src_prev p
> LEFT OUTER JOIN src_today t ON p.id = t.id
> WHERE t.id IS NULL -- Branch 1: anti-join, returns 0 rows
> UNION ALL
> SELECT id, val FROM src_today; -- Branch 2: returns rows {code}
> 4. Inspect the output partition on HDFS:
> {code:java}
> hdfs dfs -ls /warehouse/output/run_dt=2024-01-01/ {code}
> Actual Result:
> {code:java}
> drwxr-xr-x -tmp.HIVE_UNION_SUBDIR_1.moved <- leftover empty directory
> -rw-r--r-- 1_000001_0
> -rw-r--r-- 2_000001_0 {code}
> Expected Result:
> {code:java}
> -rw-r--r-- 1_000001_0
> -rw-r--r-- 2_000001_0 {code}
> No temporary directories in the final partition location.
>
> *– Query on this table from spark engine fails with FileNotFountException*
>
> In \{{Utilities.mvFileToFinalPath()}}
> (ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java)
> *Line 1474* — staging dir renamed to .moved unconditionally on success:
> {code:java}
> tmpPath = new Path(tmpPath.getParent(), tmpPath.getName() + ".moved");
> Utilities.rename(fs, tmpPathOriginal, tmpPath);
> // tmpPath is now: .../-tmp.HIVE_UNION_SUBDIR_1.moved {code}
> *Line 1485* — files checked, but empty branch has no files:
> {code:java}
> FileStatus[] statuses = HiveStatsUtils.getFileStatusRecurse(tmpPath,
> ...).toArray(...);
> if (statuses != null && statuses.length > 0) {
> // move files → renameOrMoveFilesInParallel(hconf, fs, tmpPath, specPath);
> }
> // ← NO else: when statuses.length == 0 (empty UNION branch),
> // .moved directory is never deleted ← BUG at line 1531
> {code}
> *Line 1532* — the else at this level is only for success=false:
> {code:java}
> } else {
> fs.delete(tmpPath, true); // only reached when success=false
> } {code}
> When a UNION ALL branch produces 0 rows, Tez still creates the
> HIVE_UNION_SUBDIR_N staging directory but writes no files. mvFileToFinalPath
> renames it to .moved but the statuses.length == 0 check causes the entire
> block to be skipped — the .moved directory is never deleted. MoveTask
> subsequently moves it to the final partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
