Yibo Dong created SPARK-60129:
---------------------------------
Summary: RECOVER PARTITIONS changes an existing partition's
location after SET LOCATION
Key: SPARK-60129
URL: https://issues.apache.org/jira/browse/SPARK-60129
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark: 5.0.0-SNAPSHOT
Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Reporter: Yibo Dong
h2. What happened
After changing a partitioned table's root location using {{ALTER TABLE ... SET
LOCATION}}, {{ALTER TABLE ... RECOVER PARTITIONS}} can change the location of
an already registered partition.
As a result, subsequent queries return different data for the same partition,
even though its location was never explicitly changed.
h2. How to reproduce
Save the following SQL as {{tmp.sql}}:
{code:sql}
DROP TABLE IF EXISTS t_set_location_existing;
DROP TABLE IF EXISTS t_set_location_new;
CREATE TABLE t_set_location_existing (
id INT,
value STRING
)
USING PARQUET
PARTITIONED BY (p STRING)
LOCATION '/tmp/spark_t_set_location_old';
INSERT INTO t_set_location_existing VALUES
(1, 'old-a', 'a'),
(2, 'old-b', 'b');
CREATE TABLE t_set_location_new (
id INT,
value STRING
)
USING PARQUET
PARTITIONED BY (p STRING)
LOCATION '/tmp/spark_t_set_location_new';
INSERT INTO t_set_location_new VALUES (3, 'new-b', 'b');
ALTER TABLE t_set_location_existing RECOVER PARTITIONS;
SELECT id, value, p
FROM t_set_location_existing
ORDER BY id;
ALTER TABLE t_set_location_existing
SET LOCATION '/tmp/spark_t_set_location_new';
SELECT id, value, p
FROM t_set_location_existing
ORDER BY id;
ALTER TABLE t_set_location_existing RECOVER PARTITIONS;
SELECT id, value, p
FROM t_set_location_existing
ORDER BY id;
{code}
Start with clean directories and run:
{code:bash}
rm -rf /tmp/spark_t_set_location_old
/tmp/spark_t_set_location_new
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.shuffle.partitions=1
-f tmp.sql
{code}
h2. Expected result
Changing the table's root location should not change the locations of already
registered partitions. The subsequent {{RECOVER PARTITIONS}} should not replace
existing partitions.
All three queries should return:
{code:none}
id | value | p
---+-------+---
1 | old-a | a
2 | old-b | b
{code}
h2. Actual result
After the initial {{RECOVER PARTITIONS}}, the query returns:
{code:none}
id | value | p
---+-------+---
1 | old-a | a
2 | old-b | b
{code}
After {{ALTER TABLE ... SET LOCATION}}, the query still returns:
{code:none}
id | value | p
---+-------+---
1 | old-a | a
2 | old-b | b
{code}
However, after running {{RECOVER PARTITIONS}} again, the result changes to:
{code:none}
id | value | p
---+-------+---
1 | old-a | a
3 | new-b | b
{code}
The existing partition {{p=b}} now returns data from the new table root instead
of its original location, although no partition-level location change was
requested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
