Yibo Dong created SPARK-60128:
---------------------------------
Summary: DROP DATABASE CASCADE leaves managed-table files after
database location changes
Key: SPARK-60128
URL: https://issues.apache.org/jira/browse/SPARK-60128
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark: 5.0.0-SNAPSHOT
Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Reporter: Yibo Dong
h2. What happened
After changing a database's location using {{ALTER DATABASE ... SET LOCATION}},
{{DROP DATABASE ... CASCADE}} removes the database metadata but leaves the data
files of an existing managed table at its original location.
h2. How to reproduce
Save the following SQL as {{tmp.sql}}:
{code:sql}
CREATE DATABASE t_cascade_location
LOCATION '/tmp/spark_t_cascade_location_20261002/old';
CREATE TABLE t_cascade_location.t (
id INT,
label STRING
)
USING PARQUET;
INSERT INTO t_cascade_location.t VALUES (7, 'kept');
DESCRIBE EXTENDED t_cascade_location.t;
ALTER DATABASE t_cascade_location
SET LOCATION '/tmp/spark_t_cascade_location_20261002/new';
DROP DATABASE t_cascade_location CASCADE;
SHOW DATABASES LIKE 't_cascade_location';
{code}
Use a fresh base directory or change the path suffix before running the SQL.
Run with the in-memory session catalog:
{code:bash}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.catalogImplementation=in-memory
-f tmp.sql
{code}
Then check the original managed-table location:
{code:bash}
find /tmp/spark_t_cascade_location_20261002/old/t
-type f -name '*.parquet' -print
{code}
h2. Expected result
{{DROP DATABASE ... CASCADE}} should remove the database and its managed table,
including the table's data at its original location.
{{SHOW DATABASES LIKE 't_cascade_location'}} should return no rows.
The filesystem check should return no Parquet files:
{code:none}
(no output)
{code}
h2. Actual result
The database is removed from the catalog, but the original managed-table data
remains.
{{SHOW DATABASES LIKE 't_cascade_location'}} returns no rows.
The filesystem check still returns a Parquet file:
{code:none}
/tmp/spark_t_cascade_location_20261002/old/t/part-....snappy.parquet
{code}
The managed table's data files remain on disk after {{DROP DATABASE ...
CASCADE}} completes successfully.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
