Zhenqi Li created HIVE-29843:
--------------------------------
Summary: Concurrent create_database requests may recursively
delete a shared database directory
Key: HIVE-29843
URL: https://issues.apache.org/jira/browse/HIVE-29843
Project: Hive
Issue Type: Bug
Components: Standalone Metastore
Affects Versions: 4.0.0-beta-1
Environment: Hive Metastore: 4.0.0-beta-1
MySQL: 8.0.30
Hadoop: 3.3.1
FileSystem: S3A-compatible object storage
Standalone reproduction object storage: MinIO
Iceberg: 1.6.1
Concurrency: 16 independent Hive Metastore clients
Reporter: Zhenqi Li
Assignee: Zhenqi Li
When multiple clients concurrently initialize the same Iceberg namespace, Hive
Metastore may process identical {{create_database}} requests and create the
same database directory. Only one request can successfully insert the database
metadata; the other requests fail because of the database unique constraint.
The failed requests then use a local {{madeDir}} or {{madeExternalDir}} flag to
decide whether to recursively clean up the entire database directory. However,
this flag only records that the request created or observed a usable directory;
it does not prove that the request still has exclusive ownership of that
directory. Another request may already have committed metadata for the same
path or written data below it.
As a result, a failed {{create_database}} request may move or delete a shared
database directory, leaving valid Metastore metadata pointing to missing
object-storage files.
h2. Affected Versions
The issue was reproduced against Hive Metastore {{{}4.0.0-beta-1{}}}.
Source inspection found the same directory-ownership and rollback-cleanup
pattern in Hive branches from {{1.0}} through {{{}4.2.1{}}}. I did not find an
upstream fix specifically addressing the {{create_database}} directory
ownership race.
h2. Description
Multiple clients initializing the same Iceberg namespace can concurrently call
Hive Metastore with identical {{create_database}} requests.
A typical target path is:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamls3a://warehouse/iceberg/test1.db
{code}
The following sequence can occur:
# Multiple requests check that the database does not exist.
# Multiple requests check that the target directory does not exist.
# Multiple requests call {{mkdirs}} for the same path.
# Multiple requests set a local {{madeDir}} or {{madeExternalDir}} flag to
{{{}true{}}}.
# One request successfully inserts the database into the Metastore {{DBS}}
table.
# The other requests fail because of the {{DBS.UNIQUE_DATABASE}} unique
constraint.
# The failed requests enter the rollback cleanup path.
# Based on their local flag, they assume that they own the database directory.
# A failed request recursively moves or deletes the entire database directory.
There is also a more general interleaving, similar to the one documented in
HIVE-23437:
# Request R1 creates the directory and is paused before committing metadata.
# Request R2 observes that the directory already exists, so R2 does not create
it.
# R2 successfully commits database metadata referencing that directory.
# R1 resumes, loses the metadata race, and rolls back.
# R1 still has {{madeDir=true}} and recursively deletes the directory already
adopted by R2.
This second interleaving shows that even a perfectly atomic {{mkdir}}
implementation would not be sufficient. Directory creation provenance is not
equivalent to current deletion ownership.
h2. Steps to Reproduce
h3. 1. Prepare Hive Metastore
Start Hive Metastore with a MySQL metadata database and S3A-compatible object
storage. Confirm that the target database path does not exist.
For example:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamls3a://warehouse/hms_direct_race.db
{code}
h3. 2. Concurrently invoke create_database
Use multiple independent Hive Metastore clients and concurrently call:
{code:java}
client.createDatabase(database);
{code}
All requests must use the same catalog, database name, and location. The
standalone reproduction used:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlcatalog: hive
database: hms_direct_race
concurrency: 16
{code}
h3. 3. Observe the Metastore result
The reproduced result was:
* 1 request succeeded.
* 15 requests failed.
* The failed requests reported a MySQL unique-key conflict:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlDuplicate entry 'hms_direct_race-hive'
for key 'DBS.UNIQUE_DATABASE'
{code}
* Multiple handlers logged that the same database path was created.
* Failed requests entered the database-directory cleanup path.
h3. 4. Observe directory cleanup
If another request writes content below the database directory before rollback
cleanup runs, the failed request may move or delete that content together with
the entire database directory.
For example, affected Iceberg content may include:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamliceberg/test1.db/test3/metadata/00000-....metadata.json
{code}
When Hadoop Trash is enabled, the path may be moved under a location similar to:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamls3a://warehouse/user/root/.Trash/Current/iceberg/test1.db
{code}
h2. Reproduction Log Excerpts
The following logs are from a standalone reproduction that bypassed all
higher-level application code. It used 16 independent Hive Metastore clients.
The run started at {{2026-08-27 02:23:40 UTC}} and targeted database
{{{}hms_direct_race_20260827T022340Z{}}}.
h3. Client result
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlHMS_CREATE_DATABASE_RACE host=hive-metastore port=9083
database=hms_direct_race_20260827T022340Z workers=16
location=s3a://warehouse/hms_direct_race_20260827T022340Z.db
worker=00 result=FAILURE error=MetaException: JDODataStoreException: Exception
thrown flushing changes to datastore Root cause:
java.sql.SQLIntegrityConstraintViolationException: Duplicate entry
'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
worker=01 result=FAILURE error=MetaException: JDODataStoreException: Exception
thrown flushing changes to datastore Root cause:
java.sql.SQLIntegrityConstraintViolationException: Duplicate entry
'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
...
worker=11 result=SUCCESS
...
worker=15 result=FAILURE error=MetaException: JDODataStoreException: Exception
thrown flushing changes to datastore Root cause:
java.sql.SQLIntegrityConstraintViolationException: Duplicate entry
'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
HMS_CREATE_DATABASE_RACE_COMPLETE elapsed_ms=10380
{code}
The final result was one successful request and 15 failures caused by
{{{}DBS.UNIQUE_DATABASE{}}}.
h3. HMS audit and directory creation
Hive Metastore audit logs recorded multiple concurrent {{create_database}}
requests for the same database. One example is:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yaml2026-08-27T02:23:48,142 INFO [Metastore-Handler-Pool: Thread-1341]
HiveMetaStore.audit: ugi=root ip=172.19.0.6 cmd=source:172.19.0.6
create_database: Database(name:hms_direct_race_20260827T022340Z,
description:direct concurrent create_database test,
locationUri:s3a://warehouse/hms_direct_race_20260827T022340Z.db, parameters:{},
catalogName:hive)
{code}
Multiple handlers then processed the same database path:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yaml2026-08-27T02:23:48,401 INFO [Metastore-Handler-Pool: Thread-1340]
HMSHandler: Creating database path in external directory
s3a://warehouse/hms_direct_race_20260827T022340Z.db
2026-08-27T02:23:48,403 INFO [Metastore-Handler-Pool: Thread-1334] HMSHandler:
Creating database path in external directory
s3a://warehouse/hms_direct_race_20260827T022340Z.db
2026-08-27T02:23:48,434 INFO [Metastore-Handler-Pool: Thread-1338] HMSHandler:
Created database path in external directory
s3a://warehouse/hms_direct_race_20260827T022340Z.db
2026-08-27T02:23:48,438 INFO [Metastore-Handler-Pool: Thread-1334] HMSHandler:
Created database path in external directory
s3a://warehouse/hms_direct_race_20260827T022340Z.db
{code}
h3. Sentinel and cleanup
A watcher observed the database directory during the race window and uploaded a
sentinel object below it:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlWATCHER_START 2026-08-27T02:23:41Z
DIRECTORY_VISIBLE 2026-08-27T02:23:48Z
s3://warehouse/hms_direct_race_20260827T022340Z.db
46 bytes ->
`minio/warehouse/hms_direct_race_20260827T022340Z.db/sentinel/metadata/race-sentinel.txt`
SENTINEL_UPLOAD_DONE 2026-08-27T02:23:48Z
WATCHER_END 2026-08-27T02:23:48Z
{code}
The unique-key conflict and subsequent cleanup appeared together in the Hive
Metastore logs:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlCaused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate
entry 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
at
org.apache.hadoop.hive.metastore.HMSHandler.create_database_core(HMSHandler.java:1274)
at
org.apache.hadoop.hive.metastore.HMSHandler.create_database(HMSHandler.java:1343)
2026-08-27T02:23:58,482 WARN [Metastore-Handler-Pool: Thread-1342] FileUtils:
No such file or directory: s3a://warehouse/hms_direct_race_20260827T022340Z.db;
Force to delete it.
2026-08-27T02:23:58,487 ERROR [Metastore-Handler-Pool: Thread-1341] FileUtils:
Failed to delete s3a://warehouse/hms_direct_race_20260827T022340Z.db
{code}
After the test, no objects remained under the original path or the
corresponding Trash path:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml===
OBJECTS UNDER DIRECT PATH ===
=== OBJECTS UNDER TRASH PATH ===
=== DIRECT PATH STAT ===
mc: <ERROR> Unable to stat
`minio/warehouse/hms_direct_race_20260827T022340Z.db/`. Object does not exist.
{code}
At the same time, the database row still existed in the MySQL {{DBS}} table:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlDB_ID NAME CTLG_NAME DB_LOCATION_URI
104 hms_direct_race_20260827t022340z hive
s3a://warehouse/hms_direct_race_20260827T022340Z.db
{code}
Trash was disabled in this standalone reproduction, so cleanup directly deleted
the directory instead of moving it to {{{}.Trash/Current{}}}.
h2. Actual Result
A failed {{create_database}} request may recursively clean up the entire final
database directory rather than cleaning up only request-private resources.
This can leave the system in the following state:
* Database metadata still exists in Hive Metastore.
* Table metadata may still exist in Hive Metastore.
* Iceberg {{metadata_location}} values still reference the original
object-storage paths.
* The corresponding Iceberg metadata files have been moved to Trash or
recursively deleted.
* Subsequent Iceberg queries, startup recovery, or metadata commits receive
404/not-found errors.
* Iceberg tables become unreadable or unwritable.
h2. Expected Result
When multiple requests concurrently create the same database:
* One request should successfully create the database.
* Other requests should receive {{AlreadyExistsException}} or an equivalent
database-already-exists error.
* Failed requests must not delete, move, or otherwise modify the shared final
database directory.
* Failed requests may only recursively clean up resources that are provably
private to that request, such as a request-scoped staging path.
h2. Root Cause
{{create_database}} uses a non-atomic check-then-create sequence:
{code:java}
get_database_core(...);
if (database does not exist) {
create_database_core(...);
}
{code}
Inside {{{}create_database_core{}}}, filesystem ownership is inferred from
logic similar to:
{code:java}
if (!wh.isDir(dbPath)) {
if (!wh.mkdirs(dbPath)) {
throw new MetaException(...);
}
madeDir = true;
}
{code}
A successful {{mkdirs}} call only shows that the directory became available
through this request. It does not prove continuing exclusive ownership.
Multiple requests may set {{madeDir}} to {{{}true{}}}, and even a directory
created exclusively by one request may subsequently be adopted by a different
request that wins the metadata transaction.
When one request commits the Metastore metadata and other requests fail because
of {{{}DBS.UNIQUE_DATABASE{}}}, the failed requests execute cleanup similar to:
{code:java}
if (!success) {
ms.rollbackTransaction();
if (madeDir) {
wh.deleteDir(dbPath, true);
}
}
{code}
The corresponding external database path cleanup in Hive {{4.0.0-beta-1}} is
equivalent to:
{code:java}
if (madeExternalDir && dbExtPath != null) {
wh.deleteDir(dbExtPath, true, db);
}
{code}
The cleanup scope is the entire final database directory, not an empty
request-private directory or an object written only by the failed request.
The unique-key conflict itself is an expected concurrency outcome. The
data-loss risk comes from treating local directory-creation provenance as
authorization to recursively delete a shared final location after metadata
rollback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
