[
https://issues.apache.org/jira/browse/HIVE-29843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-29843 started by Zhenqi Li.
----------------------------------------
> Concurrent create_database requests may recursively delete a shared database
> directory
> --------------------------------------------------------------------------------------
>
> Key: HIVE-29843
> URL: https://issues.apache.org/jira/browse/HIVE-29843
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Affects Versions: 4.0.0-beta-1
> Environment: Hive Metastore: 4.0.0-beta-1
> MySQL: 8.0.30
> Hadoop: 3.3.1
> FileSystem: S3A-compatible object storage
> Standalone reproduction object storage: MinIO
> Iceberg: 1.6.1
> Concurrency: 16 independent Hive Metastore clients
> Reporter: Zhenqi Li
> Assignee: Zhenqi Li
> Priority: Major
>
> When multiple clients concurrently initialize the same Iceberg namespace,
> Hive Metastore may process identical {{create_database}} requests and create
> the same database directory. Only one request can successfully insert the
> database metadata; the other requests fail because of the database unique
> constraint.
> The failed requests then use a local {{madeDir}} or {{madeExternalDir}} flag
> to decide whether to recursively clean up the entire database directory.
> However, this flag only records that the request created or observed a usable
> directory; it does not prove that the request still has exclusive ownership
> of that directory. Another request may already have committed metadata for
> the same path or written data below it.
> As a result, a failed {{create_database}} request may move or delete a shared
> database directory, leaving valid Metastore metadata pointing to missing
> object-storage files.
> h2. Affected Versions
> The issue was reproduced against Hive Metastore {{{}4.0.0-beta-1{}}}.
> Source inspection found the same directory-ownership and rollback-cleanup
> pattern in Hive branches from {{1.0}} through {{{}4.2.1{}}}. I did not find
> an upstream fix specifically addressing the {{create_database}} directory
> ownership race.
> h2. Description
> Multiple clients initializing the same Iceberg namespace can concurrently
> call Hive Metastore with identical {{create_database}} requests.
> A typical target path is:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamls3a://warehouse/iceberg/test1.db
> {code}
> The following sequence can occur:
> # Multiple requests check that the database does not exist.
> # Multiple requests check that the target directory does not exist.
> # Multiple requests call {{mkdirs}} for the same path.
> # Multiple requests set a local {{madeDir}} or {{madeExternalDir}} flag to
> {{{}true{}}}.
> # One request successfully inserts the database into the Metastore {{DBS}}
> table.
> # The other requests fail because of the {{DBS.UNIQUE_DATABASE}} unique
> constraint.
> # The failed requests enter the rollback cleanup path.
> # Based on their local flag, they assume that they own the database
> directory.
> # A failed request recursively moves or deletes the entire database
> directory.
> There is also a more general interleaving, similar to the one documented in
> HIVE-23437:
> # Request R1 creates the directory and is paused before committing metadata.
> # Request R2 observes that the directory already exists, so R2 does not
> create it.
> # R2 successfully commits database metadata referencing that directory.
> # R1 resumes, loses the metadata race, and rolls back.
> # R1 still has {{madeDir=true}} and recursively deletes the directory
> already adopted by R2.
> This second interleaving shows that even a perfectly atomic {{mkdir}}
> implementation would not be sufficient. Directory creation provenance is not
> equivalent to current deletion ownership.
> h2. Steps to Reproduce
> h3. 1. Prepare Hive Metastore
> Start Hive Metastore with a MySQL metadata database and S3A-compatible object
> storage. Confirm that the target database path does not exist.
> For example:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamls3a://warehouse/hms_direct_race.db
> {code}
> h3. 2. Concurrently invoke create_database
> Use multiple independent Hive Metastore clients and concurrently call:
> {code:java}
> client.createDatabase(database);
> {code}
> All requests must use the same catalog, database name, and location. The
> standalone reproduction used:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlcatalog: hive
> database: hms_direct_race
> concurrency: 16
> {code}
> h3. 3. Observe the Metastore result
> The reproduced result was:
> * 1 request succeeded.
> * 15 requests failed.
> * The failed requests reported a MySQL unique-key conflict:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlDuplicate entry 'hms_direct_race-hive'
> for key 'DBS.UNIQUE_DATABASE'
> {code}
> * Multiple handlers logged that the same database path was created.
> * Failed requests entered the database-directory cleanup path.
> h3. 4. Observe directory cleanup
> If another request writes content below the database directory before
> rollback cleanup runs, the failed request may move or delete that content
> together with the entire database directory.
> For example, affected Iceberg content may include:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamliceberg/test1.db/test3/metadata/00000-....metadata.json
> {code}
> When Hadoop Trash is enabled, the path may be moved under a location similar
> to:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamls3a://warehouse/user/root/.Trash/Current/iceberg/test1.db
> {code}
> h2. Reproduction Log Excerpts
> The following logs are from a standalone reproduction that bypassed all
> higher-level application code. It used 16 independent Hive Metastore clients.
> The run started at {{2026-08-27 02:23:40 UTC}} and targeted database
> {{{}hms_direct_race_20260827T022340Z{}}}.
> h3. Client result
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlHMS_CREATE_DATABASE_RACE host=hive-metastore port=9083
> database=hms_direct_race_20260827T022340Z workers=16
> location=s3a://warehouse/hms_direct_race_20260827T022340Z.db
> worker=00 result=FAILURE error=MetaException: JDODataStoreException:
> Exception thrown flushing changes to datastore Root cause:
> java.sql.SQLIntegrityConstraintViolationException: Duplicate entry
> 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> worker=01 result=FAILURE error=MetaException: JDODataStoreException:
> Exception thrown flushing changes to datastore Root cause:
> java.sql.SQLIntegrityConstraintViolationException: Duplicate entry
> 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> ...
> worker=11 result=SUCCESS
> ...
> worker=15 result=FAILURE error=MetaException: JDODataStoreException:
> Exception thrown flushing changes to datastore Root cause:
> java.sql.SQLIntegrityConstraintViolationException: Duplicate entry
> 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> HMS_CREATE_DATABASE_RACE_COMPLETE elapsed_ms=10380
> {code}
> The final result was one successful request and 15 failures caused by
> {{{}DBS.UNIQUE_DATABASE{}}}.
> h3. HMS audit and directory creation
> Hive Metastore audit logs recorded multiple concurrent {{create_database}}
> requests for the same database. One example is:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yaml2026-08-27T02:23:48,142 INFO [Metastore-Handler-Pool: Thread-1341]
> HiveMetaStore.audit: ugi=root ip=172.19.0.6 cmd=source:172.19.0.6
> create_database: Database(name:hms_direct_race_20260827T022340Z,
> description:direct concurrent create_database test,
> locationUri:s3a://warehouse/hms_direct_race_20260827T022340Z.db,
> parameters:{}, catalogName:hive)
> {code}
> Multiple handlers then processed the same database path:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yaml2026-08-27T02:23:48,401 INFO [Metastore-Handler-Pool: Thread-1340]
> HMSHandler: Creating database path in external directory
> s3a://warehouse/hms_direct_race_20260827T022340Z.db
> 2026-08-27T02:23:48,403 INFO [Metastore-Handler-Pool: Thread-1334]
> HMSHandler: Creating database path in external directory
> s3a://warehouse/hms_direct_race_20260827T022340Z.db
> 2026-08-27T02:23:48,434 INFO [Metastore-Handler-Pool: Thread-1338]
> HMSHandler: Created database path in external directory
> s3a://warehouse/hms_direct_race_20260827T022340Z.db
> 2026-08-27T02:23:48,438 INFO [Metastore-Handler-Pool: Thread-1334]
> HMSHandler: Created database path in external directory
> s3a://warehouse/hms_direct_race_20260827T022340Z.db
> {code}
> h3. Sentinel and cleanup
> A watcher observed the database directory during the race window and uploaded
> a sentinel object below it:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlWATCHER_START 2026-08-27T02:23:41Z
> DIRECTORY_VISIBLE 2026-08-27T02:23:48Z
> s3://warehouse/hms_direct_race_20260827T022340Z.db
> 46 bytes ->
> `minio/warehouse/hms_direct_race_20260827T022340Z.db/sentinel/metadata/race-sentinel.txt`
> SENTINEL_UPLOAD_DONE 2026-08-27T02:23:48Z
> WATCHER_END 2026-08-27T02:23:48Z
> {code}
> The unique-key conflict and subsequent cleanup appeared together in the Hive
> Metastore logs:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlCaused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate
> entry 'hms_direct_race_20260827t022340z-hive' for key 'DBS.UNIQUE_DATABASE'
> at
> org.apache.hadoop.hive.metastore.HMSHandler.create_database_core(HMSHandler.java:1274)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.create_database(HMSHandler.java:1343)
> 2026-08-27T02:23:58,482 WARN [Metastore-Handler-Pool: Thread-1342] FileUtils:
> No such file or directory:
> s3a://warehouse/hms_direct_race_20260827T022340Z.db; Force to delete it.
> 2026-08-27T02:23:58,487 ERROR [Metastore-Handler-Pool: Thread-1341]
> FileUtils: Failed to delete
> s3a://warehouse/hms_direct_race_20260827T022340Z.db
> {code}
> After the test, no objects remained under the original path or the
> corresponding Trash path:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yaml=== OBJECTS UNDER DIRECT PATH ===
> === OBJECTS UNDER TRASH PATH ===
> === DIRECT PATH STAT ===
> mc: <ERROR> Unable to stat
> `minio/warehouse/hms_direct_race_20260827T022340Z.db/`. Object does not exist.
> {code}
> At the same time, the database row still existed in the MySQL {{DBS}} table:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlDB_ID NAME CTLG_NAME DB_LOCATION_URI
> 104 hms_direct_race_20260827t022340z hive
> s3a://warehouse/hms_direct_race_20260827T022340Z.db
> {code}
> Trash was disabled in this standalone reproduction, so cleanup directly
> deleted the directory instead of moving it to {{{}.Trash/Current{}}}.
> h2. Actual Result
> A failed {{create_database}} request may recursively clean up the entire
> final database directory rather than cleaning up only request-private
> resources.
> This can leave the system in the following state:
> * Database metadata still exists in Hive Metastore.
> * Table metadata may still exist in Hive Metastore.
> * Iceberg {{metadata_location}} values still reference the original
> object-storage paths.
> * The corresponding Iceberg metadata files have been moved to Trash or
> recursively deleted.
> * Subsequent Iceberg queries, startup recovery, or metadata commits receive
> 404/not-found errors.
> * Iceberg tables become unreadable or unwritable.
> h2. Expected Result
> When multiple requests concurrently create the same database:
> * One request should successfully create the database.
> * Other requests should receive {{AlreadyExistsException}} or an equivalent
> database-already-exists error.
> * Failed requests must not delete, move, or otherwise modify the shared
> final database directory.
> * Failed requests may only recursively clean up resources that are provably
> private to that request, such as a request-scoped staging path.
> h2. Root Cause
> {{create_database}} uses a non-atomic check-then-create sequence:
> {code:java}
> get_database_core(...);
> if (database does not exist) {
> create_database_core(...);
> }
> {code}
> Inside {{{}create_database_core{}}}, filesystem ownership is inferred from
> logic similar to:
> {code:java}
> if (!wh.isDir(dbPath)) {
> if (!wh.mkdirs(dbPath)) {
> throw new MetaException(...);
> }
> madeDir = true;
> }
> {code}
> A successful {{mkdirs}} call only shows that the directory became available
> through this request. It does not prove continuing exclusive ownership.
> Multiple requests may set {{madeDir}} to {{{}true{}}}, and even a directory
> created exclusively by one request may subsequently be adopted by a different
> request that wins the metadata transaction.
> When one request commits the Metastore metadata and other requests fail
> because of {{{}DBS.UNIQUE_DATABASE{}}}, the failed requests execute cleanup
> similar to:
> {code:java}
> if (!success) {
> ms.rollbackTransaction();
> if (madeDir) {
> wh.deleteDir(dbPath, true);
> }
> }
> {code}
> The corresponding external database path cleanup in Hive {{4.0.0-beta-1}} is
> equivalent to:
> {code:java}
> if (madeExternalDir && dbExtPath != null) {
> wh.deleteDir(dbExtPath, true, db);
> }
> {code}
> The cleanup scope is the entire final database directory, not an empty
> request-private directory or an object written only by the failed request.
> The unique-key conflict itself is an expected concurrency outcome. The
> data-loss risk comes from treating local directory-creation provenance as
> authorization to recursively delete a shared final location after metadata
> rollback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
