[
https://issues.apache.org/jira/browse/HIVE-29584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-29584:
----------------------------------
Labels: datanucleus hikaricp hms metastore mysql pull-request-available
threadlocal (was: datanucleus hikaricp hms metastore mysql threadlocal)
> ObjectStore.handleDirectSqlError() leaks broken JDBC connection wrapper in PM
> ThreadLocal cache after MySQL transient outage
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29584
> URL: https://issues.apache.org/jira/browse/HIVE-29584
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore
> Affects Versions: 3.1.3, 4.0.0, 4.2.0
> Environment: Hive: 3.1.3 (also verified affected: 4.0.0, 4.1.0)
> Standalone Metastore: yes
> Backend DB: MySQL 8.0
> JDBC Driver: mysql-connector-j 8.0.30
> Connection Pool: HikariCP 2.6.1
> ORM: DataNucleus 4.1.19 + datanucleus-core 4.1.17 + datanucleus-api-jdo
> 4.2.4
> JVM: OpenJDK 1.8.0_412
> Affected Client: Apache Amoro (long-lived thrift connection client)
> Reporter: soowan4147
> Priority: Critical
> Labels: datanucleus, hikaricp, hms, metastore, mysql,
> pull-request-available, threadlocal
>
> #
> ## Summary
> After a brief MySQL outage (e.g., 4-second network glitch from a planned
> DBA operation), one HMS Thrift worker thread can permanently retain a broken
> HikariProxyConnection in its ObjectStore.pm ThreadLocal cache, leading to
> indefinite reuse of the same broken wrapper for hours until HMS is
> restarted.
> ## Root Cause
> `MetaStoreDirectSql.prepareTxn()` executes `SET
> @@session.sql_mode=ANSI_QUOTES`
> on every transaction. When this fails on a broken connection (e.g.,
> "Connection is closed" SQLException after MySQL transient outage),
> `ObjectStore.handleDirectSqlError()` falls back to ORM mode but does NOT
> invalidate the PersistenceManager. As a result:
> 1. Same `ObjectStore.pm` is reused on next RPC
> 2. Same broken HikariProxyConnection wrapper is reused (held via
> ThreadLocal)
> 3. HikariCP cannot evict the in-use connection per design
> 4. Only HMS restart releases the wrapper
> ## Production Incident Evidence (2026-04-26)
> - pool-6-thread-93872 retained broken wrapper for ~5 hours
> - 41,302 audit RPCs all from same client IP, all failing with same error
> - 6,047 "Falling back to ORM" + 14,309 ERROR logs in 5 hours
> - master02 normal threads' RPC throughput dropped 90%+ during incident
> - catalogd's ALTER_TABLE processing stalled 1~3 minutes per event
> - Resolved only by master HMS restart (3h 10m total impact)
> ## Source References
> Verified the defect exists in:
> - 3.1.3: `ObjectStore.java#L3646-L3697` (handleDirectSqlError)
> - 4.0.0: `ObjectStore.java#L4449-L4495` (same defect, no PM cleanup)
> - `MetaStoreDirectSql.java#L2026-L2034` (prepareTxn trigger)
> ## Steps to Reproduce
> 1. Set up HMS 3.1.3+ with HikariCP backed by MySQL
> 2. Create a long-lived metastore client that maps permanently to one HMS
> worker thread (e.g., Apache Amoro pod, Spark Thrift Server)
> 3. Briefly disconnect MySQL (4 seconds via iptables drop or KILL CONNECTION)
> 4. Observe: one worker thread continues to reuse the broken wrapper
> indefinitely
> 5. Verify: log shows continuous "Falling back to ORM path due to direct SQL
> failure: Error setting ansi quotes: Connection is closed" from same
> thread
> ## Proposed Fix
> In `ObjectStore.handleDirectSqlError()`, when the cause is a
> connection-level
> SQLException, invalidate the PM:
> ```java
> if (isConnectionLevelError(ex)) {
> if (pm != null) {
> try {
> if (pm.currentTransaction().isActive())
> { pm.currentTransaction().rollback(); }
> pm.close(); // releases HikariCP wrapper to pool
> } catch (Exception e)
> { // best effort }
> pm = null;
> directSql = null;
> }
> }
> This forces a fresh PM (and thus a fresh connection) on the next RPC,
> allowing the broken connection to be properly evicted by HikariCP.
> Workarounds (currently in use)
> - Client-side: shorten hive.metastore.client.socket.timeout on long-lived
> clients (e.g., Amoro) so they auto-reconnect every few minutes, breaking
> the permanent thread mapping
> - Operational: enable HikariCP leakDetectionThreshold, alarm on
> "Connection leak detection triggered" log, and auto-restart the affected HMS
> Related JIRAs (none directly fix this)
> - HIVE-22804 (sessionVariables workaround) — does not prevent the leak
> - HIVE-20192 (PM cleanup at thread exit) — different mechanism
> - HIVE-28788 (commit failure → starvation) — different trigger
> - HIVE-28839 (DataNucleus connection starvation) — different code path
> To my knowledge, this specific defect (PM ThreadLocal retaining broken
> wrapper after SQLException in handleDirectSqlError) has not been reported
> before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
