[
https://issues.apache.org/jira/browse/HIVE-29753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh updated HIVE-29753:
--------------------------------------------
Description:
When StorageBasedAuthorizationProvider.checkPermissions() calls
FileUtils.getFileStatusOrNull() to resolve HDFS path permissions, the
underlying NameNode RPC (getFileInfo) can block for an extended duration.
During this time, Metastore Handler threads are silently stuck with no INFO or
WARN log emitted — the issue is only detectable via thread dumps or DEBUG-level
logging.
Example thread dump showing the silent block:
{code:java}
java.lang.Thread.State: WAITING (on object monitor)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(...)
at
org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:1114)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(...)
at org.apache.hadoop.hive.metastore.HMSHandler.get_database(...){code}
In production this can manifest as Metastore thread pool exhaustion with no
actionable log evidence.
Advantages of having this:
1. Client-side visibility without NameNode access
**ipc.server.log.slow.rpc** requires NameNode config changes and a restart.
Hive/HMS operators in most enterprise environments control only the HMS process
— they cannot touch NameNode configs. This fix gives them actionable signal
without any dependency on the infrastructure team.
2. Operation context
The NameNode sees an anonymous **getFileInfo** RPC. It has no idea why it was
called. The Hive-side log tells you exactly which operation triggered it:
- WARN StorageBasedAuthorizationProvider - Slow getFileInfo during
- storage-based authorization check: path=hdfs://nn/warehouse/mydb,
elapsed=4523ms
Without this, even with **ipc.server.log.slow.rpc** enabled on the NameNode,
you still cannot correlate the slow RPC to a specific HMS operation (auth check
vs table scan vs partition listing).
3. Path information for diagnosis
The Hive-side log includes the HDFS path being authorized. This is critical
for diagnosis — it tells operators whether the slow call is on a warehouse
root, a specific database, or a partition path, helping narrow down the issue
to a specific table or misconfigured location.
4. Lower blast radius restart
Both require a process restart to change the config. However, restarting the
NameNode affects the entire HDFS cluster and all clients — it is a high-impact
operation that typically requires a change request and scheduled maintenance
window. Restarting HMS affects only Hive metadata operations and is a routine,
low-impact operation that the Hive team can do independently without involving
the infrastructure team or affecting other HDFS workloads.
5. Threshold tuning per deployment
Different deployments have different NN latency baselines. A threshold of 1s
may be normal for one cluster and a warning sign for another. Having the config
on the Hive side lets operators tune it to their environment independently.
Both mechanisms complement each other — **ipc.server.log.slow.rpc** gives
NameNode-level visibility, while this fix gives HMS-level visibility with
operation context.
was:
When StorageBasedAuthorizationProvider.checkPermissions() calls
FileUtils.getFileStatusOrNull() to resolve HDFS path permissions, the
underlying NameNode RPC (getFileInfo) can block for an extended duration.
During this time, Metastore Handler threads are silently stuck with no INFO or
WARN log emitted — the issue is only detectable via thread dumps or DEBUG-level
logging.
Example thread dump showing the silent block:
{code:java}
java.lang.Thread.State: WAITING (on object monitor)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(...)
at
org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:1114)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(...)
at org.apache.hadoop.hive.metastore.HMSHandler.get_database(...){code}
In production this can manifest as Metastore thread pool exhaustion with no
actionable log evidence.
> StorageBasedAuthorizationProvider silently blocks on slow HDFS getFileInfo
> calls with no observable logging
> -----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29753
> URL: https://issues.apache.org/jira/browse/HIVE-29753
> Project: Hive
> Issue Type: Improvement
> Components: Authorization, Metastore
> Reporter: Indhumathi Muthumurugesh
> Assignee: Indhumathi Muthumurugesh
> Priority: Minor
> Labels: pull-request-available
>
> When StorageBasedAuthorizationProvider.checkPermissions() calls
> FileUtils.getFileStatusOrNull() to resolve HDFS path permissions, the
> underlying NameNode RPC (getFileInfo) can block for an extended duration.
> During this time, Metastore Handler threads are silently stuck with no INFO
> or WARN log emitted — the issue is only detectable via thread dumps or
> DEBUG-level logging.
> Example thread dump showing the silent block:
> {code:java}
> java.lang.Thread.State: WAITING (on object monitor)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(...)
> at
> org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:1114)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(...)
> at org.apache.hadoop.hive.metastore.HMSHandler.get_database(...){code}
>
> In production this can manifest as Metastore thread pool exhaustion with no
> actionable log evidence.
>
> Advantages of having this:
> 1. Client-side visibility without NameNode access
> **ipc.server.log.slow.rpc** requires NameNode config changes and a restart.
> Hive/HMS operators in most enterprise environments control only the HMS
> process — they cannot touch NameNode configs. This fix gives them actionable
> signal without any dependency on the infrastructure team.
> 2. Operation context
> The NameNode sees an anonymous **getFileInfo** RPC. It has no idea why it
> was called. The Hive-side log tells you exactly which operation triggered it:
> - WARN StorageBasedAuthorizationProvider - Slow getFileInfo during
> - storage-based authorization check: path=hdfs://nn/warehouse/mydb,
> elapsed=4523ms
> Without this, even with **ipc.server.log.slow.rpc** enabled on the NameNode,
> you still cannot correlate the slow RPC to a specific HMS operation (auth
> check vs table scan vs partition listing).
> 3. Path information for diagnosis
> The Hive-side log includes the HDFS path being authorized. This is critical
> for diagnosis — it tells operators whether the slow call is on a warehouse
> root, a specific database, or a partition path, helping narrow down the issue
> to a specific table or misconfigured location.
> 4. Lower blast radius restart
> Both require a process restart to change the config. However, restarting
> the NameNode affects the entire HDFS cluster and all clients — it is a
> high-impact operation that typically requires a change request and scheduled
> maintenance window. Restarting HMS affects only Hive metadata operations and
> is a routine, low-impact operation that the Hive team can do independently
> without involving the infrastructure team or affecting other HDFS workloads.
> 5. Threshold tuning per deployment
> Different deployments have different NN latency baselines. A threshold of
> 1s may be normal for one cluster and a warning sign for another. Having the
> config on the Hive side lets operators tune it to their environment
> independently.
> Both mechanisms complement each other — **ipc.server.log.slow.rpc** gives
> NameNode-level visibility, while this fix gives HMS-level visibility with
> operation context.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
