[
https://issues.apache.org/jira/browse/HIVE-29753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh updated HIVE-29753:
--------------------------------------------
Description:
When StorageBasedAuthorizationProvider.checkPermissions() calls
FileUtils.getFileStatusOrNull() to resolve HDFS path permissions, the
underlying NameNode RPC (getFileInfo) can block for an extended duration.
During this time, Metastore Handler threads are silently stuck with no INFO or
WARN log emitted — the issue is only detectable via thread dumps or DEBUG-level
logging.
Example thread dump showing the silent block:
{code:java}
java.lang.Thread.State: WAITING (on object monitor)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(...)
at
org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:1114)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(...)
at org.apache.hadoop.hive.metastore.HMSHandler.get_database(...){code}
In production this can manifest as Metastore thread pool exhaustion with no
actionable log evidence.
Advantages of having this:
1. Client-side visibility without NameNode access
*{*}ipc.server.log.slow.rpc{*}* requires NameNode config changes and a
restart. Hive/HMS operators in most enterprise environments control only the
HMS process — they cannot touch NameNode configs. This fix gives them
actionable signal without any dependency on the infrastructure team.
2. Operation context
The NameNode sees an anonymous *{*}getFileInfo{*}* RPC. It has no idea why it
was called. The Hive-side log tells you exactly which operation triggered it:
- WARN StorageBasedAuthorizationProvider - Slow getFileInfo during
- storage-based authorization check: path=hdfs://nn/warehouse/mydb,
elapsed=4523ms
Without this, even with *{*}ipc.server.log.slow.rpc{*}* enabled on the
NameNode, you still cannot correlate the slow RPC to a specific HMS operation
(auth check vs table scan vs partition listing).
3. Path information for diagnosis
The Hive-side log includes the HDFS path being authorized. This is critical
for diagnosis — it tells operators whether the slow call is on a warehouse
root, a specific database, or a partition path, helping narrow down the issue
to a specific table or misconfigured location.
4. Lower blast radius restart
Both require a process restart to change the config. However, restarting the
NameNode affects the entire HDFS cluster and all clients — it is a high-impact
operation that typically requires a change request and scheduled maintenance
window. Restarting HMS affects only Hive metadata operations and is a routine,
low-impact operation that the Hive team can do independently without involving
the infrastructure team or affecting other HDFS workloads.
5. Threshold tuning per deployment
Different deployments have different NN latency baselines. A threshold of 1s
may be normal for one cluster and a warning sign for another. Having the config
on the Hive side lets operators tune it to their environment independently.
6. Filesystem agnostic — works for any future FileSystem
ipc.server.log.slow.rpc is specific to Hadoop IPC and only covers HDFS
NameNode RPC calls. With storage backends like S3, Azure ADLS, GCS, or Vast S3
becoming increasingly common in Hive deployments, slow getFileInfo calls can
occur on any FileSystem implementation — not just HDFS. Since this fix
instruments FileUtils.getFileStatusOrNull() at the Hive level, it will
automatically provide slow-call visibility for any FileSystem that
storage-based authorization runs against today or in the future, without
requiring any equivalent mechanism to exist in those storage systems.
Both mechanisms complement each other — *{*}ipc.server.log.slow.rpc{*}* gives
NameNode-level visibility, while this fix gives HMS-level visibility with
operation context.
was:
When StorageBasedAuthorizationProvider.checkPermissions() calls
FileUtils.getFileStatusOrNull() to resolve HDFS path permissions, the
underlying NameNode RPC (getFileInfo) can block for an extended duration.
During this time, Metastore Handler threads are silently stuck with no INFO or
WARN log emitted — the issue is only detectable via thread dumps or DEBUG-level
logging.
Example thread dump showing the silent block:
{code:java}
java.lang.Thread.State: WAITING (on object monitor)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(...)
at
org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:1114)
at
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(...)
at org.apache.hadoop.hive.metastore.HMSHandler.get_database(...){code}
In production this can manifest as Metastore thread pool exhaustion with no
actionable log evidence.
Advantages of having this:
1. Client-side visibility without NameNode access
**ipc.server.log.slow.rpc** requires NameNode config changes and a restart.
Hive/HMS operators in most enterprise environments control only the HMS process
— they cannot touch NameNode configs. This fix gives them actionable signal
without any dependency on the infrastructure team.
2. Operation context
The NameNode sees an anonymous **getFileInfo** RPC. It has no idea why it was
called. The Hive-side log tells you exactly which operation triggered it:
- WARN StorageBasedAuthorizationProvider - Slow getFileInfo during
- storage-based authorization check: path=hdfs://nn/warehouse/mydb,
elapsed=4523ms
Without this, even with **ipc.server.log.slow.rpc** enabled on the NameNode,
you still cannot correlate the slow RPC to a specific HMS operation (auth check
vs table scan vs partition listing).
3. Path information for diagnosis
The Hive-side log includes the HDFS path being authorized. This is critical
for diagnosis — it tells operators whether the slow call is on a warehouse
root, a specific database, or a partition path, helping narrow down the issue
to a specific table or misconfigured location.
4. Lower blast radius restart
Both require a process restart to change the config. However, restarting the
NameNode affects the entire HDFS cluster and all clients — it is a high-impact
operation that typically requires a change request and scheduled maintenance
window. Restarting HMS affects only Hive metadata operations and is a routine,
low-impact operation that the Hive team can do independently without involving
the infrastructure team or affecting other HDFS workloads.
5. Threshold tuning per deployment
Different deployments have different NN latency baselines. A threshold of 1s
may be normal for one cluster and a warning sign for another. Having the config
on the Hive side lets operators tune it to their environment independently.
Both mechanisms complement each other — **ipc.server.log.slow.rpc** gives
NameNode-level visibility, while this fix gives HMS-level visibility with
operation context.
> StorageBasedAuthorizationProvider silently blocks on slow HDFS getFileInfo
> calls with no observable logging
> -----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29753
> URL: https://issues.apache.org/jira/browse/HIVE-29753
> Project: Hive
> Issue Type: Improvement
> Components: Authorization, Metastore
> Reporter: Indhumathi Muthumurugesh
> Assignee: Indhumathi Muthumurugesh
> Priority: Minor
> Labels: pull-request-available
>
> When StorageBasedAuthorizationProvider.checkPermissions() calls
> FileUtils.getFileStatusOrNull() to resolve HDFS path permissions, the
> underlying NameNode RPC (getFileInfo) can block for an extended duration.
> During this time, Metastore Handler threads are silently stuck with no INFO
> or WARN log emitted — the issue is only detectable via thread dumps or
> DEBUG-level logging.
> Example thread dump showing the silent block:
> {code:java}
> java.lang.Thread.State: WAITING (on object monitor)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(...)
> at
> org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:1114)
> at
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(...)
> at org.apache.hadoop.hive.metastore.HMSHandler.get_database(...){code}
>
> In production this can manifest as Metastore thread pool exhaustion with no
> actionable log evidence.
>
> Advantages of having this:
> 1. Client-side visibility without NameNode access
> *{*}ipc.server.log.slow.rpc{*}* requires NameNode config changes and a
> restart. Hive/HMS operators in most enterprise environments control only the
> HMS process — they cannot touch NameNode configs. This fix gives them
> actionable signal without any dependency on the infrastructure team.
> 2. Operation context
> The NameNode sees an anonymous *{*}getFileInfo{*}* RPC. It has no idea why
> it was called. The Hive-side log tells you exactly which operation triggered
> it:
> - WARN StorageBasedAuthorizationProvider - Slow getFileInfo during
> - storage-based authorization check: path=hdfs://nn/warehouse/mydb,
> elapsed=4523ms
> Without this, even with *{*}ipc.server.log.slow.rpc{*}* enabled on the
> NameNode, you still cannot correlate the slow RPC to a specific HMS operation
> (auth check vs table scan vs partition listing).
> 3. Path information for diagnosis
> The Hive-side log includes the HDFS path being authorized. This is critical
> for diagnosis — it tells operators whether the slow call is on a warehouse
> root, a specific database, or a partition path, helping narrow down the issue
> to a specific table or misconfigured location.
> 4. Lower blast radius restart
> Both require a process restart to change the config. However, restarting
> the NameNode affects the entire HDFS cluster and all clients — it is a
> high-impact operation that typically requires a change request and scheduled
> maintenance window. Restarting HMS affects only Hive metadata operations and
> is a routine, low-impact operation that the Hive team can do independently
> without involving the infrastructure team or affecting other HDFS workloads.
> 5. Threshold tuning per deployment
> Different deployments have different NN latency baselines. A threshold of
> 1s may be normal for one cluster and a warning sign for another. Having the
> config on the Hive side lets operators tune it to their environment
> independently.
> 6. Filesystem agnostic — works for any future FileSystem
> ipc.server.log.slow.rpc is specific to Hadoop IPC and only covers HDFS
> NameNode RPC calls. With storage backends like S3, Azure ADLS, GCS, or Vast
> S3 becoming increasingly common in Hive deployments, slow getFileInfo calls
> can occur on any FileSystem implementation — not just HDFS. Since this fix
> instruments FileUtils.getFileStatusOrNull() at the Hive level, it will
> automatically provide slow-call visibility for any FileSystem that
> storage-based authorization runs against today or in the future, without
> requiring any equivalent mechanism to exist in those storage systems.
> Both mechanisms complement each other — *{*}ipc.server.log.slow.rpc{*}*
> gives NameNode-level visibility, while this fix gives HMS-level visibility
> with operation context.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
