[
https://issues.apache.org/jira/browse/HDFS-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated HDFS-17899:
----------------------------
Description:
This is a follow-up of HDFS-17897.
HDFS-17897 fixed {{InvalidEncryptionKeyException}} handling in DFSClient
read/write and striped file checksum paths. However, three other
code paths that establish SASL-encrypted connections still lack this handling:
{{Dispatcher.PendingMove.dispatch()}} — Balancer block moves
{{BlockDispatcher.moveBlock()}} — SPS block moves
{{DataNode.DataTransfer.run()}} — DataNode block replication
When encrypted data transfer is enabled, the Balancer uses a cached data
encryption key to establish a SASL-encrypted connection with a DataNode during
block movement. If block keys rotate, or if HA failover/RPC failure prevents
the Balancer KeyUpdater from refreshing its key state, the Dispatcher may
continue using a stale encryption key. Meanwhile, a DataNode may have already
refreshed or rebuilt its key state, for example after restart, and no longer
contains the corresponding key.
As a result, the DataNode rejects the connection with
InvalidEncryptionKeyException. The bug is that
Dispatcher.PendingMove.dispatch() does not clear the cached stale key and retry
with a fresh one. Therefore, the same invalid key can be reused for later
Balancer moves, causing repeated transfer failures until the key is refreshed
or the process restarts.
Fix: Add the same retry pattern to all three paths — catch the exception,
clear the cached encryption key via a new {{clearDataEncryptionKey()}} default
method on {{DataEncryptionKeyFactory}}, and retry once with a fresh key.
was:
This is a follow-up of HDFS-17897.
HDFS-17897 fixed {{InvalidEncryptionKeyException}} handling in DFSClient
read/write and striped file checksum paths. However, three other
code paths that establish SASL-encrypted connections still lack this handling:
{{Dispatcher.PendingMove.dispatch()}} — Balancer block moves
{{BlockDispatcher.moveBlock()}} — SPS block moves
{{DataNode.DataTransfer.run()}} — DataNode block replication
When {{dfs.encrypt.data.transfer=true}} and block keys rotate, these paths
are possible to fail with {{InvalidEncryptionKeyException}} and the stale key
stays cached, causing all subsequent transfers to fail until process restart.
Fix: Add the same retry pattern to all three paths — catch the exception,
clear the cached encryption key via a new {{clearDataEncryptionKey()}} default
method on {{DataEncryptionKeyFactory}}, and retry once with a fresh key.
> Handle InvalidEncryptionKeyException in Balancer Dispatcher, SPS
> BlockDispatcher and DataNode DataTransfer
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17899
> URL: https://issues.apache.org/jira/browse/HDFS-17899
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, datanode, encryption
> Affects Versions: 3.4.1
> Reporter: ZhenyuLi
> Assignee: ZhenyuLi
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
>
> This is a follow-up of HDFS-17897.
> HDFS-17897 fixed {{InvalidEncryptionKeyException}} handling in DFSClient
> read/write and striped file checksum paths. However, three other
> code paths that establish SASL-encrypted connections still lack this
> handling:
> {{Dispatcher.PendingMove.dispatch()}} — Balancer block moves
> {{BlockDispatcher.moveBlock()}} — SPS block moves
> {{DataNode.DataTransfer.run()}} — DataNode block replication
> When encrypted data transfer is enabled, the Balancer uses a cached data
> encryption key to establish a SASL-encrypted connection with a DataNode
> during block movement. If block keys rotate, or if HA failover/RPC failure
> prevents the Balancer KeyUpdater from refreshing its key state, the
> Dispatcher may continue using a stale encryption key. Meanwhile, a DataNode
> may have already refreshed or rebuilt its key state, for example after
> restart, and no longer contains the corresponding key.
> As a result, the DataNode rejects the connection with
> InvalidEncryptionKeyException. The bug is that
> Dispatcher.PendingMove.dispatch() does not clear the cached stale key and
> retry with a fresh one. Therefore, the same invalid key can be reused for
> later Balancer moves, causing repeated transfer failures until the key is
> refreshed or the process restarts.
> Fix: Add the same retry pattern to all three paths — catch the exception,
> clear the cached encryption key via a new {{clearDataEncryptionKey()}}
> default
> method on {{DataEncryptionKeyFactory}}, and retry once with a fresh key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
