Aleksei Ieshin created HDFS-17964:
-------------------------------------
Summary: HDFS delegation token not found in cache errors seen on
Observer NameNodes
Key: HDFS-17964
URL: https://issues.apache.org/jira/browse/HDFS-17964
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Reporter: Aleksei Ieshin
Observer NameNodes fatally reject delegation tokens they have not yet tailed.
`DelegationTokenSecretManager.retriableRetrievePassword` and
`FSNamesystem.verifyToken` wrap `InvalidToken` in a `RetriableException` only
when `inTransitionToActive()`. An observer in steady state never is, so a token
the active issued moments ago fails with
```
InvalidToken: Token for real user: <user>, can't be found in cache
```
HDFS-5322 added that guard in 2.3.0, before Observer NameNodes existed
(HDFS-12943). Widening it to cover observers fixes both paths.
h6. Details:
The common trigger is WebHDFS op=OPEN, the one operation with a second hop:
The common trigger is WebHDFS op=OPEN, the one operation with a second hop: the
NameNode redirects to a DataNode, and that DataNode validates the token against
a NameNode chosen by its own ConfiguredFailoverProxyProvider. Nothing the
client configures influences that choice, so in a three-NameNode HA setup with
one observer a substantial fraction of reads land on the observer and fail with
HTTP 403.
Fix: treat an observer as possibly behind on tailing at both sites, so a failed
lookup yields RetriableException. Callers already handle it - that is what
HDFS-5322 relies on. No config, no new mechanism.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
