[
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LuChengxiang updated FLINK-41435:
---------------------------------
Description:
*Background*
Flink periodically obtains HDFS delegation tokens and distributes them through
HadoopDelegationTokenReceiver. The receiver currently adds the new credentials
to UserGroupInformation.getCurrentUser() on the refresh thread. Long-lived
Hadoop clients capture a UGI when their DFSClient is created and use tokens
from that UGI for later RPC connections.
If the refresh thread's current UGI is not the UGI captured by the client, the
client continues using the startup token even after a new token was created
successfully. The job later fails with SecretManager$InvalidToken at the 24h
deadline.
*Observed failure*
This was observed in a Kubernetes application-mode job. The refresh created a
new HDFS token and the receiver reported success, but FileSystemJobResultStore
and checkpoint storage continued using the startup token. An earlier test build
that skipped the second security installation (`Security configuration is
already installed`) still hit the 24h deadline; install idempotency alone is
therefore not sufficient.
FLINK-27204 (1.19+) moved FileSystemJobResultStore I/O onto the shared
cluster-io executor. DefaultDelegationTokenManager uses the same executor for
periodic token refresh. The provider and receiver are invoked sequentially on
the same refresh thread. The provider's freshUGI.doAs() scope ends before the
receiver runs, so the receiver's getCurrentUser() reflects the Subject
inherited by the cluster-io worker, not the provider's short-lived UGI.
The 1.18 path had synchronous FileSystemJobResultStore I/O, so this
worker/Subject interaction was not triggered in the deployments observed. 1.15
did not have the unified delegation-token manager path. These version
differences explain the observed frequency; they do not prove that those
versions are immune.
YARN with ZooKeeper HA uses the same FileSystemJobResultStore(ioExecutor) and
token-refresh code path, but this has not been reproduced on YARN. YARN is
therefore not listed as a confirmed affected deployment type.
*Root cause*
HadoopDelegationTokenReceiver targets getCurrentUser() of the refresh thread.
That target is not guaranteed to be the UGI held by HadoopSecurityContext or
the UGI captured by a long-lived FileSystemJobResultStore/checkpoint DFSClient.
*Proposed change*
This issue addresses the case where a token is obtained successfully but the
refreshed credentials do not reach long-lived Hadoop clients.
* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, add refreshed credentials to the
security-context UGI and login UGI, with the current UGI as a fallback,
deduplicated by object identity.
* Add a test where the refresh thread's current user differs from the
security-context user, and verify that the security-context/login UGI receives
the credentials.
Out of scope for this issue:
* The separate case where HadoopFSDelegationTokenProvider evaluates
delegation-token requirements against the wrong current UGI and does not obtain
a token at all.
* Closing FileSystem instances created under the short-lived fresh UGI; this is
separate lifecycle cleanup.
* SecurityUtils.install idempotency.
*Testing*
* A JVM probe confirmed that a fixed thread pool created under Subject A but
first submitted under Subject B runs subsequent tasks under Subject B.
* Receiver test as described above.
* A deployed application-mode job should survive the 24h token deadline and
keep checkpointing without SecretManager$InvalidToken.
*Acceptance criteria*
* A token refresh updates the UGI used by FileSystemJobResultStore/checkpoint
DFSClient.
* Refresh does not depend on the Subject inherited by the shared cluster-io
worker.
* A long-running application-mode job survives the 24h HDFS token deadline.
was:
*Background*
Flink periodically obtains HDFS delegation tokens and distributes them through
HadoopDelegationTokenReceiver. The receiver currently adds the new credentials
to UserGroupInformation.getCurrentUser() on the refresh thread. Long-lived
Hadoop clients capture a UGI when their DFSClient is created, and subsequent
RPC connections use tokens from that captured UGI.
If the refresh thread's current UGI is not the UGI captured by the client, the
client continues using the startup token even though a new token was obtained
successfully. The job later fails with SecretManager$InvalidToken when the
startup token reaches its 24h deadline.
*How it is triggered*
This was observed in a Kubernetes application-mode job with no
connector-specific security setup. A token refresh generated a new HDFS token
and the receiver logged success, but FileSystemJobResultStore and checkpoint
storage continued using the token created at startup. An earlier test build
already skipped the second security installation (`Security configuration is
already installed`) but still hit the 24h delegation-token deadline. This
confirms that install idempotency alone does not address the issue; the
token-refresh target UGI is still wrong.
FLINK-27204 (1.19+) moved FileSystemJobResultStore I/O onto the shared
cluster-io executor. DefaultDelegationTokenManager uses the same executor for
periodic token refresh. Java thread-pool workers inherit the
AccessControlContext/Subject of the thread that first creates the worker, not
the Subject at executor construction. A worker can therefore see a different
UGI through getCurrentUser() than the login/security-context UGI captured by
the Hadoop client.
This is consistent with why the older 1.18 path did not show the issue:
FileSystemJobResultStore I/O was synchronous, and the token-refresh executor
was not shared with JRS I/O. 1.15 did not have the unified delegation-token
manager path at all.
*Root cause*
HadoopDelegationTokenReceiver targets getCurrentUser() of the refresh thread.
That target is not guaranteed to be the UGI held by the HadoopSecurityContext
or the UGI captured by a long-lived FileSystemJobResultStore / checkpoint
DFSClient. HadoopFSDelegationTokenProvider also performs its
Kerberos/token-required checks using the current user and creates a short-lived
UGI for token acquisition.
*Proposed change*
* Add an accessor for the UGI held by HadoopSecurityContext.
* In HadoopDelegationTokenReceiver, add refreshed credentials to the
security-context UGI, current UGI, and login UGI, deduplicated by object
identity.
* In HadoopFSDelegationTokenProvider, resolve delegation-token necessity under
the security-context UGI and close FileSystem instances created under the
short-lived login UGI.
* Add regression coverage for a refresh thread whose current user differs from
the security-context user.
*Scope*
This issue covers receiver/provider UGI targeting and client credential
visibility. It does not cover SecurityUtils.install idempotency,
partial-install rollback, or arbitrary later changes to the process login user
by user code.
*Testing*
* A JVM probe confirmed that a fixed thread pool created under Subject A but
first submitted under Subject B runs subsequent tasks under Subject B.
* Unit/integration coverage should verify that refreshed credentials reach the
UGI held by HadoopSecurityContext and the login/current UGIs used by long-lived
clients.
* A deployed build with the focused fix should survive the 24h token deadline
and continue checkpointing without SecretManager$InvalidToken.
*Acceptance criteria*
* Token refresh updates the UGI used by FileSystemJobResultStore and checkpoint
DFSClient.
* Refresh does not depend on the Subject inherited by the shared cluster-io
worker.
* A long-running job survives the 24h HDFS token deadline.
> Hadoop delegation token refresh can target a different UGI than Hadoop clients
> ------------------------------------------------------------------------------
>
> Key: FLINK-41435
> URL: https://issues.apache.org/jira/browse/FLINK-41435
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
> Affects Versions: 2.0.0, 1.19.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
> Environment: Flink 1.19.0 and later (1.19.x, 1.20.x, 2.0 through
> 2.4), plus master (2.5-SNAPSHOT); Kubernetes and standalone application mode
> with Kerberos and HDFS delegation tokens. The mismatch is more likely when
> FileSystemJobResultStore and the token manager share the cluster-io executor
> (FLINK-27204).
> Reporter: LuChengxiang
> Priority: Major
>
> *Background*
> Flink periodically obtains HDFS delegation tokens and distributes them
> through HadoopDelegationTokenReceiver. The receiver currently adds the new
> credentials to UserGroupInformation.getCurrentUser() on the refresh thread.
> Long-lived Hadoop clients capture a UGI when their DFSClient is created and
> use tokens from that UGI for later RPC connections.
> If the refresh thread's current UGI is not the UGI captured by the client,
> the client continues using the startup token even after a new token was
> created successfully. The job later fails with SecretManager$InvalidToken at
> the 24h deadline.
> *Observed failure*
> This was observed in a Kubernetes application-mode job. The refresh created a
> new HDFS token and the receiver reported success, but
> FileSystemJobResultStore and checkpoint storage continued using the startup
> token. An earlier test build that skipped the second security installation
> (`Security configuration is already installed`) still hit the 24h deadline;
> install idempotency alone is therefore not sufficient.
> FLINK-27204 (1.19+) moved FileSystemJobResultStore I/O onto the shared
> cluster-io executor. DefaultDelegationTokenManager uses the same executor for
> periodic token refresh. The provider and receiver are invoked sequentially on
> the same refresh thread. The provider's freshUGI.doAs() scope ends before the
> receiver runs, so the receiver's getCurrentUser() reflects the Subject
> inherited by the cluster-io worker, not the provider's short-lived UGI.
> The 1.18 path had synchronous FileSystemJobResultStore I/O, so this
> worker/Subject interaction was not triggered in the deployments observed.
> 1.15 did not have the unified delegation-token manager path. These version
> differences explain the observed frequency; they do not prove that those
> versions are immune.
> YARN with ZooKeeper HA uses the same FileSystemJobResultStore(ioExecutor) and
> token-refresh code path, but this has not been reproduced on YARN. YARN is
> therefore not listed as a confirmed affected deployment type.
> *Root cause*
> HadoopDelegationTokenReceiver targets getCurrentUser() of the refresh thread.
> That target is not guaranteed to be the UGI held by HadoopSecurityContext or
> the UGI captured by a long-lived FileSystemJobResultStore/checkpoint
> DFSClient.
> *Proposed change*
> This issue addresses the case where a token is obtained successfully but the
> refreshed credentials do not reach long-lived Hadoop clients.
> * Add an accessor for the UGI held by HadoopSecurityContext.
> * In HadoopDelegationTokenReceiver, add refreshed credentials to the
> security-context UGI and login UGI, with the current UGI as a fallback,
> deduplicated by object identity.
> * Add a test where the refresh thread's current user differs from the
> security-context user, and verify that the security-context/login UGI
> receives the credentials.
> Out of scope for this issue:
> * The separate case where HadoopFSDelegationTokenProvider evaluates
> delegation-token requirements against the wrong current UGI and does not
> obtain a token at all.
> * Closing FileSystem instances created under the short-lived fresh UGI; this
> is separate lifecycle cleanup.
> * SecurityUtils.install idempotency.
> *Testing*
> * A JVM probe confirmed that a fixed thread pool created under Subject A but
> first submitted under Subject B runs subsequent tasks under Subject B.
> * Receiver test as described above.
> * A deployed application-mode job should survive the 24h token deadline and
> keep checkpointing without SecretManager$InvalidToken.
> *Acceptance criteria*
> * A token refresh updates the UGI used by FileSystemJobResultStore/checkpoint
> DFSClient.
> * Refresh does not depend on the Subject inherited by the shared cluster-io
> worker.
> * A long-running application-mode job survives the 24h HDFS token deadline.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
