[
https://issues.apache.org/jira/browse/FLINK-41435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LuChengxiang updated FLINK-41435:
---------------------------------
Release Note: SecurityUtils.install is now idempotent for equivalent
security configurations. This avoids a second Kerberos login that could replace
the Hadoop login UGI in Kubernetes and standalone application mode. (was:
SecurityUtils.install is now idempotent for equivalent security configurations.
This avoids a second Kerberos login that could replace the Hadoop login UGI in
application mode.)
Description:
*Background*
In Kubernetes and standalone application mode, Flink installs the security
context twice:
* KubernetesApplicationClusterEntrypoint and
StandaloneApplicationClusterEntryPoint call
ClusterEntrypoint.installSecurityContext() before fetching user artifacts.
* ClusterEntrypoint.startCluster() calls installSecurityContext() again.
Both calls are part of the normal startup path. Each call creates a new
SecurityConfiguration and runs the security modules and context factory again.
With Kerberos this performs another login and can replace the process login UGI.
Components created under the first UGI, for example Hadoop clients used while
fetching artifacts, then remain bound to a UGI that no longer receives
refreshed delegation tokens. The refresh path can update a different UGI than
the one used by those clients, so token refresh can silently stop; the job
later fails with errors such as SecretManager$InvalidToken or an expired HDFS
delegation token.
*How it is triggered*
This happens in Kubernetes and standalone application mode when Kerberos is
enabled. The first install is required before fetching user artifacts; the
second install is part of ClusterEntrypoint.startCluster(). The second call
cannot simply be removed, because startCluster() still needs a security context.
A naive comparison of the two Configuration objects would also not help:
ApplicationClusterEntryPoint.configureExecution() adds pipeline.jars,
pipeline.classpaths and execution.target between the two calls. The
security-relevant options are unchanged, so the fix must compare only the
options consumed by the security modules.
*Root cause*
SecurityUtils.install() unconditionally runs the security modules and creates a
new security context. It does not remember the last successfully installed
configuration, so an equivalent configuration is treated as a real
re-installation.
*Proposed change*
* Give SecurityConfiguration value semantics for the options consumed by the
bundled security modules: security.*, zookeeper.sasl.*, flink.hadoop.*,
fs.hdfs.* and io.tmp.dirs. Include the configured security context and module
factories.
* Remember the last successfully installed SecurityConfiguration in
SecurityUtils.
* Make SecurityUtils.install() synchronized and return early when the new
configuration equals the installed one.
* Clear the remembered configuration in SecurityUtils.uninstall().
* Do not reinstall because of unrelated runtime options such as pipeline.jars,
pipeline.classpaths or execution.target.
*Scope*
This issue only covers idempotent installation. Delegation token
receiver/provider target-UGI handling, partial-install rollback and file-system
cleanup during token acquisition are out of scope. The fix keeps a single UGI
for the two Flink install calls; it does not make the process resilient to
arbitrary later changes of the global Hadoop login user by user code or
connectors.
*Testing*
* SecurityUtilsInstallTest: an equal configuration after adding pipeline
options does not reinstall; a different security-relevant configuration does
reinstall; install after uninstall installs again.
* SecurityConfigurationTest: security-relevant prefixes affect equality;
unrelated options do not.
* Existing SecurityUtilsTest still passes.
* Verified on master (2.5-SNAPSHOT, Java 17, Hadoop 3.4.3): 16 tests pass.
*Acceptance criteria*
* Repeated install() with an equivalent security configuration keeps the
existing context and modules.
* A different security-relevant configuration still installs a new context and
modules.
* Unrelated runtime options do not trigger reinstallation.
* uninstall() clears the stored installation state.
was:
*Background*
In application mode, Flink installs the security context twice:
* KubernetesApplicationClusterEntrypoint and
StandaloneApplicationClusterEntryPoint call
ClusterEntrypoint.installSecurityContext() before fetching user artifacts.
* ClusterEntrypoint.startCluster() calls installSecurityContext() again.
Both calls are part of the normal startup path. Each call creates a new
SecurityConfiguration and runs the security modules and context factory again.
With Kerberos this performs another login and can replace the process login UGI.
Components created under the first UGI, for example Hadoop clients used while
fetching artifacts, then remain bound to a UGI that no longer receives
refreshed delegation tokens. Token refresh can silently stop; the job later
fails with errors such as SecretManager$InvalidToken or an expired HDFS
delegation token.
*How it is triggered*
This happens in Kubernetes and standalone application mode when Kerberos is
enabled. The first install is required before fetching user artifacts; the
second install is part of ClusterEntrypoint.startCluster(). The second call
cannot simply be removed, because startCluster() still needs a security context.
A naive comparison of the two Configuration objects would also not help:
ApplicationClusterEntryPoint.configureExecution() adds pipeline.jars,
pipeline.classpaths and execution.target between the two calls. The
security-relevant options are unchanged, so the fix must compare only the
options consumed by the security modules.
*Root cause*
SecurityUtils.install() unconditionally runs the security modules and creates a
new security context. It does not remember the last successfully installed
configuration, so an equivalent configuration is treated as a real
re-installation.
*Proposed change*
* Give SecurityConfiguration value semantics for the options consumed by the
bundled security modules: security.*, zookeeper.sasl.*, flink.hadoop.*,
fs.hdfs.* and io.tmp.dirs. Include the configured security context and module
factories.
* Remember the last successfully installed SecurityConfiguration in
SecurityUtils.
* Make SecurityUtils.install() synchronized and return early when the new
configuration equals the installed one.
* Clear the remembered configuration in SecurityUtils.uninstall().
* Do not reinstall because of unrelated runtime options such as pipeline.jars,
pipeline.classpaths or execution.target.
*Scope*
This issue only covers idempotent installation. Delegation token
receiver/provider target-UGI handling, module rollback and file-system cleanup
during token acquisition are out of scope.
*Testing*
* Install an equal configuration after adding pipeline options: modules and
context must be unchanged.
* Change a security-relevant option: modules and context must be reinstalled.
* Install after uninstall: a new context/modules must be installed.
* Security-relevant prefixes affect equality; unrelated options do not.
*Acceptance criteria*
* Repeated install() with an equivalent security configuration keeps the
existing context and modules.
* A different security-relevant configuration still installs a new context and
modules.
* uninstall() clears the stored installation state.
Environment: Flink 1.20.x and 2.0 through 2.4, plus master (2.5-SNAPSHOT);
Kubernetes and standalone application mode with Kerberos and HDFS delegation
tokens. (was: Flink 1.20.x and 2.0 through 2.4 (including master) Kubernetes
Application Mode with Kerberos and HDFS delegation tokens.)
> SecurityUtils.install should be idempotent for equivalent security
> configurations
> ---------------------------------------------------------------------------------
>
> Key: FLINK-41435
> URL: https://issues.apache.org/jira/browse/FLINK-41435
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
> Affects Versions: 2.0.0, 1.20.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
> Environment: Flink 1.20.x and 2.0 through 2.4, plus master
> (2.5-SNAPSHOT); Kubernetes and standalone application mode with Kerberos and
> HDFS delegation tokens.
> Reporter: LuChengxiang
> Priority: Major
> Labels: pull-request-available
>
> *Background*
> In Kubernetes and standalone application mode, Flink installs the security
> context twice:
> * KubernetesApplicationClusterEntrypoint and
> StandaloneApplicationClusterEntryPoint call
> ClusterEntrypoint.installSecurityContext() before fetching user artifacts.
> * ClusterEntrypoint.startCluster() calls installSecurityContext() again.
> Both calls are part of the normal startup path. Each call creates a new
> SecurityConfiguration and runs the security modules and context factory
> again. With Kerberos this performs another login and can replace the process
> login UGI.
> Components created under the first UGI, for example Hadoop clients used while
> fetching artifacts, then remain bound to a UGI that no longer receives
> refreshed delegation tokens. The refresh path can update a different UGI than
> the one used by those clients, so token refresh can silently stop; the job
> later fails with errors such as SecretManager$InvalidToken or an expired HDFS
> delegation token.
> *How it is triggered*
> This happens in Kubernetes and standalone application mode when Kerberos is
> enabled. The first install is required before fetching user artifacts; the
> second install is part of ClusterEntrypoint.startCluster(). The second call
> cannot simply be removed, because startCluster() still needs a security
> context.
> A naive comparison of the two Configuration objects would also not help:
> ApplicationClusterEntryPoint.configureExecution() adds pipeline.jars,
> pipeline.classpaths and execution.target between the two calls. The
> security-relevant options are unchanged, so the fix must compare only the
> options consumed by the security modules.
> *Root cause*
> SecurityUtils.install() unconditionally runs the security modules and creates
> a new security context. It does not remember the last successfully installed
> configuration, so an equivalent configuration is treated as a real
> re-installation.
> *Proposed change*
> * Give SecurityConfiguration value semantics for the options consumed by the
> bundled security modules: security.*, zookeeper.sasl.*, flink.hadoop.*,
> fs.hdfs.* and io.tmp.dirs. Include the configured security context and module
> factories.
> * Remember the last successfully installed SecurityConfiguration in
> SecurityUtils.
> * Make SecurityUtils.install() synchronized and return early when the new
> configuration equals the installed one.
> * Clear the remembered configuration in SecurityUtils.uninstall().
> * Do not reinstall because of unrelated runtime options such as
> pipeline.jars, pipeline.classpaths or execution.target.
> *Scope*
> This issue only covers idempotent installation. Delegation token
> receiver/provider target-UGI handling, partial-install rollback and
> file-system cleanup during token acquisition are out of scope. The fix keeps
> a single UGI for the two Flink install calls; it does not make the process
> resilient to arbitrary later changes of the global Hadoop login user by user
> code or connectors.
> *Testing*
> * SecurityUtilsInstallTest: an equal configuration after adding pipeline
> options does not reinstall; a different security-relevant configuration does
> reinstall; install after uninstall installs again.
> * SecurityConfigurationTest: security-relevant prefixes affect equality;
> unrelated options do not.
> * Existing SecurityUtilsTest still passes.
> * Verified on master (2.5-SNAPSHOT, Java 17, Hadoop 3.4.3): 16 tests pass.
> *Acceptance criteria*
> * Repeated install() with an equivalent security configuration keeps the
> existing context and modules.
> * A different security-relevant configuration still installs a new context
> and modules.
> * Unrelated runtime options do not trigger reinstallation.
> * uninstall() clears the stored installation state.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
