gaborgsomogyi commented on code in PR #24891:
URL: https://github.com/apache/flink/pull/24891#discussion_r1628273295
##########
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java:
##########
@@ -1374,6 +1374,13 @@ private void setTokensFor(ContainerLaunchContext
containerLaunchContext, boolean
for (Map.Entry<String, byte[]> e :
container.getTokens().entrySet()) {
if (e.getKey().equals("hadoopfs")) {
credentials.addAll(HadoopDelegationTokenConverter.deserialize(e.getValue()));
+ } else {
+ // Add hdfs tokens, maybe fetched by custom
DelegationTokenProvider
+ Credentials fetchedCredentials =
+
HadoopDelegationTokenConverter.deserialize(e.getValue());
Review Comment:
The assumption that all the tokens are Hadoop compatible is simply wrong.
Let's say somebody adds s3 provider, when that byte array is tried to be
deserialized with `HadoopDelegationTokenConverter` that it's going to blow up
and terminating the workload. What I can imagine as a solution is to add a YARN
specific list of tokens (service name list practically) which should be added
to the YARN launch context. Also, it would be good to add some test for
coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
