[
https://issues.apache.org/jira/browse/FLINK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuannan Su updated FLINK-37246:
-------------------------------
Fix Version/s: 1.20.4
(was: 1.20.3)
> Delegation Token expire after enable delegation token framework
> ---------------------------------------------------------------
>
> Key: FLINK-37246
> URL: https://issues.apache.org/jira/browse/FLINK-37246
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.18.1, 1.20.0, 1.19.1
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.20.4
>
>
> We see delegation token expire after enabling the delegation token in Flink
> 1.18.
> 2025-01-30 13:51:20.967 [Checkpoint Timer] WARN
> org.apache.flink.runtime.checkpoint.CheckpointFailureManager - Failed to
> trigger or complete checkpoint 24406 for job
> 757e9207f2c8cf3f0954e578438f7f01. (0 consecutive failed attempts so far)
> org.apache.flink.runtime.checkpoint.CheckpointException: An Exception
> occurred while triggering the checkpoint. IO-problem detected.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.getCheckpointException(CheckpointCoordinator.java:2226)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:989)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:967)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:695)
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
> at
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.hadoop.ipc.RemoteException: token
> (HDFS_DELEGATION_TOKEN token 46908908 for rt_data) can't be found in cache
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1519)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at jdk.proxy2/jdk.proxy2.$Proxy22.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:598)
> at jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:411)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:348)
> at jdk.proxy2/jdk.proxy2.$Proxy23.mkdirs(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2446)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2421)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1191)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1188)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1188)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1180)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1943)
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.mkdirs(HadoopFileSystem.java:183)
> at
> org.apache.flink.runtime.state.filesystem.FsCheckpointStorageAccess.initializeLocationForCheckpoint(FsCheckpointStorageAccess.java:135)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.initializeCheckpointLocation(CheckpointCoordinator.java:827)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$4(CheckpointCoordinator.java:618)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> ... 3 common frames omitted
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
