luoyuxia commented on PR #2569:
URL: https://github.com/apache/fluss/pull/2569#issuecomment-3868929801
> ~Found further issues upon enabling snapshots.. debugging...~
>
> Paimon / Iceberg tiering are working, currently kv snapshot that are
causing failures within lakehouse examples.
>
> I have kept snapshots disabled for streaming lakehouse Paimon and Iceberg
examples. The reason is that the tiering job failed due to the following:
>
> 1. When kv snapshot is enabled, the tiering job uses
KvSnapshotBatchScanner, which requires credentials through the token delegation
mechanism. It might be my lack of familiarity around this area but I am not
entirely sure why KvSnapshotBatchScanner is involved here, my expectation would
to for the tiering job to read from the kv table and existing tiered storage in
Paimon instead of snapshots even with snapshot configured..
> 2. No credentials was supplied through token delegation mechanism because
the configuration passed to S3FileSystemPlugin was completely empty. I was not
able to trace the entry point through which this configuration can be provided.
>
> See stack trace:
>
> ```
> 2026-02-08 17:55:21,430 WARN org.apache.flink.runtime.taskmanager.Task
[] - Source: TieringSource -> TieringCommitter -> end: Writer
(1/1)#0 (105de9762bb2ebef68e1adde3cd3686c_dc9385c83dde82d19e268af15cd693e7_0_0)
switched from RUNNING to FAILED with failure cause:
> java.lang.RuntimeException: One or more fetchers have encountered exception
> at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:333)
~[flink-connector-files-1.20.3.jar:1.20.3]
> ...
> Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received
unexpected exception while polling the records
> at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:168)
~[flink-connector-files-1.20.3.jar:1.20.3]
> ...
> Caused by: org.apache.hadoop.fs.s3a.auth.NoAwsCredentialsException:
Dynamic session credentials for Fluss: No AWS Credentials
> at
org.apache.fluss.fs.s3.token.S3DelegationTokenReceiver.updateHadoopConfig(S3DelegationTokenReceiver.java:73)
~[fluss-fs-s3-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.fs.s3.S3FileSystemPlugin.setCredentialProvider(S3FileSystemPlugin.java:141)
~[fluss-fs-s3-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.fs.s3.S3FileSystemPlugin.create(S3FileSystemPlugin.java:65)
~[fluss-fs-s3-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.fs.FileSystem.lambda$getUnguardedFileSystem$3(FileSystem.java:405)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:42)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:310)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at org.apache.fluss.fs.FileSystem.get(FileSystem.java:305)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at org.apache.fluss.fs.FsPath.getFileSystem(FsPath.java:265)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.fs.utils.FileDownloadUtils.downloadFile(FileDownloadUtils.java:139)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.fs.utils.FileDownloadUtils.lambda$transferDataToDirectoryAsync$1(FileDownloadUtils.java:117)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
org.apache.fluss.utils.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:29)
~[fluss-flink-1.20-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
> at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) ~[?:?]
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) ~[?:?]
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) ~[?:?]
> ... 1 more
> ```
IIRC, looks similar to the issue with minio #214 , which can not generate
sts token for clients to use to read the snapshot files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
