lupan created FLINK-25603: ----------------------------- Summary: make the BlobServer use aws s3 Key: FLINK-25603 URL: https://issues.apache.org/jira/browse/FLINK-25603 Project: Flink Issue Type: Improvement Components: Runtime / Network Affects Versions: 1.13.3 Reporter: lupan
Currently, blob.storage.directory does not support using aws s3 as storage。details as follows: When I use the following configuration: {code:java} blob.storage.directory: s3://iceberg-bucket/flink/blob {code} I get the following error: {code:java} taskmanager | 2022-01-11 02:41:11,460 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Terminating TaskManagerRunner with exit code 1. taskmanager | org.apache.flink.util.FlinkException: Failed to start the TaskManagerRunner. taskmanager | at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:374) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerProcessSecurely$3(TaskManagerRunner.java:413) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:413) [flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:396) [flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:354) [flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | Caused by: java.io.IOException: Could not create storage directory for BLOB store in 's3:/iceberg-bucket/flink/blob'. taskmanager | at org.apache.flink.runtime.blob.BlobUtils.initLocalStorageDirectory(BlobUtils.java:139) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.blob.AbstractBlobCache.<init>(AbstractBlobCache.java:89) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.blob.PermanentBlobCache.<init>(PermanentBlobCache.java:93) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.blob.BlobCacheService.<init>(BlobCacheService.java:55) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:169) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:367) ~[flink-dist_2.12-1.13.3.jar:1.13.3] taskmanager | ... 5 more{code} -- This message was sent by Atlassian Jira (v8.20.1#820001)