Hey Piotr,
Thanks for the proposal! It would be great improvement!
Some questions from my side:
In order to configure s5cmd Flink’s user would need
to specify path to the s5cmd binary.
Could you please also add the configuration property
for this? An example showing how users would set this
parameter would be helpful.
Would this affect any filesystem connectors that use
FileSystem[1][2] dependencies?
Best,
Muhammet
[1]:
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/
[2]:
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/
On 2024-04-30 13:15, Piotr Nowojski wrote:
Hi all!
I would like to put under discussion:
FLIP-444: Native file copy support
https://cwiki.apache.org/confluence/x/rAn9EQ
This proposal aims to speed up Flink recovery times, by speeding up
state
download times. However in the future, the same mechanism could be also
used to speed up state uploading (checkpointing/savepointing).
I'm curious to hear your thoughts.
Best,
Piotrek
