Triggering Savepoint fails to write data to S3 store

2021-05-27 Thread Robert Cullen
I triggered a savepoint from a currently running job. Although the directory structure gets created in the MINIO S3 store, the command ultimately fails without writing the data. root@flink-client:/opt/flink# ./bin/flink list --target kubernetes-session -Dkubernetes.cluster-id=flink-jobmanager -Dku

Re: Triggering Savepoint fails to write data to S3 store

2021-05-28 Thread Matthias Pohl
Hi Robert, it would be interesting to see the corresponding taskmanager/jobmanager logs. That would help in finding out why the savepoint creation failed. Just to verify: The savepoint data wasn't written to S3 even after the timeout happened, was it? Best, Matthias On Thu, May 27, 2021 at 7:50 P

Re: Triggering Savepoint fails to write data to S3 store

2021-05-28 Thread Robert Cullen
Hi Matthias, You are correct. After a few minutes I took another look at my savepoint folder and the data was there. I think increasing the timeout may resolve the problem? On Fri, May 28, 2021 at 8:21 AM Matthias Pohl wrote: > Hi Robert, > it would be interesting to see the corresponding tas

Re: Triggering Savepoint fails to write data to S3 store

2021-05-28 Thread Matthias Pohl
Yes, that would work. But it might be still interesting to understand why you ran into the timeout. Was it just a big state that just took longer than expected? Or some network issue? ...that's just for you to understand the underlying issue in a better way. But I'm glad the savepoint creation was