Hi Folks,
I am using flink 1.8 with externalised checkpointing enabled and saving the
checkpoints to aws S3.
My configuration is as follows :
flink-conf.yaml :
state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
In application code :
env.setStateBackend(new
RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
As per my understanding, the externalized checkpoint’s meta data is
determined from the configuration key "state.checkpoints.dir" and
checkpoint data is stored in state backend path.
However, In my case, I don't see anything in the metadata directory. The
_metadata file is present inside each of the checkpoint directory (chk-6043
...).
Is this the expected behavior ? If yes, what is the use of
"state.checkpoints.dir" configuration ?
My goal is to establish a process to automatically restart the job from
last completed externalised checkpoint in case of failure. For this
to happen, I need to able to figure out path for the metadata of latest
checkpoint.
Thanks,
Vishal Sharma
--
*_Grab is hiring. Learn more at _**https://grab.careers
<https://grab.careers/>*
By communicating with Grab Inc and/or its
subsidiaries, associate companies and jointly controlled entities (“Grab
Group”), you are deemed to have consented to the processing of your
personal data as set out in the Privacy Notice which can be viewed at
https://grab.com/privacy/ <https://grab.com/privacy/>
This email contains
confidential information and is only for the intended recipient(s). If you
are not the intended recipient(s), please do not disseminate, distribute or
copy this email Please notify Grab Group immediately if you have received
this by mistake and delete this email from your system. Email transmission
cannot be guaranteed to be secure or error-free as any information therein
could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or
contain viruses. Grab Group do not accept liability for any errors or
omissions in the contents of this email arises as a result of email
transmission. All intellectual property rights in this email and
attachments therein shall remain vested in Grab Group, unless otherwise
provided by law.
