Hi Vishal Maybe the slide[1] (page 40) can be helpful [1] https://files.alicdn.com/tpsservice/c421720fcb1c51026257cd770923844a.pdf Best, Congxian
Vishal Sharma <vishal.sha...@grab.com> 于2019年6月20日周四 下午5:42写道： > Hi Congxian, > > I am not sure how can I track the checkpoint path. Can you suggestion > regarding this ? > > Thanks, > Vishal Sharma > > On Thu, Jun 20, 2019 at 11:17 AM Congxian Qiu <qcx978132...@gmail.com> > wrote: > >> Hi, Vishal >> If you want to restart from the last competed external checkpoint of the >> previous stoped job, you need to track the checkpoint path and restart from >> it. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint >> >> Best, >> Congxian >> >> >> Vishal Sharma <vishal.sha...@grab.com> 于2019年6月19日周三 下午11:38写道： >> >>> Hi Chesnay, >>> >>> Can you suggest, How should I go about automating job restart from last >>> completed externalised checkpoint in case of failure ? I am not sure about >>> the path for the latest completed checkpoint. >>> >>> Thanks, >>> Vishal Sharma >>> >>> On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <ches...@apache.org> >>> wrote: >>> >>>> The _metadata is always stored in the same directory as the checkpoint >>>> data. >>>> >>>> As outlined here >>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html#directory-structure> >>>> "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ >>>> be overwritten with a job-specific setting when creating the state-backend. >>>> >>>> If you want the state-backend to use the configured directory you must >>>> configure the state-backend in the configuration as well, as outlined >>>> here >>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#setting-default-state-backend> >>>> . >>>> >>>> On 19/06/2019 16:26, Vishal Sharma wrote: >>>> >>>> Hi Folks, >>>> >>>> I am using flink 1.8 with externalised checkpointing enabled and saving >>>> the checkpoints to aws S3. >>>> >>>> My configuration is as follows : >>>> >>>> flink-conf.yaml : >>>> state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata >>>> >>>> In application code : >>>> env.setStateBackend(new >>>> RocksDBStateBackend("s3a://test-bucket/checkpoints", true)) >>>> >>>> As per my understanding, the externalized checkpoint’s meta data is >>>> determined from the configuration key "state.checkpoints.dir" and >>>> checkpoint data is stored in state backend path. >>>> >>>> However, In my case, I don't see anything in the metadata directory. >>>> The _metadata file is present inside each of the checkpoint directory >>>> (chk-6043 >>>> ...). >>>> >>>> Is this the expected behavior ? If yes, what is the use of >>>> "state.checkpoints.dir" configuration ? >>>> >>>> My goal is to establish a process to automatically restart the job from >>>> last completed externalised checkpoint in case of failure. For this >>>> to happen, I need to able to figure out path for the metadata of latest >>>> checkpoint. >>>> >>>> Thanks, >>>> Vishal Sharma >>>> >>>> *Grab is hiring. Learn more at https://grab.careers >>>> <https://grab.careers/>* >>>> >>>> By communicating with Grab Inc and/or its subsidiaries, associate >>>> companies and jointly controlled entities (“Grab Group”), you are deemed to >>>> have consented to the processing of your personal data as set out in the >>>> Privacy Notice which can be viewed at https://grab.com/privacy/ >>>> >>>> This email contains confidential information and is only for the >>>> intended recipient(s). If you are not the intended recipient(s), please do >>>> not disseminate, distribute or copy this email Please notify Grab Group >>>> immediately if you have received this by mistake and delete this email from >>>> your system. Email transmission cannot be guaranteed to be secure or >>>> error-free as any information therein could be intercepted, corrupted, >>>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do >>>> not accept liability for any errors or omissions in the contents of this >>>> email arises as a result of email transmission. All intellectual property >>>> rights in this email and attachments therein shall remain vested in Grab >>>> Group, unless otherwise provided by law. >>>> >>>> >>>> >>> *Grab is hiring. Learn more at https://grab.careers >>> <https://grab.careers/>* >>> >>> By communicating with Grab Inc and/or its subsidiaries, associate >>> companies and jointly controlled entities (“Grab Group”), you are deemed to >>> have consented to the processing of your personal data as set out in the >>> Privacy Notice which can be viewed at https://grab.com/privacy/ >>> >>> This email contains confidential information and is only for the >>> intended recipient(s). If you are not the intended recipient(s), please do >>> not disseminate, distribute or copy this email Please notify Grab Group >>> immediately if you have received this by mistake and delete this email from >>> your system. Email transmission cannot be guaranteed to be secure or >>> error-free as any information therein could be intercepted, corrupted, >>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do >>> not accept liability for any errors or omissions in the contents of this >>> email arises as a result of email transmission. All intellectual property >>> rights in this email and attachments therein shall remain vested in Grab >>> Group, unless otherwise provided by law. >>> >> > *Grab is hiring. Learn more at https://grab.careers > <https://grab.careers/>* > > By communicating with Grab Inc and/or its subsidiaries, associate > companies and jointly controlled entities (“Grab Group”), you are deemed to > have consented to the processing of your personal data as set out in the > Privacy Notice which can be viewed at https://grab.com/privacy/ > > This email contains confidential information and is only for the intended > recipient(s). If you are not the intended recipient(s), please do not > disseminate, distribute or copy this email Please notify Grab Group > immediately if you have received this by mistake and delete this email from > your system. Email transmission cannot be guaranteed to be secure or > error-free as any information therein could be intercepted, corrupted, > lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do > not accept liability for any errors or omissions in the contents of this > email arises as a result of email transmission. All intellectual property > rights in this email and attachments therein shall remain vested in Grab > Group, unless otherwise provided by law. >
