Hi 1. I think this issue[1] can help to understand the directory layout 2. chk-6 directory or the metafilePath, for more information, you can ref to[2][3] 3. every checkpoint contains a meta-file record such, maybe you can ref to[4], and debug it for more information 4. currently, you need to go through the restore logic to see if the files' contents
[1] https://issues.apache.org/jira/browse/FLINK-8531 [2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1153 [3] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint [4] https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointMetadataLoadingTest.java Best, Congxian M Singh <mans2si...@yahoo.com> 于2019年11月21日周四 下午7:44写道： > Hi Congxian: > > For my application i see many uuids under the chk-6 directory ( I posted > one in the sample above). I am trying to understand that if I restart the > application with this checkpoint (which I believe I can just like a > savepoint - I am using chk-6 as an example below) > 1. I believe each chk-<d> - is a complete state of checkpoint. Is that > correct ? > 2. How to I point it to the checkpoint (chk-6) when I submit the job - Do > I point it to the jobid or the chk-6 directory ? I am presuming the latter > (ie, pointing to the chk06 directory but just want to confirm. > 3. Secondly, how does the application map the the files under the chk-6 to > restore the state of each of the stateful operators ? > 4. Is there any API by which I can examine the contents of the files under > the chk-6 directory ? > > Thanks for your help. > > Mans > > On Wednesday, November 20, 2019, 09:13:39 PM EST, Congxian Qiu < > qcx978132...@gmail.com> wrote: > > > Hi > > Currently, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) is > generated by UUID.randomUUID(), so there is not a easy way to map this to > the assigned in the application. > In another word, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) > belongs > to one checkpoint, and the assigned in the application belongs to one > operator, they are different. > > Best, > Congxian > > > M Singh <mans2si...@yahoo.com> 于2019年11月21日周四 上午6:18写道： > > Hi Arvid: > > Thanks for your clarification. > > I am giving supplying uid for the stateful operators and find the > following directory structure on in the chkpoint directory: > > f4e78cb47f9dc12859558be7d15f39d0/*chk* > -6/a4d87cda-2afd-47d4-8d3f-b0658466fb2d > The first part f4e78cb47f9dc12859558be7d15f39d0 is the job_id > Is there a way to map the last part (uuid > a4d87cda-2afd-47d4-8d3f-b0658466fb2d) - to the uid assigned in the > application ? > > Thanks > > On Wednesday, November 20, 2019, 07:52:49 AM EST, Arvid Heise < > ar...@ververica.com> wrote: > > > Hi Mans, > > just to follow up. There are no limitations for name or uuid. > > The uuid will be in fact hashed internally while the StreamGraph is being > generated, so all characters are allowed. > The name is only for debugging purposes and web ui. If you use very > special characters, you may see oddities in logs/web ui, but nothing should > break. > Spaces or parentheses should work in any case. > > Best, > > Arvid > > On Sat, Nov 16, 2019 at 6:40 PM M Singh <mans2si...@yahoo.com> wrote: > > Thanks Jiayi for your response. I am thinking on the same lines. > > Regarding using the same name and uuid, I believe the checkpoint state for > an operator will be easy to identify if the uuid is the same as name. But > I am not sure if having a very long name and uuid or a character like > parenthesis, etc might cause any issues, so just wanted to check. > > Mans > > On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao < > bupt_...@163.com> wrote: > > > Hi Mans! > > > Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s > name is used in WebUI and metrics reporting, and uid is used to mark the > uniqueness of operator which is useful when you’re using state[1]. > > > > Are there any restrictions on the length of the name and uuid attributes? > > It’s pretty much the same as you define a string value, so there is no > special restrictions on this. > > > > Are there any restrictions on the characters used for name and uuid > (blank spaces, etc) ? > > I’m not a hundred percent sure about this but I run a testing program and > it works fine. > > > > Can the name and uuid be the same ? > > Yes. But uuids accross operators cannot be same. > > > For me I usually set name and uuid for almost every operator, which gives > me better experience in monitoring and scaling. > > > Hope this helps. > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state > > > > Best, > > Jiayi Liao > > > At 2019-11-16 18:35:38, "M Singh" <mans2si...@yahoo.com> wrote: > > Hi: > > I am working on a project and wanted to find out what are the best > practices for setting name and uuid for operators: > > 1. Are there any restrictions on the length of the name and uuid > attributes ? > 2. Are there any restrictions on the characters used for name and uuid > (blank spaces, etc) ? > 3. Can the name and uuid be the same ? > > Please let me know if there is any other advice. > > Thanks > > Mans > > > > > >
