Re: Re: Re: flink on yarn 异常停电问题咨询
Hi 一般来说只是 YARN 集群异常停电不会影响已经完成的历史 Checkpoint（最后一次 Checkpoint 可能会写 hdfs 异常） 有更详细的 JobManager 日志吗？可以先确认下 Flink 在恢复时检索到了多少个 completedCheckpoint 以及最终尝试从哪一次 cp 恢复的。 也可以尝试按照 Yanfei 所说指定历史的 cp 作为 savepoint 恢复 Best, Weihua On Fri, Mar 10, 2023 at 10:38 AM guanyq wrote: > 没有开启增量chk > 文件损坏是看了启动日志，启动日志尝试从10个chk启动，但是都因为以下块损坏启动失败了 > 错误日志为： > > java.io.IOException: Got error, status message opReadBlock > BP-1003103929-192.168.200.11-1668473836936:blk_1301252639_227512278 > received exception > org.apache.hadoop.hdfs.server.datanode.CorruptMetaHeaderException: > The meta file length 0 is less than the expected length 7, for > OP_READ_BLOCK, self=/ip:45534, remote=/ip:9866, > for file > /tmp/flink/ha/application_1678102326043_0007/completedCheckpoint58755403e33a, > for pool BP-1003103929-192.168.200.11-1668473836936 block > 1301252639_227512278 > at > org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:142) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424) > > > > > > > > > > > > > > > > > 在 2023-03-10 10:26:11，"Yanfei Lei" 写道： > >Hi 可以尝试去flink配置的checkpoint dir下面去找一找历史chk-x文件夹，如果能找到历史的chk-x，可以尝试手工指定 > chk重启[1]。 > > > >> flink任务是10个checkpoint，每个checkpoint间隔5秒，如果突然停电，为什么所有的checkpoint都损坏了。 > > >请问作业开启增量checkpoint了吗？在开启了增量的情况下，如果是比较早的一个checkpoint的文件损坏了，会影响后续基于它进行增量的checkpoint。 > > > >> checkpoint落盘的机制，这个应该和hdfs写入有关系，flink任务checkpoint成功，但是hdfs却没有落盘。 > >是观察到checkpoint dir下面没有文件吗？ > > > >[1] > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/state/savepoints/#resuming-from-savepoints > > > >guanyq 于2023年3月10日周五 08:58写道： > >> > >> 目前也想着用savepoint处理异常停电的问题 > >> 但是我这面还有个疑问： > >> flink任务是10个checkpoint，每个checkpoint间隔5秒，如果突然停电，为什么所有的checkpoint都损坏了。 > >> 就很奇怪是不是10个checkpoint都没落盘导致的。 > >> 想问下： > >> checkpoint落盘的机制，这个应该和hdfs写入有关系，flink任务checkpoint成功，但是hdfs却没有落盘。 > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> 在 2023-03-10 08:47:11，"Shammon FY" 写道： > >> >Hi > >> > > >> >我觉得Flink > >> > >作业恢复失败时，作业本身很难确定失败是checkpoint的文件块损坏之类的原因。如果你的作业做过savepoint，可以尝试从指定的savepoint恢复作业 > >> > > >> >Best, > >> >Shammon > >> > > >> >On Thu, Mar 9, 2023 at 10:06 PM guanyq wrote: > >> > > >> >> 前提 > >> >> 1.flink配置了高可用 > >> >> 2.flink配置checkpoint数为10 > >> >> 3.yarn集群配置了任务恢复 > >> >> 疑问 > >> >> yarn集群停电重启后，恢复flink任务时，如果最近的checkpoint由于停电导致块损坏，是否会尝试从其他checkpoint启动 > >> >> > >> >> > >> >> > >> >> > > > > > > > >-- > >Best, > >Yanfei >
Re: Re: Re: Re: flink on yarn 异常停电问题咨询
Hi 确认一下这些 ha 文件的 last modification time 是一致的还是错开的？ 另外，指定 chk- 恢复尝试了没有？可以恢复吗？ Best, Guojun On Fri, Mar 10, 2023 at 11:56 AM guanyq wrote: > flink ha路径为 /tmp/flink/ha/ > flink chk路径为 /tmp/flink/checkpoint > > > 我现在不确定是这个ha的文件损坏了，还是所有chk都损坏，但是这个需要模拟验证一下。 > > > > > 会尝试从10个chk恢复，日志有打印 > 2023-03-0718:37:43,703INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Recovering checkpoints from ZooKeeper. > 2023-03-0718:37:43,730INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Found 10 checkpoints in ZooKeeper. > 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to fetch 10 checkpoints from storage. > 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7079. > 2023-03-0718:37:43,837INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7080. > 2023-03-0718:37:43,868INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7081. > 2023-03-0718:37:43,896INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7082. > 2023-03-0718:37:43,906INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7083. > 2023-03-0718:37:43,928INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7084. > 2023-03-0718:37:43,936INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7085. > 2023-03-0718:37:43,947INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7086. > > > > 详细日志为，后面重复部分我给省略了，不同点就是尝试不同的/tmp/flink/ha/application_1678102326043_0007/completedCheckpointxx启动 > 2023-03-0718:37:43,621INFOorg.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl > - Starting the SlotManager. > 2023-03-0718:37:43,621INFOorg.apache.flink.runtime.jobmaster.JobMaster - > Successfully ran initialization on master in 0 ms. > 2023-03-0718:37:43,660INFOorg.apache.flink.runtime.util.ZooKeeperUtils - > Initialized ZooKeeperCompletedCheckpointStore in > '/checkpoints/3844b96b002601d932e66233dd46899c'. > 2023-03-0718:37:43,680INFOorg.apache.flink.runtime.jobmaster.JobMaster - > Using application-defined state backend: File State Backend (checkpoints: > 'hdfs:/tmp/flink/checkpoint', savepoints: 'null', asynchronous: UNDEFINED, > fileStateThreshold: -1) > 2023-03-0718:37:43,680INFOorg.apache.flink.runtime.jobmaster.JobMaster - > Configuring application-defined state backend with job/cluster config > 2023-03-0718:37:43,703INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Recovering checkpoints from ZooKeeper. > 2023-03-0718:37:43,730INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Found 10 checkpoints in ZooKeeper. > 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to fetch 10 checkpoints from storage. > 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7079. > 2023-03-0718:37:43,837INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7080. > 2023-03-0718:37:43,868INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7081. > 2023-03-0718:37:43,896INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7082. > 2023-03-0718:37:43,906INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7083. > 2023-03-0718:37:43,928INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7084. > 2023-03-0718:37:43,936INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7085. > 2023-03-0718:37:43,947INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore > - Trying to retrieve checkpoint 7086. > 2023-03-0718:37:43,979WARNorg.apache.hadoop.hdfs.BlockReaderFactory - I/O > error constructing remote block reader. > java.io.IOException: Got error, status message opReadBlock > BP-1003103929-192.168.200.11-1668473836936:blk_1301252639_227512278 > received exception > org.apache.hadoop.hdfs.server.datanode.CorruptMetaHeaderException: > The meta file length 0 is less than the expected length 7, for > OP_READ_BLOCK, self=/192.168.200.23:45534, remote=/192.168.200.21:9866, > for file > /tmp/flink/ha/application_1678102326043_0007/completedCheckpoint58755403e33a, > for pool BP-1003103929-192.168.200.11-1668473836936 block > 1301252639_227512278 > at > org.apache.hadoop.hdfs.protocol.datatransfer.DataTran
Re:Re: Re: Re: Re: flink on yarn 异常停电问题咨询
我昨天模拟下断电的情况 10个ha文件的日期是错开的5秒一个 chk-xxx也不是都损坏了，有的是可以启动的，这个我也试了 现在情况是 yarn集群停电重启首先会循环尝试从10个ha的文件中启动应用，ha文件记录的chk的相关原数据 1.如果ha文件都损坏了，即使chk没有损坏，flink应用也是拉不起来的 现在想的是让hdfs上存在至少1组个可用的的ha文件及其对应的chk 现在是5秒一个chk，保存了10个，也会出现损坏无法启动的问题 5秒*10 = 50秒，也想知道多长时间的存档才能保证存在一组没有损坏ha和chk呢。 在 2023-03-14 10:16:48，"Guojun Li" 写道： >Hi > >确认一下这些 ha 文件的 last modification time 是一致的还是错开的？ > >另外，指定 chk- 恢复尝试了没有？可以恢复吗？ > >Best, >Guojun > >On Fri, Mar 10, 2023 at 11:56 AM guanyq wrote: > >> flink ha路径为 /tmp/flink/ha/ >> flink chk路径为 /tmp/flink/checkpoint >> >> >> 我现在不确定是这个ha的文件损坏了，还是所有chk都损坏，但是这个需要模拟验证一下。 >> >> >> >> >> 会尝试从10个chk恢复，日志有打印 >> 2023-03-0718:37:43,703INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Recovering checkpoints from ZooKeeper. >> 2023-03-0718:37:43,730INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Found 10 checkpoints in ZooKeeper. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to fetch 10 checkpoints from storage. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7079. >> 2023-03-0718:37:43,837INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7080. >> 2023-03-0718:37:43,868INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7081. >> 2023-03-0718:37:43,896INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7082. >> 2023-03-0718:37:43,906INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7083. >> 2023-03-0718:37:43,928INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7084. >> 2023-03-0718:37:43,936INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7085. >> 2023-03-0718:37:43,947INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7086. >> >> >> >> 详细日志为，后面重复部分我给省略了，不同点就是尝试不同的/tmp/flink/ha/application_1678102326043_0007/completedCheckpointxx启动 >> 2023-03-0718:37:43,621INFOorg.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl >> - Starting the SlotManager. >> 2023-03-0718:37:43,621INFOorg.apache.flink.runtime.jobmaster.JobMaster - >> Successfully ran initialization on master in 0 ms. >> 2023-03-0718:37:43,660INFOorg.apache.flink.runtime.util.ZooKeeperUtils - >> Initialized ZooKeeperCompletedCheckpointStore in >> '/checkpoints/3844b96b002601d932e66233dd46899c'. >> 2023-03-0718:37:43,680INFOorg.apache.flink.runtime.jobmaster.JobMaster - >> Using application-defined state backend: File State Backend (checkpoints: >> 'hdfs:/tmp/flink/checkpoint', savepoints: 'null', asynchronous: UNDEFINED, >> fileStateThreshold: -1) >> 2023-03-0718:37:43,680INFOorg.apache.flink.runtime.jobmaster.JobMaster - >> Configuring application-defined state backend with job/cluster config >> 2023-03-0718:37:43,703INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Recovering checkpoints from ZooKeeper. >> 2023-03-0718:37:43,730INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Found 10 checkpoints in ZooKeeper. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to fetch 10 checkpoints from storage. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7079. >> 2023-03-0718:37:43,837INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7080. >> 2023-03-0718:37:43,868INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7081. >> 2023-03-0718:37:43,896INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7082. >> 2023-03-0718:37:43,906INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7083. >> 2023-03-0718:37:43,928INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7084. >> 2023-03-0718:37:43,936INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7085. >> 2023-03-0718:37:43,947INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7086. >> 2023-03-0718:37:43,979WARNorg.apache.hadoop.hdfs.BlockReaderFactory - I/O >> error constructing remote block reader. >> java.io.IOException: Got error, status message opReadBlock >> BP-1003103929-192.168.200.11-1668473836936:blk_1301252639_227512278 >> received exception >> org.apache.hadoop.hdfs.server
