[ https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17430316#comment-17430316 ]
liyuanzhao435 edited comment on HUDI-2576 at 10/19/21, 5:33 AM: ---------------------------------------------------------------- flink jobmanager deleted the file : *2021-10-19 12:47:34,606 INFO org.apache.hudi.common.util.CommitUtils [] - Creating metadata for null numWriteStats:1numReplaceFileIds:0* *2021-10-19 12:47:34,607 INFO org.apache.hudi.client.AbstractHoodieWriteClient [] - Committing 20211019124727 action deltacommit* *2021-10-19 12:47:34,615 INFO org.apache.hudi.table.HoodieTable [] - Removing duplicate data files created due to spark retries before committing. Paths=[aa/c6eff439-d4e0-4deb-af43-f6906ab71d2b-0_0-1-0_20211019124727.parquet]* *2021-10-19 12:47:34,617 INFO org.apache.hudi.table.HoodieTable [] - {color:#de350b}Deleting invalid data files{color}=[(hdfs://XXXX:9999/tmp/test_liyz2/aa,hdfs://XXXX:9999/tmp/test_liyz2/aa/c6eff439-d4e0-4deb-af43-f6906ab71d2b-0_0-1-0_20211019124727.parquet)]* *2021-10-19 12:47:34,676 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Marking instant complete [==>20211019124727__deltacommit__INFLIGHT]* was (Author: liyuanzhao435): flink jobmanager deleted the file : *2021-10-19 12:47:34,606 INFO org.apache.hudi.common.util.CommitUtils [] - Creating metadata for null numWriteStats:1numReplaceFileIds:0* *2021-10-19 12:47:34,607 INFO org.apache.hudi.client.AbstractHoodieWriteClient [] - Committing 20211019124727 action deltacommit* *2021-10-19 12:47:34,615 INFO org.apache.hudi.table.HoodieTable [] - Removing duplicate data files created due to spark retries before committing. Paths=[aa/c6eff439-d4e0-4deb-af43-f6906ab71d2b-0_0-1-0_20211019124727.parquet]* *2021-10-19 12:47:34,617 INFO org.apache.hudi.table.HoodieTable [] - {color:#de350b}Deleting invalid data files{color}=[(hdfs://XXXX:9999/tmp/test_liyz2/aa,hdfs://XXXX:9999/tmp/test_liyz2/aa/c6eff439-d4e0-4deb-af43-f6906ab71d2b-0_0-1-0_20211019124727.parquet)]* *2021-10-19 12:47:34,676 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Marking instant complete [==>20211019124727__deltacommit__INFLIGHT]* *2021-10-19 12:47:34,677 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Checking for file exists ?hdfs://26.6.4.165:8020/tmp/test_liyz2/.hoodie/20211019124727.deltacommit.inflight* *2021-10-19 12:47:34,691 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Create new file for toInstant ?hdfs://26.6.4.165:8020/tmp/test_liyz2/.hoodie/20211019124727.deltacommit* *2021-10-19 12:47:34,691 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Completed [==>20211019124727__deltacommit__INFLIGHT]* *20* > flink do checkpoint error because parquet file is missing > ---------------------------------------------------------- > > Key: HUDI-2576 > URL: https://issues.apache.org/jira/browse/HUDI-2576 > Project: Apache Hudi > Issue Type: Bug > Components: Flink Integration > Affects Versions: 0.10.0 > Reporter: liyuanzhao435 > Priority: Major > Labels: flink, hudi > Fix For: 0.10.0 > > Attachments: error.txt > > Original Estimate: 96h > Remaining Estimate: 96h > > hudi:0.10.0, flink 1.13.1 > some times when flink do checkpoint , error occurs, the error shows a hudi > parquet file is missing (says file not exists) : > *2021-10-19 09:20:03,796 INFO > org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle [] - start close > hoodie row data* > *2021-10-19 09:20:03,800 WARN org.apache.hadoop.hdfs.DataStreamer [] - > DataStreamer Exception* > *java.io.FileNotFoundException: File does not exist: > /tmp/test_liyz2/aa/2ff301cc-8db2-478e-b707-e8f2327ba38f-0_0-1-4_20211019091917.parquet > (inode 32234795) Holder DFSClient_NONMAPREDUCE_633610786_99 does not have > any open files.* > *at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2815)* > > detail see appendix -- This message was sent by Atlassian Jira (v8.3.4#803005)