Hi Susu, thanks for the response. Can you please explain whats the impact of deleting these commit files?
Thanks! On Wed, Jun 23, 2021 at 8:09 AM Susu Dong <susudo...@gmail.com> wrote: > Hi Aakash, > > I believe there were schema level changes from Hudi 0.5.0 to 0.6.0 > regarding those commit files. So if you are jumping from 0.5.0 to 0.8.0 > right away, you will likely experience such an error, i.e. Failed to > archive commits. You shouldn't need to delete archived files; instead, you > should try deleting some, if not all, active commit files under your > *.hoodie* folder. The reason for that is 0.8.0 is using a new AVRO schema > to parse your old commit files, so you got the failure. Can you try the > above approach and let us know? Thank you. :) > > Best, > Susu > > On Wed, Jun 23, 2021 at 12:21 PM aakash aakash <email2aak...@gmail.com> > wrote: > > > Hi, > > > > I am trying to use Hudi 0.8 with Spark 3.0 in my prod environment and > > earlier we were running Hudi 0.5 with Spark 2.4.4. > > > > While updating a very old index, I am getting this error : > > > > *from the logs it seem its error out while reading this file : > > hudi/.hoodie/archived/.commits_.archive.119_1-0-1 in s3* > > > > 21/06/22 19:18:06 ERROR HoodieTimelineArchiveLog: Failed to archive > > commits, .commit file: 20200715192915.rollback.inflight > > java.io.IOException: Not an Avro data file > > at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:50) > > at > > > > > org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:175) > > at > > > > > org.apache.hudi.client.utils.MetadataConversionUtils.createMetaWrapper(MetadataConversionUtils.java:84) > > at > > > > > org.apache.hudi.table.HoodieTimelineArchiveLog.convertToAvroRecord(HoodieTimelineArchiveLog.java:370) > > at > > > > > org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:311) > > at > > > > > org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:128) > > at > > > > > org.apache.hudi.client.AbstractHoodieWriteClient.postCommit(AbstractHoodieWriteClient.java:430) > > at > > > > > org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:186) > > at > > > > > org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:121) > > at > > > > > org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:479) > > > > > > Is this a backward compatibility issue? I have deleted a few archive > files > > but the problem is persisting so it does not look like a file corruption > > issue. > > > > Regards, > > Aakash > > >
