Hi, I am using a spark job to upsert the incremental delta files from S3 into Hudi storage using HoodieDeltaStreamer.sync() API , The incremental spark job is failing with below exception
java.lang.RuntimeException: org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable to find previous checkpoint. Please double check if this table was indeed built via delta streamer at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:36) at com.emr.java.HudiDataLoadJob.run(HudiDataLoadJob.java:28) at com.emr.java.HiveDeltaStreamer.main(HiveDeltaStreamer.java:19) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684) Caused by: org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable to find previous checkpoint. Please double check if this table was indeed built via delta streamer at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:252) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:214) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:120) at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:30) ... 7 more I found the recent commit file does not have ""deltastreamer.checkpoint.key" in the commit file. I checked the second last commit file and it has this key. Link to driver log(has delta streamer config passed and other info) - https://pastebin.pl/view/raw/9606beb0 Link to most recent commit - https://pastebin.pl/view/raw/9606beb0 When this happened for the first time, I was able to rollback the latest commit and loaded the data again and went past this exception. Since, this exception has started occurring again, I would like to understand the issue here and find the fix if any. Would highly appreciate any help on this. Thanks Venkatesh
