Hi Venki, Thanks for reporting this. The latest commit file seems to be empty? I am wondering if this is happening because there was no new data to process and the tool wrote an empty commit file.. Can you confirm if this seems to match the case?
Thanks Vinoth On Mon, Jan 20, 2020 at 4:00 PM Venki g <venke...@gmail.com> wrote: > Correcting the link to commit file > > On Mon, Jan 20, 2020 at 3:50 PM Venki g <venke...@gmail.com> wrote: > > > Hi, > > > > I am using a spark job to upsert the incremental delta files from S3 into > > Hudi storage using HoodieDeltaStreamer.sync() API , The incremental spark > > job is failing with below exception > > > > java.lang.RuntimeException: > > org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable > to > > find previous checkpoint. Please double check if this table was indeed > > built via delta streamer > > at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:36) > > at com.emr.java.HudiDataLoadJob.run(HudiDataLoadJob.java:28) > > at com.emr.java.HiveDeltaStreamer.main(HiveDeltaStreamer.java:19) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684) > > Caused by: > > org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable > to > > find previous checkpoint. Please double check if this table was indeed > > built via delta streamer > > at > > > org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:252) > > at > > > org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:214) > > at > > > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:120) > > at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:30) > > ... 7 more > > > > I found the recent commit file does not have > > ""deltastreamer.checkpoint.key" in the commit file. I checked the second > > last commit file and it has this key. > > > > Link to driver log(has delta streamer config passed and other info) - > > https://pastebin.pl/view/raw/9606beb0 > > > > *Link to most recent commit - https://pastebin.pl/view/raw/defc32ae > > <https://pastebin.pl/view/raw/defc32ae> * > > > > When this happened for the first time, I was able to rollback the latest > > commit and loaded the data again and went past this exception. Since, > this > > exception has started occurring again, I would like to understand the > issue > > here and find the fix if any. > > > > Would highly appreciate any help on this. > > > > Thanks > > Venkatesh > > >
