Correcting the link to commit file On Mon, Jan 20, 2020 at 3:50 PM Venki g <venke...@gmail.com> wrote:
> Hi, > > I am using a spark job to upsert the incremental delta files from S3 into > Hudi storage using HoodieDeltaStreamer.sync() API , The incremental spark > job is failing with below exception > > java.lang.RuntimeException: > org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable to > find previous checkpoint. Please double check if this table was indeed > built via delta streamer > at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:36) > at com.emr.java.HudiDataLoadJob.run(HudiDataLoadJob.java:28) > at com.emr.java.HiveDeltaStreamer.main(HiveDeltaStreamer.java:19) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684) > Caused by: > org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable to > find previous checkpoint. Please double check if this table was indeed > built via delta streamer > at > org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:252) > at > org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:214) > at > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:120) > at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:30) > ... 7 more > > I found the recent commit file does not have > ""deltastreamer.checkpoint.key" in the commit file. I checked the second > last commit file and it has this key. > > Link to driver log(has delta streamer config passed and other info) - > https://pastebin.pl/view/raw/9606beb0 > > *Link to most recent commit - https://pastebin.pl/view/raw/defc32ae > <https://pastebin.pl/view/raw/defc32ae> * > > When this happened for the first time, I was able to rollback the latest > commit and loaded the data again and went past this exception. Since, this > exception has started occurring again, I would like to understand the issue > here and find the fix if any. > > Would highly appreciate any help on this. > > Thanks > Venkatesh >