Correcting the link to commit file

On Mon, Jan 20, 2020 at 3:50 PM Venki g <venke...@gmail.com> wrote:

> Hi,
>
> I am using a spark job to upsert the incremental delta files from S3 into
> Hudi storage using HoodieDeltaStreamer.sync() API , The incremental spark
> job is failing with below exception
>
> java.lang.RuntimeException:
> org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable to
> find previous checkpoint. Please double check if this table was indeed
> built via delta streamer
> at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:36)
> at com.emr.java.HudiDataLoadJob.run(HudiDataLoadJob.java:28)
> at com.emr.java.HiveDeltaStreamer.main(HiveDeltaStreamer.java:19)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
> Caused by:
> org.apache.hudi.utilities.exception.HoodieDeltaStreamerException: Unable to
> find previous checkpoint. Please double check if this table was indeed
> built via delta streamer
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:252)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:214)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:120)
> at com.emr.java.HiveDeltaStreamer.loadData(HiveDeltaStreamer.java:30)
> ... 7 more
>
> I found the recent commit file does not have
> ""deltastreamer.checkpoint.key" in the commit file. I checked the second
> last commit file and it has this key.
>
> Link to driver log(has delta streamer config passed and other info) -
> https://pastebin.pl/view/raw/9606beb0
>
> *Link to most recent commit - https://pastebin.pl/view/raw/defc32ae
> <https://pastebin.pl/view/raw/defc32ae> *
>
> When this happened for the first time, I was able to rollback the latest
> commit and loaded the data again and went past this exception. Since, this
> exception has started occurring again, I would like to understand the issue
> here and find the fix if any.
>
> Would highly appreciate any help on this.
>
> Thanks
> Venkatesh
>

Reply via email to