Hi Shahida, It seems like a bug from the rename changes we did recently. Could you please raise a JIRA, tagged to 0.5.1 release? Should be easy to fix, but DeltaStreamer should not be reading the clean timeline.. (Also btw this is also related to the one issue that you filed).
Balaji, could you take a look at this when you get a chance and see if this is related. Thanks Vinoth On Fri, Dec 27, 2019 at 5:38 AM lamberken <lamber...@163.com> wrote: > > > Hi @Shahida Khan, > > > In the past few days, I faced similar issue. This bug seems happened after > HUDI-398 merged. > You can try to build source before that commit, then continue your work. > > > Here are the details: > > https://lists.apache.org/thread.html/f7834b3389e67b2b66b65386f59eb6646942206865133300c0416a6a%40%3Cdev.hudi.apache.org%3E > > > best, > lamber-ken > On 12/27/2019 21:02，Shahida Khan<srkhan2...@gmail.com> wrote： > @lamberken, when I have checked, folder .aux was empty ... > :( > > On Fri, 27 Dec 2019 at 6:28 PM, lamberken <lamber...@163.com> wrote: > > > > Hi @Shahida Khan, > > > I have a question that the size of *.clean.requested files is 0 ? > > > best, > lamber-ken > > > > > On 12/27/2019 19:54，Shahida Khan<srkhan2...@gmail.com> wrote： > Hi, > > Greetings!! > I have currently using Delta Streamer and upserting data via hudi in > real-time. > Have used the latest master branch. > Job was running fine from last 10days, suddenly, most of the streaming job > started failing and below is the error which I am facing : > > > > > > > > > > > > > > > > > > *java.util.concurrent.ExecutionException: > org.apache.hudi.exception.HoodieException: Could not read commit > details from > > hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested > at > > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at > > org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService.waitForShutdown(AbstractDeltaStreamerService.java:72) > at > > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:117) > at > > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:297) > at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at > java.lang.reflect.Method.invoke(Method.java:498) at > > > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:688)Caused > by: org.apache.hudi.exception.HoodieException: Could not read commit > details from > > hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested > at > > org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:411) > at > > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > java.lang.Thread.run(Thread.java:748)* > > > It seems issue has already been raised : hudi-1128 > <https://github.com/apache/incubator-hudi/pull/1128/files> > Is this issue related to same which i am facing..?? > > > *Regards,* > *Shahida R. Khan* > > -- > Regards, > Shahida Rashid Khan > 9167538366 > > > > > ****kindly ignore typo error **** Sent from handheld device ...***** >
