Hi Wei Li, Left some detailed comments on the JIRA https://issues.apache.org/jira/browse/HUDI-944?focusedCommentId=17119668&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17119668
This is a great discussion.. thanks for getting this kicked off Thanks Vinoth On Mon, May 25, 2020 at 8:10 AM wei li <lw309637...@gmail.com> wrote: > Now hudi just support write、compaction concurrency control. But some > scenario need write concurrency control.Such as two spark job with > different data source ,need to write to the same hudi table. > > I have two Proposal： > > 1. first step :support write concurrency control on different partition > but now when two client write data to different partition, will meet these > error > > a、Rolling back commits failed > > b、instants version already exist > [2020-05-25 21:20:34,732] INFO Checking for file exists > ?/tmp/HudiDLATestPartition/.hoodie/20200525212031.clean.inflight > (org.apache.hudi.common.table.timeline.HoodieActiveTimeline) > Exception in thread "main" org.apache.hudi.exception.HoodieIOException: > Failed to create file > /tmp/HudiDLATestPartition/.hoodie/20200525212031.clean > at > > org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:437) > at > > org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:327) > at > > org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionCleanInflightToComplete(HoodieActiveTimeline.java:290) > at > > org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:183) > at > > org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:142) > at > > org.apache.hudi.client.HoodieCleanClient.lambda$clean$0(HoodieCleanClient.java:88) > at > > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > > c、two client's archiving conflict > > d、the read client meets "Unable to infer schema for Parquet. It must be > specified manually.;" > > 2. second step: support insert、upsert、compaction concurrency control on > different isolation level such as Serializable、WriteSerializable. > > hudi can design a mechanism to check the confict in > AbstractHoodieWriteClient.commit() > > I created a issue https://issues.apache.org/jira/browse/HUDI-944 > > Best Regards, > Wei Li. >
