[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leesf updated HUDI-389: --- Fix Version/s: (was: 0.5.2) 0.5.1 > Updates sent to diff partition for a given key with Global Index > - > > Key: HUDI-389 > URL: https://issues.apache.org/jira/browse/HUDI-389 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.1 > > Original Estimate: 48h > Time Spent: 20m > Remaining Estimate: 47h 40m > > Updates sent to diff partition for a given key with Global Index should > succeed by updating the record under original partition. As of now, it throws > exception. > [https://github.com/apache/incubator-hudi/issues/1021] > > > error log: > {code:java} > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants > java.util.stream.ReferencePipeline$Head@d02b1c7 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file > system view for partition (2016/04/15) > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found > in partition (2016/04/15) =0, Time taken =0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - > addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding > file-groups for partition :2016/04/15, #FileGroups=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load > partition (2016/04/15) =0 > 14754 [Executor task launch worker-0] ERROR > com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType > UPDATE for partition :0 > java.util.NoSuchElementException: No value present > at com.uber.hoodie.common.util.Option.get(Option.java:112) > at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > at > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) > at > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suneel Marthi updated HUDI-389: --- Fix Version/s: (was: 0.5.1) 0.5.2 > Updates sent to diff partition for a given key with Global Index > - > > Key: HUDI-389 > URL: https://issues.apache.org/jira/browse/HUDI-389 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.2 > > Original Estimate: 48h > Time Spent: 20m > Remaining Estimate: 47h 40m > > Updates sent to diff partition for a given key with Global Index should > succeed by updating the record under original partition. As of now, it throws > exception. > [https://github.com/apache/incubator-hudi/issues/1021] > > > error log: > {code:java} > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants > java.util.stream.ReferencePipeline$Head@d02b1c7 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file > system view for partition (2016/04/15) > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found > in partition (2016/04/15) =0, Time taken =0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - > addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding > file-groups for partition :2016/04/15, #FileGroups=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load > partition (2016/04/15) =0 > 14754 [Executor task launch worker-0] ERROR > com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType > UPDATE for partition :0 > java.util.NoSuchElementException: No value present > at com.uber.hoodie.common.util.Option.get(Option.java:112) > at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > at > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) > at > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-389: Status: Closed (was: Patch Available) > Updates sent to diff partition for a given key with Global Index > - > > Key: HUDI-389 > URL: https://issues.apache.org/jira/browse/HUDI-389 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.1 > > Original Estimate: 48h > Time Spent: 20m > Remaining Estimate: 47h 40m > > Updates sent to diff partition for a given key with Global Index should > succeed by updating the record under original partition. As of now, it throws > exception. > [https://github.com/apache/incubator-hudi/issues/1021] > > > error log: > {code:java} > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants > java.util.stream.ReferencePipeline$Head@d02b1c7 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file > system view for partition (2016/04/15) > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found > in partition (2016/04/15) =0, Time taken =0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - > addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding > file-groups for partition :2016/04/15, #FileGroups=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load > partition (2016/04/15) =0 > 14754 [Executor task launch worker-0] ERROR > com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType > UPDATE for partition :0 > java.util.NoSuchElementException: No value present > at com.uber.hoodie.common.util.Option.get(Option.java:112) > at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > at > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) > at > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-389: Status: Patch Available (was: In Progress) > Updates sent to diff partition for a given key with Global Index > - > > Key: HUDI-389 > URL: https://issues.apache.org/jira/browse/HUDI-389 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.1 > > Original Estimate: 48h > Time Spent: 10m > Remaining Estimate: 47h 50m > > Updates sent to diff partition for a given key with Global Index should > succeed by updating the record under original partition. As of now, it throws > exception. > [https://github.com/apache/incubator-hudi/issues/1021] > > > error log: > {code:java} > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants > java.util.stream.ReferencePipeline$Head@d02b1c7 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file > system view for partition (2016/04/15) > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found > in partition (2016/04/15) =0, Time taken =0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - > addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding > file-groups for partition :2016/04/15, #FileGroups=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load > partition (2016/04/15) =0 > 14754 [Executor task launch worker-0] ERROR > com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType > UPDATE for partition :0 > java.util.NoSuchElementException: No value present > at com.uber.hoodie.common.util.Option.get(Option.java:112) > at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > at > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) > at > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-389: Status: In Progress (was: Open) > Updates sent to diff partition for a given key with Global Index > - > > Key: HUDI-389 > URL: https://issues.apache.org/jira/browse/HUDI-389 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.1 > > Original Estimate: 48h > Time Spent: 10m > Remaining Estimate: 47h 50m > > Updates sent to diff partition for a given key with Global Index should > succeed by updating the record under original partition. As of now, it throws > exception. > [https://github.com/apache/incubator-hudi/issues/1021] > > > error log: > {code:java} > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants > java.util.stream.ReferencePipeline$Head@d02b1c7 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file > system view for partition (2016/04/15) > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found > in partition (2016/04/15) =0, Time taken =0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - > addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding > file-groups for partition :2016/04/15, #FileGroups=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load > partition (2016/04/15) =0 > 14754 [Executor task launch worker-0] ERROR > com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType > UPDATE for partition :0 > java.util.NoSuchElementException: No value present > at com.uber.hoodie.common.util.Option.get(Option.java:112) > at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > at > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) > at > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-389: Labels: pull-request-available (was: ) > Updates sent to diff partition for a given key with Global Index > - > > Key: HUDI-389 > URL: https://issues.apache.org/jira/browse/HUDI-389 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.1 > > Original Estimate: 48h > Remaining Estimate: 48h > > Updates sent to diff partition for a given key with Global Index should > succeed by updating the record under original partition. As of now, it throws > exception. > [https://github.com/apache/incubator-hudi/issues/1021] > > > error log: > {code:java} > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants > java.util.stream.ReferencePipeline$Head@d02b1c7 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file > system view for partition (2016/04/15) > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found > in partition (2016/04/15) =0, Time taken =0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - > addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding > file-groups for partition :2016/04/15, #FileGroups=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load > partition (2016/04/15) =0 > 14754 [Executor task launch worker-0] ERROR > com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType > UPDATE for partition :0 > java.util.NoSuchElementException: No value present > at com.uber.hoodie.common.util.Option.get(Option.java:112) > at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > at > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) > at > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) >
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-389: Description: Updates sent to diff partition for a given key with Global Index should succeed by updating the record under original partition. As of now, it throws exception. [https://github.com/apache/incubator-hudi/issues/1021] error log: {code:java} 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants java.util.stream.ReferencePipeline$Head@d02b1c7 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file system view for partition (2016/04/15) 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found in partition (2016/04/15) =0, Time taken =0 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding file-groups for partition :2016/04/15, #FileGroups=0 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load partition (2016/04/15) =0 14754 [Executor task launch worker-0] ERROR com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType UPDATE for partition :0 java.util.NoSuchElementException: No value present at com.uber.hoodie.common.util.Option.get(Option.java:112) at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) at com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) at com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) at com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) at com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {code} Refer [here|[https://github.com/apache/incubator-hudi/issues/1021]] for some context. was: Updates sent to diff partition for a given key with Global Index should succeed by updating the record under original partition. As of now, it throws exception. error log: 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants java.util.stream.ReferencePipeline$Head@d02b1c7 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file system view for partition (2016/04/15) 14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found in partition (2016/04/15)
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[ https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-389: Fix Version/s: 0.5.1 > Updates sent to diff partition for a given key with Global Index > - > > Key: HUDI-389 > URL: https://issues.apache.org/jira/browse/HUDI-389 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Index >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Major > Fix For: 0.5.1 > > Original Estimate: 48h > Remaining Estimate: 48h > > Updates sent to diff partition for a given key with Global Index should > succeed by updating the record under original partition. As of now, it throws > exception. > [https://github.com/apache/incubator-hudi/issues/1021] > > > error log: > {code:java} > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants > java.util.stream.ReferencePipeline$Head@d02b1c7 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file > system view for partition (2016/04/15) > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found > in partition (2016/04/15) =0, Time taken =0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - > addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding > file-groups for partition :2016/04/15, #FileGroups=0 > 14738 [Executor task launch worker-0] INFO > com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load > partition (2016/04/15) =0 > 14754 [Executor task launch worker-0] ERROR > com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType > UPDATE for partition :0 > java.util.NoSuchElementException: No value present > at com.uber.hoodie.common.util.Option.get(Option.java:112) > at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180) > at > com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263) > at > com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336) > at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973) > at > org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) > at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) > at > org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Refer [here|[https://github.com/apache/incubator-hudi/i