[
https://issues.apache.org/jira/browse/HUDI-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359328#comment-17359328
]
Vinay commented on HUDI-1892:
-
Looking at the code this issue could be because the defaultValue is not empty
but the actual value of the field is null (This could be the case if field
schema is Union type like "null,string")
Creating a PR to use String.valueOf(value) in the below function to handle this
edge case
{code:java}
public Boolean overwriteField(Object value, Object defaultValue) {
return defaultValue == null ? value == null :
defaultValue.toString().equals(value.toString());
}
{code}
> NullPointerException when using OverwriteNonDefaultsWithLatestAvroPayload at
> hudi 0.9.0
> ---
>
> Key: HUDI-1892
> URL: https://issues.apache.org/jira/browse/HUDI-1892
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: shenbing
>Assignee: Vinay
>Priority: Major
>
> using compiled hudi 0.9.0 with hadoop3.0.0 and hive3.1.1 after resolving
> dependency conflicts,
> I import hudi-spark-bundle_2.11-0.9.0-SNAPSHOT.jar into my project.
> When I using OverwriteNonDefaultsWithLatestAvroPayload to update field with
> new value, I got the error.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hudi.exception.HoodieUpsertException: Failed to combine/merge new
> record with old value in storage, for new record {HoodieRecord{key=HoodieKey
> { recordKey=1 partitionPath=date=1}, currentLocation='HoodieRecordLocation
> {instantTime=20210510160355, fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}',
> newLocation='HoodieRecordLocation {instantTime=20210510160400,
> fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}'}}, old value
> {{"_hoodie_commit_time": "20210510160355", "_hoodie_commit_seqno":
> "20210510160355_0_50", "_hoodie_record_key": "1", "_hoodie_partition_path":
> "date=1", "_hoodie_file_name":
> "9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0_0-1502-1519_20210510160355.parquet",
> "uuid": "1", "name": "jerry", "age": 10, "date": "1", "update_time": "1"}}
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
> ... 34 more
> Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to
> combine/merge new record with old value in storage, for new record
> {HoodieRecord{key=HoodieKey { recordKey=1 partitionPath=date=1},
> currentLocation='HoodieRecordLocation {instantTime=20210510160355,
> fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}',
> newLocation='HoodieRecordLocation {instantTime=20210510160400,
> fileId=9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0}'}}, old value
> {{"_hoodie_commit_time": "20210510160355", "_hoodie_commit_seqno":
> "20210510160355_0_50", "_hoodie_record_key": "1", "_hoodie_partition_path":
> "date=1", "_hoodie_file_name":
> "9a0fcb8e-8cd9-4c9c-bea8-46bbf509035e-0_0-1502-1519_20210510160355.parquet",
> "uuid": "1", "name": "jerry", "age": 10, "date": "1", "update_time": "1"}}
> at
> org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:290)
> at
> org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
> at
> org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload.overwriteField(OverwriteWithLatestAvroPayload.java:97)
> at
> org.apache.hudi.common.model.OverwriteNonDefaultsWithLatestAvroPayload.lambda$combineAndGetUpdateValue$0(OverwriteNonDefaultsWithLatestAvroPayload.java:67)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at
> org.apache.hudi.common.model.OverwriteNonDefaultsWithLatestAvroPayload.combineAndGetUpdateValue(OverwriteNonDefaultsWithLatestAvroPayload.java:64)
> at
> org.apache.hudi.common.model.HoodieRecordPayload.combineAndGetUpdateValue(HoodieRecordPayload.java:81)
> at
> org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:276)
> ... 8 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
