cshuo commented on code in PR #19858:
URL: https://github.com/apache/hudi/pull/19858#discussion_r3953913998
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/RecordContext.java:
##########
@@ -440,11 +441,8 @@ private SerializableBiFunction<T, HoodieSchema, String>
metadataKeyExtractor() {
return (record, schema) -> getValue(record, schema,
RECORD_KEY_METADATA_FIELD).toString();
}
- private SerializableBiFunction<T, HoodieSchema, String>
virtualKeyExtractor(String[] recordKeyFields) {
- if (recordKeyFields.length == 1) {
- // there might be consistency for record key encoding when partition
fields are multiple for cow merging,
- // currently the incoming records are using the keys from HoodieRecord
which utilities the write config and by default encodes the field name with the
value
- // while here the field names are ignored, this function would be used
to extract record keys from old base file.
+ private SerializableBiFunction<T, HoodieSchema, String>
virtualKeyExtractor(String[] recordKeyFields, int numPartitionFields) {
Review Comment:
This fixes the default-encoding mismatch, but with
`hoodie.write.complex.keygen.new.encoding=true`, the writer still generates
`k1`, while this extractor now returns `id:k1`. That causes missed
updates/deletes.
Could we account for the writer’s encoding rules and add coverage for this
case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
