[GitHub] [hudi] tandonraghav commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
tandonraghav commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-870831328 @fanaticjo ok, but you should implement `preCombine` also, as the logic should be uniform, might be that is the reason? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tandonraghav commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
tandonraghav commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-870826823 @fanaticjo As per the above comments, you should also implement `preCombine` method. And also do u faced the same problem while Schema Evolution? or something else? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tandonraghav commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
tandonraghav commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-869211898 @guanziyue Well, in my testing combinAndGetUpdateValue in not getting called at all. I am not worried about how compaction is happening for delta records. Do you have a reference implementation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tandonraghav commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
tandonraghav commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-866269573 @n3nash I am trying to implement exactly `https://github.com/apache/hudi/pull/2106/files` , in this also `PartialAvroPayload` class only implements `combineAndGetUpdateValue` . Also, i do not have `Schema` in preCombine method. Also, i am saying if the record preexists in (.parquet) and if i add the field in incoming records (log file .avro files) then on compaction `combineAndGetUpdateValue` is not at all getting called but this doesnt happen if the schema is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tandonraghav commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
tandonraghav commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-864611953 @n3nash I am not able to resolve this. `preCombine` can be called even if there are no duplicates in the incoming records? Can you please explain with a sample ? I fail to understand how will i combine the existing record if it doesn't call the method `combineAndGetUpdateValue` as nowhere else i have ref to `IndexedRecord currentValue`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] tandonraghav commented on issue #3078: [SUPPORT] combineAndGetUpdateValue is not getting called when Schema evolution happens
tandonraghav commented on issue #3078: URL: https://github.com/apache/hudi/issues/3078#issuecomment-862168089 @n3nash I thought `preCombine` is only used to dedup incoming records. Then why `OverwriteNonDefaultsWithLatestAvroPayload` is not overriding `preCombine`? As well as in this [PR](https://github.com/apache/hudi/pull/3035/files) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
