deepakpanda93 commented on issue #11348: URL: https://github.com/apache/hudi/issues/11348#issuecomment-4850487179
This is addressed in Hudi 1.x. **Root cause:** on 0.14.x, the Hudi Streamer's --payload-class argument defaulted to OverwriteWithLatestAvroPayload, and that CLI default took precedence over hoodie.datasource.write.payload.class=PartialUpdateAvroPayload in your hudi.properties. So the Streamer never used PartialUpdateAvroPayload, and the incoming null for column1 overwrote the existing value — which is why it behaved differently from Spark structured streaming (which read the payload class from the properties directly). - Workaround on 0.14.x: pass --payload-class org.apache.hudi.common.model.PartialUpdateAvroPayload explicitly on the spark-submit command (not just in properties). - Fix in 1.x: the Streamer's --payload-class default was changed to null/deprecated (HUDI-8203, [#11943](https://github.com/apache/hudi/pull/11943), 1.0.0+), so it no longer overrides the properties value. And partial update is now natively supported via record merge mode — HUDI-8835 ([#13263](https://github.com/apache/hudi/pull/13263)) and HUDI-8401 ([#13498](https://github.com/apache/hudi/pull/13498)) in 1.1.0 (payload classes are deprecated in favor of merge mode, HUDI-9713). On a recent 1.1.x release, partial update with the Streamer works — either keep PartialUpdateAvroPayload (now honored from properties) or migrate to hoodie.record.merge.mode for partial updates. Closing as fixed in 1.x; please reopen if you still see full-overwrite behavior on 1.1.x. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
