OverwriteWithLatestAvroPayload is used for Delta Streamer. Is there a way for DataSource Writer?
please correct me , if I am wrong. Thanks, Selva On Mon, Feb 24, 2020 at 1:15 PM Gary Li <yanjia.gary...@gmail.com> wrote: > Hi, in this case you need to design your own logic to handle merging. > Please check OverwriteWithLatestAvroPlayload class. You can write your own > one and pass it as DataSourceWriteOptions.PAYLOAD_CLASS_OPT_KEY to Hudi. > > On Mon, Feb 24, 2020 at 12:25 PM selvaraj periyasamy < > selvaraj.periyasamy1...@gmail.com> wrote: > > > Hi, I am experimenting Hudi 0.5.0 version for some of the update use > cases. > > > > Our flow is as below > > > > RDBMS -> CDC Log -> Hive -> COW table. > > > > CCDC log for update would have the value only for the primary key > columns + > > updated columns. Remaining column values are null. While upserting values > > on CCOW table, would need to update only the column values , which are > > updated and retain the values for other columns. When I tested, Hudi > > updates remaining column values as null since log has null values. > > > > Is there a way to merge rows for columns which are having values during > > update? > > > > Thanks, > > Selva > > >
