ad1happy2go commented on issue #9469: URL: https://github.com/apache/hudi/issues/9469#issuecomment-1686194536
@praneethh I tried the similar stuff using spark-sql. It gave me two records only. below if the code I used - ``` CREATE TABLE issue_9469_23 USING HUDI PARTITIONED BY(log_dt) tblproperties ( "hoodie.payload.ordering.field"="load_ts", "hoodie.datasource.write.recordkey.field"="emp_id", "hoodie.datasource.write.partitionpath.field"="log_dt", "hoodie.datasource.write.precombine.field"="load_ts", "hoodie.index.type"="GLOBAL_SIMPLE", "hoodie.simple.index.update.partition.path"="true", "hoodie.datasource.write.payload.class"="org.apache.hudi.common.model.PartialUpdateAvroPayload", "hoodie.datasource.write.reconcile.schema"="true", "hoodie.schema.on.read.enable"="true", "hoodie.datasource.write.hive_style_partitioning"="true", "hoodie.datasource.write.row.writer.enable"="false" ) AS SELECT 1 as emp_id, 'neo' as emp_name, cast('2023-08-04 12:00:00' as timestamp) as log_ts, cast('2023-08-04 12:00:00' as timestamp) as load_ts, cast('2023-08-04' as date) as log_dt; merge into issue_9469_23 as target using ( SELECT 1 as emp_id, 'neo_1' as emp_name, cast('2023-08-05 12:00:00' as timestamp) as log_ts, cast('2023-08-05 12:00:00' as timestamp) as load_ts, cast('2023-08-05' as date) as log_dt UNION SELECT 2 as emp_id, 'trinity' as emp_name, cast('2023-08-05 14:00:00' as timestamp) as log_ts, cast('2023-08-05 15:00:00' as timestamp) as load_ts, cast('2023-08-05' as date) as log_dt ) source on target.emp_id = source.emp_id when matched then update set target.log_ts = source.load_ts, target.log_ts = source.log_dt, target.load_ts = source.load_ts when not matched then insert *; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org