wombatu-kun commented on code in PR #20544:
URL: https://github.com/apache/hudi/pull/20544#discussion_r4511301420


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeModeEventTimeOrdering.scala:
##########
@@ -31,6 +32,34 @@ import 
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase
 import 
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.validateTableConfig
 
 class TestMergeModeEventTimeOrdering extends HoodieSparkSqlTestBase {
+  Seq(
+    ("date", "date'2026-10-01'", "date'2026-10-02'", false),
+    ("timestamp", "timestamp'2026-10-01 00:00:00'", "timestamp'2026-10-02 
00:00:00'", false),
+    ("timestamp", "timestamp'2026-10-01 00:00:00'", "timestamp'2026-10-02 
00:00:00'", true),
+    ("decimal(10,2)", "cast(1.50 as decimal(10,2))", "cast(2.50 as 
decimal(10,2))", false)
+  ).foreach { case (orderingType, oldValue, newValue, consistentTimestamp) =>
+    test(s"COW SQL upsert with $orderingType ordering field 
(consistentTimestamp=$consistentTimestamp)") {
+      withSQLConf(SPARK_SQL_INSERT_INTO_OPERATION.key -> "upsert",
+        
"hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled" -> 
consistentTimestamp.toString) {
+        withRecordType(Seq(HoodieRecordType.AVRO)) {
+          withTempDir { tmp =>
+            val tableName = generateTableName
+            spark.sql(
+              s"""
+                 | create table $tableName (id int, v int, ts $orderingType)
+                 | using hudi
+                 | tblproperties (type = 'cow', primaryKey = 'id', 
preCombineField = 'ts')
+                 | location '${tmp.getCanonicalPath}'
+               """.stripMargin)
+            spark.sql(s"insert into $tableName values (1, 1, $oldValue)")
+            spark.sql(s"insert into $tableName values (1, 2, $newValue)")
+            checkAnswer(s"select id, v from $tableName")(Seq(1, 2))

Review Comment:
   Both upserts move the ordering value forward, which commit-time ordering 
would also keep, so this only proves the merge no longer throws. Could each 
case add a third upsert with `oldValue` and assert `v` stays 2, so the 
logical-type comparison itself is covered?



##########
hudi-common/src/main/java/org/apache/hudi/common/avro/AvroRecordContext.java:
##########
@@ -107,6 +118,36 @@ public Object getValue(IndexedRecord record, HoodieSchema 
schema, String fieldNa
     return getFieldValueFromIndexedRecord(record, fieldName);
   }
 
+  @Override
+  public Comparable getOrderingValue(IndexedRecord record, HoodieSchema 
schema, List<String> orderingFieldNames) {

Review Comment:
   Inline delete blocks written from `HoodieFlinkRecord` or `HoodieSparkRecord` 
carry a DATE ordering value as `Integer` (`LocalDate` is not in the 
`HoodieDeleteRecordList` union), and `AvroRecordContext` passes it through 
unchanged, so with this override an Avro-context merge now throws in 
`shouldKeepNewerRecord` comparing it to `LocalDate`, where both sides used to 
be `Integer`. Should delete ordering values be converted per field here, as 
Spark and Flink do with `OrderingValueEngineTypeConverter`, or is that a 
follow-up?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to