[
https://issues.apache.org/jira/browse/HUDI-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8936:
------------------------------
Description:
After fixing HUDI-8850 on existing tests,
[https://github.com/apache/hudi/pull/12725] introduces more test cases on
COMMIT_TIME_ORDERING, and some of them failed because MERGE INTO delete does
not take effect.
Test dimensions:
All failures from "Test merge operations .." are due to the validation failure
below
!Screenshot 2025-01-29 at 12.31.01.png|width=1411,height=311!
Sample error message:
{code:java}
Expected Array([3,C,30.0,100], [4,D2,45.0,101], [5,E2,55.0,99],
[6,F2,65.0,100]), but got Array([0,X,20.0,100], [1,A,10.0,100], [2,B,20.0,100],
[3,C,30.0,100], [4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100])
ScalaTestFailureLocation:
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase at
(HoodieSparkSqlTestBase.scala:130)
org.scalatest.exceptions.TestFailedException: Expected Array([3,C,30.0,100],
[4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100]), but got
Array([0,X,20.0,100], [1,A,10.0,100], [2,B,20.0,100], [3,C,30.0,100],
[4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100])
at
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at
org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562)
at org.scalatest.Assertions.assertResult(Assertions.scala:867)
at org.scalatest.Assertions.assertResult$(Assertions.scala:863)
at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1562)
at
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.checkAnswer(HoodieSparkSqlTestBase.scala:130)
at
org.apache.spark.sql.hudi.dml.TestMergeModeCommitTimeOrdering.$anonfun$new$9(TestMergeModeCommitTimeOrdering.scala:309)
at
org.apache.spark.sql.hudi.dml.TestMergeModeCommitTimeOrdering.$anonfun$new$9$adapted(TestMergeModeCommitTimeOrdering.scala:239)
{code}
was:
After fixing HUDI-8850 on existing tests,
[https://github.com/apache/hudi/pull/12725] introduces more test cases on
COMMIT_TIME_ORDERING, and some of them failed because MERGE INTO delete does
not take effect.
Test dimensions:
Sample error message:
{code:java}
Expected Array([3,C,30.0,100], [4,D2,45.0,101], [5,E2,55.0,99],
[6,F2,65.0,100]), but got Array([0,X,20.0,100], [1,A,10.0,100], [2,B,20.0,100],
[3,C,30.0,100], [4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100])
ScalaTestFailureLocation:
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase at
(HoodieSparkSqlTestBase.scala:130)
org.scalatest.exceptions.TestFailedException: Expected Array([3,C,30.0,100],
[4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100]), but got
Array([0,X,20.0,100], [1,A,10.0,100], [2,B,20.0,100], [3,C,30.0,100],
[4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100])
at
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at
org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562)
at org.scalatest.Assertions.assertResult(Assertions.scala:867)
at org.scalatest.Assertions.assertResult$(Assertions.scala:863)
at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1562)
at
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.checkAnswer(HoodieSparkSqlTestBase.scala:130)
at
org.apache.spark.sql.hudi.dml.TestMergeModeCommitTimeOrdering.$anonfun$new$9(TestMergeModeCommitTimeOrdering.scala:309)
at
org.apache.spark.sql.hudi.dml.TestMergeModeCommitTimeOrdering.$anonfun$new$9$adapted(TestMergeModeCommitTimeOrdering.scala:239)
{code}
> MERGE INTO delete does not work in some cases of COMMIT_TIME_ORDERING
> ---------------------------------------------------------------------
>
> Key: HUDI-8936
> URL: https://issues.apache.org/jira/browse/HUDI-8936
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Y Ethan Guo
> Assignee: Davis Zhang
> Priority: Blocker
> Fix For: 1.0.1
>
> Attachments: Screenshot 2025-01-29 at 12.31.01.png
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> After fixing HUDI-8850 on existing tests,
> [https://github.com/apache/hudi/pull/12725] introduces more test cases on
> COMMIT_TIME_ORDERING, and some of them failed because MERGE INTO delete does
> not take effect.
> Test dimensions:
> All failures from "Test merge operations .." are due to the validation
> failure below
> !Screenshot 2025-01-29 at 12.31.01.png|width=1411,height=311!
> Sample error message:
> {code:java}
> Expected Array([3,C,30.0,100], [4,D2,45.0,101], [5,E2,55.0,99],
> [6,F2,65.0,100]), but got Array([0,X,20.0,100], [1,A,10.0,100],
> [2,B,20.0,100], [3,C,30.0,100], [4,D2,45.0,101], [5,E2,55.0,99],
> [6,F2,65.0,100])
> ScalaTestFailureLocation:
> org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase at
> (HoodieSparkSqlTestBase.scala:130)
> org.scalatest.exceptions.TestFailedException: Expected Array([3,C,30.0,100],
> [4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100]), but got
> Array([0,X,20.0,100], [1,A,10.0,100], [2,B,20.0,100], [3,C,30.0,100],
> [4,D2,45.0,101], [5,E2,55.0,99], [6,F2,65.0,100])
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
> at
> org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562)
> at org.scalatest.Assertions.assertResult(Assertions.scala:867)
> at org.scalatest.Assertions.assertResult$(Assertions.scala:863)
> at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1562)
> at
> org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.checkAnswer(HoodieSparkSqlTestBase.scala:130)
> at
> org.apache.spark.sql.hudi.dml.TestMergeModeCommitTimeOrdering.$anonfun$new$9(TestMergeModeCommitTimeOrdering.scala:309)
> at
> org.apache.spark.sql.hudi.dml.TestMergeModeCommitTimeOrdering.$anonfun$new$9$adapted(TestMergeModeCommitTimeOrdering.scala:239)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
