hudi-bot opened a new issue, #16513:
URL: https://github.com/apache/hudi/issues/16513
Found two tests that don't fail but throw an exception while running. The
test succeeds but the spark context is not stopped in time due to exception.
This causes issue for other tests. For example, `test("bucket index query")` in
`TestDataSkippingQuery` succeeds but when we check the
[logs|https://github.com/apache/hudi/actions/runs/9391927778/job/25865161535#step:6:5799]
of the test we will find an error as below
{code:java}
74954 [ScalaTest-run-running-TestDataSkippingQuery] ERROR
org.apache.hudi.HoodieFileIndex [] - Failed to lookup candidate files in File
Index
java.lang.IllegalArgumentException: Property
_hoodie.record.key.gen.partition.id not found
at
org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:69)
~[classes/:?]
at
org.apache.hudi.common.config.TypedProperties.getInteger(TypedProperties.java:94)
~[classes/:?]
at
org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.generateSequenceId(AutoRecordGenWrapperKeyGenerator.java:115)
~[classes/:?]
at
org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.getRecordKey(AutoRecordGenWrapperKeyGenerator.java:67)
~[classes/:?]
at
org.apache.hudi.keygen.BaseKeyGenerator.getKey(BaseKeyGenerator.java:70)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.getBucketNumber$1(BucketIndexSupport.scala:154)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.getBucketSetFromValue$1(BucketIndexSupport.scala:168)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.getBucketsBySingleHashFields(BucketIndexSupport.scala:174)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.filterQueriesWithBucketHashField(BucketIndexSupport.scala:107)
~[classes/:?]
at
org.apache.hudi.BucketIndexSupport.computeCandidateFileNames(BucketIndexSupport.scala:78)
~[classes/:?]
at
org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3(HoodieFileIndex.scala:354)
~[classes/:?]
at
org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3$adapted(HoodieFileIndex.scala:351)
~[classes/:?]
at
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
~[scala-library-2.12.10.jar:?]
at scala.collection.immutable.List.foreach(List.scala:392)
~[scala-library-2.12.10.jar:?]
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
~[scala-library-2.12.10.jar:?]
at
org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$1(HoodieFileIndex.scala:351)
~[classes/:?]
at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.10.jar:?]
at
org.apache.hudi.HoodieFileIndex.lookupCandidateFilesInMetadataTable(HoodieFileIndex.scala:338)
~[classes/:?]
at
org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:241)
~[classes/:?]
at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:160)
~[classes/:?] {code}
The test succeeds because HoodieFileIndex listFiles is under try-catch and
falls back to no pruning (except for partition pruning). So, anyway due to the
exception spark context runs into conflict with another test which throws
exception as below:
{code:java}
org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
org.scalatest.Transformer.apply(Transformer.scala:22)
org.scalatest.Transformer.apply(Transformer.scala:20)
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
The currently active SparkContext was created at:
org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.$anonfun$test$1(HoodieSparkSqlTestBase.scala:85)
org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
org.scalatest.Transformer.apply(Transformer.scala:22)
org.scalatest.Transformer.apply(Transformer.scala:20)
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:199)
org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:181)
org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1562)
{code}
Another example is `test("Test Functional Index With Hive Sync Non
Partitioned Table")` in `TestFunctionalIndex`.
We need to do two things:
1. First make the spark context shudown properly. Note that this happens
only with tests that extend `HoodieSparkSqlTestBase`.
2. Fix the failure in individual tests.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7835
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
- Fix version(s):
- 1.1.0
---
## Comments
08/Jun/24 10:29;codope;I observed one more thing: Somehow the
TestFunctionalIndex is attempted again after TestSecondaryIndex. To verify, I
disabled TestFunctionalIndex and confirmed from the logs, it is attempted twice
(though it does not run because it is diabled). We need to root cause why
TestFunctionalIndex is being attempted twice. Note the test always succeeds
from IDE.
To reproduce run the following maven command from hudi repo root, and direct
the logs to some file where you can grep for any TestFunctionalIndex name, e.g.
`Test Create Functional Index`
{code:java}
mvn test -Pwarn-log -Dscala-2.12 -Dspark3.2 -Dflink1.18
-Dcheckstyle.skip=true -Drat.skip=true -Djacoco.skip=true -ntp -B -V -Pwarn-log
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn
-Punit-tests -Dtest=skipJavaTests -DfailIfNoTests=false
-DwildcardSuites=org.apache.hudi,org.apache.spark.hudi,org.apache.spark.sql.avro,org.apache.spark.sql.execution,org.apache.spark.sql.hudi.analysis,org.apache.spark.sql.hudi.command,org.apache.spark.sql.hudi.common,org.apache.spark.sql.hudi.dml
-pl
hudi-spark-datasource,hudi-spark-datasource/hudi-spark,hudi-spark-datasource/hudi-spark3.2.x,hudi-spark-datasource/hudi-spark3.2plus-common,hudi-spark-datasource/hudi-spark3-common,hudi-spark-datasource/hudi-spark-common
{code};;;
---
01/Oct/24 00:22;yihua;[~codope] is this still an issue?;;;
---
02/Oct/24 06:08;codope;This is still an issue -
[https://github.com/apache/hudi/actions/runs/11124499458/job/30910166426?pr=12035#step:6:11917]
We can look into this after addressing other user facing blocker tasks.;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
