[
https://issues.apache.org/jira/browse/HUDI-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-7835:
---------------------------------
Labels: pull-request-available (was: )
> Spark context not stopped properly if the test had some exception that was
> ignored
> ----------------------------------------------------------------------------------
>
> Key: HUDI-7835
> URL: https://issues.apache.org/jira/browse/HUDI-7835
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Sagar Sumit
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> Found two tests that don't fail but throw an exception while running. The
> test succeeds but the spark context is not stopped in time due to exception.
> This causes issue for other tests. For example, `test("bucket index query")`
> in `TestDataSkippingQuery` succeeds but when we check the
> [logs|https://github.com/apache/hudi/actions/runs/9391927778/job/25865161535#step:6:5799]
> of the test we will find an error as below
> {code:java}
> 74954 [ScalaTest-run-running-TestDataSkippingQuery] ERROR
> org.apache.hudi.HoodieFileIndex [] - Failed to lookup candidate files in File
> Index
> java.lang.IllegalArgumentException: Property
> _hoodie.record.key.gen.partition.id not found
> at
> org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:69)
> ~[classes/:?]
> at
> org.apache.hudi.common.config.TypedProperties.getInteger(TypedProperties.java:94)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.generateSequenceId(AutoRecordGenWrapperKeyGenerator.java:115)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.getRecordKey(AutoRecordGenWrapperKeyGenerator.java:67)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.BaseKeyGenerator.getKey(BaseKeyGenerator.java:70)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketNumber$1(BucketIndexSupport.scala:154)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketSetFromValue$1(BucketIndexSupport.scala:168)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketsBySingleHashFields(BucketIndexSupport.scala:174)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.filterQueriesWithBucketHashField(BucketIndexSupport.scala:107)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.computeCandidateFileNames(BucketIndexSupport.scala:78)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3(HoodieFileIndex.scala:354)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3$adapted(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
> ~[scala-library-2.12.10.jar:?]
> at scala.collection.immutable.List.foreach(List.scala:392)
> ~[scala-library-2.12.10.jar:?]
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
> ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$1(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.lookupCandidateFilesInMetadataTable(HoodieFileIndex.scala:338)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:241)
> ~[classes/:?]
> at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:160)
> ~[classes/:?] {code}
> The test succeeds because HoodieFileIndex listFiles is under try-catch and
> falls back to no pruning (except for partition pruning). So, anyway due to
> the exception spark context runs into conflict with another test which throws
> exception as below:
> {code:java}
> org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> org.scalatest.Transformer.apply(Transformer.scala:22)
> org.scalatest.Transformer.apply(Transformer.scala:20)
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
> org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
> org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
> org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
> The currently active SparkContext was created at:
> org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.$anonfun$test$1(HoodieSparkSqlTestBase.scala:85)
> org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> org.scalatest.Transformer.apply(Transformer.scala:22)
> org.scalatest.Transformer.apply(Transformer.scala:20)
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
> org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
> org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
> org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
> org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
> org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:199)
> org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:181)
> org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1562)
> {code}
> Another example is `test("Test Functional Index With Hive Sync Non
> Partitioned Table")` in `TestFunctionalIndex`.
> We need to do two things:
> 1. First make the spark context shudown properly. Note that this happens only
> with tests that extend `HoodieSparkSqlTestBase`.
> 2. Fix the failure in individual tests.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
