[jira] [Commented] (HUDI-7835) Spark context not stopped properly if the test had some exception that was ignored
[
https://issues.apache.org/jira/browse/HUDI-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886302#comment-17886302
]
Sagar Sumit commented on HUDI-7835:
---
This is still an issue -
[https://github.com/apache/hudi/actions/runs/11124499458/job/30910166426?pr=12035#step:6:11917]
We can look into this after addressing other user facing blocker tasks.
> Spark context not stopped properly if the test had some exception that was
> ignored
> --
>
> Key: HUDI-7835
> URL: https://issues.apache.org/jira/browse/HUDI-7835
> Project: Apache Hudi
> Issue Type: Improvement
>Reporter: Sagar Sumit
>Priority: Critical
> Fix For: 1.0.0
>
>
> Found two tests that don't fail but throw an exception while running. The
> test succeeds but the spark context is not stopped in time due to exception.
> This causes issue for other tests. For example, `test("bucket index query")`
> in `TestDataSkippingQuery` succeeds but when we check the
> [logs|https://github.com/apache/hudi/actions/runs/9391927778/job/25865161535#step:6:5799]
> of the test we will find an error as below
> {code:java}
> 74954 [ScalaTest-run-running-TestDataSkippingQuery] ERROR
> org.apache.hudi.HoodieFileIndex [] - Failed to lookup candidate files in File
> Index
> java.lang.IllegalArgumentException: Property
> _hoodie.record.key.gen.partition.id not found
> at
> org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:69)
> ~[classes/:?]
> at
> org.apache.hudi.common.config.TypedProperties.getInteger(TypedProperties.java:94)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.generateSequenceId(AutoRecordGenWrapperKeyGenerator.java:115)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.getRecordKey(AutoRecordGenWrapperKeyGenerator.java:67)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.BaseKeyGenerator.getKey(BaseKeyGenerator.java:70)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketNumber$1(BucketIndexSupport.scala:154)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketSetFromValue$1(BucketIndexSupport.scala:168)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketsBySingleHashFields(BucketIndexSupport.scala:174)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.filterQueriesWithBucketHashField(BucketIndexSupport.scala:107)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.computeCandidateFileNames(BucketIndexSupport.scala:78)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3(HoodieFileIndex.scala:354)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3$adapted(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
> ~[scala-library-2.12.10.jar:?]
> at scala.collection.immutable.List.foreach(List.scala:392)
> ~[scala-library-2.12.10.jar:?]
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
> ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$1(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.lookupCandidateFilesInMetadataTable(HoodieFileIndex.scala:338)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:241)
> ~[classes/:?]
> at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:160)
> ~[classes/:?] {code}
> The test succeeds because HoodieFileIndex listFiles is under try-catch and
> falls back to no pruning (except for partition pruning). So, anyway due to
> the exception spark context runs into conflict with another test which throws
> exception as below:
> {code:java}
> org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> org.scalatest.Transformer.apply(Transformer.scala:22)
> org.scalatest.Transformer.apply(Transformer.scala:20)
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
> org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
> org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
> org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
> The currently active SparkContext was created at:
> org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.$anonfun$test$1(Hood
[jira] [Commented] (HUDI-7835) Spark context not stopped properly if the test had some exception that was ignored
[
https://issues.apache.org/jira/browse/HUDI-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886049#comment-17886049
]
Y Ethan Guo commented on HUDI-7835:
---
[~codope] is this still an issue?
> Spark context not stopped properly if the test had some exception that was
> ignored
> --
>
> Key: HUDI-7835
> URL: https://issues.apache.org/jira/browse/HUDI-7835
> Project: Apache Hudi
> Issue Type: Improvement
>Reporter: Sagar Sumit
>Priority: Critical
> Fix For: 1.0.0
>
>
> Found two tests that don't fail but throw an exception while running. The
> test succeeds but the spark context is not stopped in time due to exception.
> This causes issue for other tests. For example, `test("bucket index query")`
> in `TestDataSkippingQuery` succeeds but when we check the
> [logs|https://github.com/apache/hudi/actions/runs/9391927778/job/25865161535#step:6:5799]
> of the test we will find an error as below
> {code:java}
> 74954 [ScalaTest-run-running-TestDataSkippingQuery] ERROR
> org.apache.hudi.HoodieFileIndex [] - Failed to lookup candidate files in File
> Index
> java.lang.IllegalArgumentException: Property
> _hoodie.record.key.gen.partition.id not found
> at
> org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:69)
> ~[classes/:?]
> at
> org.apache.hudi.common.config.TypedProperties.getInteger(TypedProperties.java:94)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.generateSequenceId(AutoRecordGenWrapperKeyGenerator.java:115)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.getRecordKey(AutoRecordGenWrapperKeyGenerator.java:67)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.BaseKeyGenerator.getKey(BaseKeyGenerator.java:70)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketNumber$1(BucketIndexSupport.scala:154)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketSetFromValue$1(BucketIndexSupport.scala:168)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketsBySingleHashFields(BucketIndexSupport.scala:174)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.filterQueriesWithBucketHashField(BucketIndexSupport.scala:107)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.computeCandidateFileNames(BucketIndexSupport.scala:78)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3(HoodieFileIndex.scala:354)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3$adapted(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
> ~[scala-library-2.12.10.jar:?]
> at scala.collection.immutable.List.foreach(List.scala:392)
> ~[scala-library-2.12.10.jar:?]
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
> ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$1(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.lookupCandidateFilesInMetadataTable(HoodieFileIndex.scala:338)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:241)
> ~[classes/:?]
> at org.apache.hudi.HoodieFileIndex.listFiles(HoodieFileIndex.scala:160)
> ~[classes/:?] {code}
> The test succeeds because HoodieFileIndex listFiles is under try-catch and
> falls back to no pruning (except for partition pruning). So, anyway due to
> the exception spark context runs into conflict with another test which throws
> exception as below:
> {code:java}
> org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> org.scalatest.Transformer.apply(Transformer.scala:22)
> org.scalatest.Transformer.apply(Transformer.scala:20)
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
> org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
> org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
> org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1562)
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:199)
> The currently active SparkContext was created at:
> org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase.$anonfun$test$1(HoodieSparkSqlTestBase.scala:85)
> org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> org.scalatest.Ou
[jira] [Commented] (HUDI-7835) Spark context not stopped properly if the test had some exception that was ignored
[
https://issues.apache.org/jira/browse/HUDI-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853358#comment-17853358
]
Sagar Sumit commented on HUDI-7835:
---
I observed one more thing: Somehow the TestFunctionalIndex is attempted again
after TestSecondaryIndex. To verify, I disabled TestFunctionalIndex and
confirmed from the logs, it is attempted twice (though it does not run because
it is diabled). We need to root cause why TestFunctionalIndex is being
attempted twice. Note the test always succeeds from IDE.
To reproduce run the following maven command from hudi repo root, and direct
the logs to some file where you can grep for any TestFunctionalIndex name, e.g.
`Test Create Functional Index`
{code:java}
mvn test -Pwarn-log -Dscala-2.12 -Dspark3.2 -Dflink1.18 -Dcheckstyle.skip=true
-Drat.skip=true -Djacoco.skip=true -ntp -B -V -Pwarn-log
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn
-Punit-tests -Dtest=skipJavaTests -DfailIfNoTests=false
-DwildcardSuites=org.apache.hudi,org.apache.spark.hudi,org.apache.spark.sql.avro,org.apache.spark.sql.execution,org.apache.spark.sql.hudi.analysis,org.apache.spark.sql.hudi.command,org.apache.spark.sql.hudi.common,org.apache.spark.sql.hudi.dml
-pl
hudi-spark-datasource,hudi-spark-datasource/hudi-spark,hudi-spark-datasource/hudi-spark3.2.x,hudi-spark-datasource/hudi-spark3.2plus-common,hudi-spark-datasource/hudi-spark3-common,hudi-spark-datasource/hudi-spark-common
{code}
> Spark context not stopped properly if the test had some exception that was
> ignored
> --
>
> Key: HUDI-7835
> URL: https://issues.apache.org/jira/browse/HUDI-7835
> Project: Apache Hudi
> Issue Type: Improvement
>Reporter: Sagar Sumit
>Priority: Major
> Fix For: 1.0.0
>
>
> Found two tests that don't fail but throw an exception while running. The
> test succeeds but the spark context is not stopped in time due to exception.
> This causes issue for other tests. For example, `test("bucket index query")`
> in `TestDataSkippingQuery` succeeds but when we check the
> [logs|https://github.com/apache/hudi/actions/runs/9391927778/job/25865161535#step:6:5799]
> of the test we will find an error as below
> {code:java}
> 74954 [ScalaTest-run-running-TestDataSkippingQuery] ERROR
> org.apache.hudi.HoodieFileIndex [] - Failed to lookup candidate files in File
> Index
> java.lang.IllegalArgumentException: Property
> _hoodie.record.key.gen.partition.id not found
> at
> org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:69)
> ~[classes/:?]
> at
> org.apache.hudi.common.config.TypedProperties.getInteger(TypedProperties.java:94)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.generateSequenceId(AutoRecordGenWrapperKeyGenerator.java:115)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator.getRecordKey(AutoRecordGenWrapperKeyGenerator.java:67)
> ~[classes/:?]
> at
> org.apache.hudi.keygen.BaseKeyGenerator.getKey(BaseKeyGenerator.java:70)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketNumber$1(BucketIndexSupport.scala:154)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketSetFromValue$1(BucketIndexSupport.scala:168)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.getBucketsBySingleHashFields(BucketIndexSupport.scala:174)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.filterQueriesWithBucketHashField(BucketIndexSupport.scala:107)
> ~[classes/:?]
> at
> org.apache.hudi.BucketIndexSupport.computeCandidateFileNames(BucketIndexSupport.scala:78)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3(HoodieFileIndex.scala:354)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$3$adapted(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
> ~[scala-library-2.12.10.jar:?]
> at scala.collection.immutable.List.foreach(List.scala:392)
> ~[scala-library-2.12.10.jar:?]
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
> ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.$anonfun$lookupCandidateFilesInMetadataTable$1(HoodieFileIndex.scala:351)
> ~[classes/:?]
> at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.10.jar:?]
> at
> org.apache.hudi.HoodieFileIndex.lookupCandidateFilesInMetadataTable(HoodieFileIndex.scala:338)
> ~[classes/:?]
> at
> org.apache.hudi.HoodieFileIndex.filterF
