[
https://issues.apache.org/jira/browse/SPARK-46283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-46283.
-----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 44204
[https://github.com/apache/spark/pull/44204]
> Avoid testing the `streaming-kinesis-asl` module in the daily tests of
> branch-3.x.
> ----------------------------------------------------------------------------------
>
> Key: SPARK-46283
> URL: https://issues.apache.org/jira/browse/SPARK-46283
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 4.0.0
> Reporter: Yang Jie
> Assignee: Apache Spark
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> After the merge of https://github.com/apache/spark/pull/43736, the master
> branch began testing the `streaming-kinesis-asl` module.
> At the same time, because the daily test will reuse `build_and_test.yml`, the
> daily test of branch-3.x also began testing `streaming-kinesis-asl`.
> However, in branch-3.x, the env `ENABLE_KINESIS_TESTS` is hard-coded as 1 in
> `dev/sparktestsupport/modules.py`:
> https://github.com/apache/spark/blob/1321b4e64deaa1e58bf297c25b72319083056568/dev/sparktestsupport/modules.py#L332-L346
> which leads to the failure of the daily test of branch-3.x:
> - branch-3.3: https://github.com/apache/spark/actions/runs/7111246311
> - branch-3.4: https://github.com/apache/spark/actions/runs/7098435892
> - branch-3.5: https://github.com/apache/spark/actions/runs/7099811235
> ```
> [info]
> org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite ***
> ABORTED *** (1 second, 14 milliseconds)
> [info] java.lang.Exception: Kinesis tests enabled using environment
> variable ENABLE_KINESIS_TESTS
> [info] but could not find AWS credentials. Please follow instructions in AWS
> documentation
> [info] to set the credentials in your system such that the
> DefaultAWSCredentialsProviderChain
> [info] can find the credentials.
> [info] at
> org.apache.spark.streaming.kinesis.KinesisTestUtils$.getAWSCredentials(KinesisTestUtils.scala:258)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
> [info] at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
> [info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
> [info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
> [info] at
> org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
> [info] at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
> [info] at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
> [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
> [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [info] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [info] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [info] at java.lang.Thread.run(Thread.java:750)
> [info] Test run
> org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite started
> [info] Test
> org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilderOldApi
> started
> [info] Test
> org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilder
> started
> [info] Test run
> org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite
> finished: 0 failed, 0 ignored, 2 total, 0.244s
> [info] ScalaTest
> [info] Run completed in 8 seconds, 542 milliseconds.
> [info] Total number of tests run: 31
> [info] Suites: completed 4, aborted 4
> [info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
> [info] *** 4 SUITES ABORTED ***
> [error] Error: Total 37, Failed 0, Errors 4, Passed 33
> [error] Error during tests:
> [error]
> org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite
> [error]
> org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
> [error] org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite
> [error]
> org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite
> [error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests
> unsuccessful
> [error] Total time: 13 s, completed Dec 5, 2023 12:03:53 PM
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
