This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 64ec5f1017e1 [SPARK-32246][BUILD][INFRA] Enable `streaming-kinesis-asl` tests in Github Action CI 64ec5f1017e1 is described below commit 64ec5f1017e1f2ca479060ca76f18b1c4a803b81 Author: Junyu Chen <junyu...@amazon.com> AuthorDate: Mon Dec 4 12:44:11 2023 -0800 [SPARK-32246][BUILD][INFRA] Enable `streaming-kinesis-asl` tests in Github Action CI ### What changes were proposed in this pull request? This PR attempts to set up Kinesis tests in one of the existing Github Actions. Note that currently there are totally 57 tests in the Kinesis-asl module, and this PR enabled 35 of them. The remaining tests requires interaction with Amazon Kinesis service which would incur billing costs to users. Hence they are not included in the Github Action. ### Why are the changes needed? Addressing the comments in this PR: https://github.com/apache/spark/pull/42581#issuecomment-1685925739 which attempts to upgrade the AWS SDK to v2 for Spark Kinesis connector. Since Kinesis tests are not being run in the Github Actions, there is no automated mechanism to verify the SDK v2 upgrade changes in this module. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? 1. All existing Github Actions passed. 2. All Kinesis tests passed when running locally: `export ENABLE_KINESIS_TESTS=1 && mvn test -Pkinesis-asl -pl connector/kinesis-asl` ``` Tests: succeeded 57, failed 0, canceled 0, ignored 0, pending 0 All tests passed. [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 13:25 min [INFO] Finished at: 2023-11-12T00:15:49+08:00 [INFO] ------------------------------------------------------------------------ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43736 from junyuc25/junyuc25/kinesis-test. Authored-by: Junyu Chen <junyu...@amazon.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- .github/workflows/build_and_test.yml | 2 +- dev/sparktestsupport/modules.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 4612b504ccdf..d58356ec1c5d 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -150,7 +150,7 @@ jobs: - >- mllib-local, mllib, graphx - >- - streaming, sql-kafka-0-10, streaming-kafka-0-10, + streaming, sql-kafka-0-10, streaming-kafka-0-10, streaming-kinesis-asl, yarn, kubernetes, hadoop-cloud, spark-ganglia-lgpl, connect, protobuf # Here, we split Hive and SQL tests into some of slow ones and the rest of them. diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py index 8995b7de0df9..15b2e8f186e5 100644 --- a/dev/sparktestsupport/modules.py +++ b/dev/sparktestsupport/modules.py @@ -359,7 +359,7 @@ streaming_kinesis_asl = Module( build_profile_flags=[ "-Pkinesis-asl", ], - environ={"ENABLE_KINESIS_TESTS": "1"}, + environ={"ENABLE_KINESIS_TESTS": "0"}, sbt_test_goals=[ "streaming-kinesis-asl/test", ], --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org