aIbrahiim commented on issue #30602:
URL: https://github.com/apache/beam/issues/30602#issuecomment-4914339222
I inevstigated the issue and here's my findings as until July 1 both Python
3.10 and 3.14 were passing after that, the job started failing, but at first
only the 3.14 job was red but the 3.10 job was still green until today
so the 3.14 failure seems to be in the Go cross language test
TestKafkaIO_BasicReadWrite and the Java io expansion service does not start
that's why I saw connection refused after 30 retries and JDBC tests pass, so it
looks specific to the Kafka expansion service. This started around July 4,
before PR #39205 also one thing to note: the 3.10 job uses
skipNonPythonTask=true, so it skips the Go test that fails on 3.14. That is why
3.10 still looked fine even when the overall workflow was failing
#39205 changed workflow default Java from 11 to 21 and after that, the 3.10
job also started failing with IllegalAccessError on sun.nio.ch.DirectBuffer. My
understanding is that XVR Spark3 starts Spark in a separate process through
run_job_server.sh but before #39205, that process was effectively running on
Java 11 (from the setup in #39058) and after #39205, it runs on Java 21, but it
does not get the same --add-opens flags that Gradle Spark tests already use in
sparkTestJvmArgs(). So #39205 added a new regression on 3.10.
so for 3.10 I would suggest passing the required --add-opens flags to the
Spark job server on Java 17+, maybe through run_job_server.sh and for 3.14, I
think the Kafka expansion service issue and I'm investigating it further.
@Abacn @Amar3tto @shunping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
