stankiewicz commented on issue #39789: URL: https://github.com/apache/beam/issues/39789#issuecomment-5868515501
### Root Cause In JDK 24+, [JEP 486](https://openjdk.org/jeps/486) permanently disabled the Security Manager, causing calls to `Subject.getSubject(AccessControlContext)` to throw an unconditional `UnsupportedOperationException`. Beam's Java expansion service forcibly pins `org.apache.kafka:kafka-clients:3.9.0`, which still invokes this legacy API inside `OAuthBearerSaslClientCallbackHandler` during the SASL handshake. When cross-language pipelines execute on an unpinned or JDK 24+ harness (such as `beam_java25_sdk`), this exception causes repeated authentication failures that surface to the user as a metadata fetch `TimeoutException`. ### Solution Bump the pinned dependency in `sdks/java/io/expansion-service/build.gradle` from `org.apache.kafka:kafka-clients:3.9.0` to `3.9.1` (or `3.9.2`). Kafka 3.9.1 backported [KAFKA-17078](https://issues.apache.org/jira/browse/KAFKA-17078), which adds the `SecurityManagerCompatibility` reflective shim to safely fall back to modern `Subject.current()` on Java 18+ runtimes. Because 3.9.1 is a backward-compatible patch release, it eliminates the crash on newer JDKs without introducing breaking API or protocol changes. ### Workaround (Until Solution is Deployed) Until the expansion service patch is deployed, pipelines should pin the Java SDK worker harness to a JDK 21 (LTS) image where `Subject.getSubject` is still supported. This can be configured by supplying `--sdk_harness_container_image_overrides=".*java.*,gcr.io/cloud-dataflow/v1beta3/beam_java21_sdk:<beam-version>"` when launching the pipeline. On JDK 21, the SASL callback handler executes without error and establishes the connection to the Kafka cluster normally. ### Alternative Considered but Rejected Upgrading `kafka-clients` directly to 4.0.0+ was considered because it natively supports modern JDKs and includes the compatibility shim. However, Kafka 4.0 is a major version release that introduces breaking changes, including the complete removal of ZooKeeper and changes to client configurations and protocol defaults. Upgrading to the 3.9.1 patch release delivers the necessary fix with zero disruption to existing Beam `KafkaIO` consumers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
