Abacn commented on issue #35522:
URL: https://github.com/apache/beam/issues/35522#issuecomment-3086431035
I think I found the cause. The dependency actually causing problem is
project(":sdks:java:io:iceberg:bqms"), i.e., the jar
https://storage.googleapis.com/spark-lib/bigquery/iceberg-bigquery-catalog-1.6.1-1.0.0.jar
shaded conscrypt jar, which won't work, essentially another realization of
https://issues.apache.org/jira/browse/BEAM-9288. See also
https://github.com/apache/beam/blob/43900635d97a98f08305eeef6dabf7adbe73e638/runners/google-cloud-dataflow-java/worker/build.gradle#L197-L199
Ironically this is a long standing issue for BQ Spark connection:
https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/274
(even though marked as solved, the cause wasn't fully understood)
It started to bite us due to some dependency change make Java classloader
picked up `org.conscrypt.NativeCryptoJni` class bqms-connector jar, and it
loaded jni in the same jar, but jni cannot find NativeCryptoJni, and aborted
JVM. Previously JVM classloader resolved to the correct unshaded conscrypt jar.
The good news is this appears to be test only, as
":sdks:java:io:iceberg:bqms" is not published and for test purpose only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
