Jaehyun Kim created FLINK-38284:
-----------------------------------
Summary: Prepare to upgrade hadoop version to 3.4.2 across Flink's
Hadoop-based FS connectors for OpenSSL 3 and Java 17 compatibility
Key: FLINK-38284
URL: https://issues.apache.org/jira/browse/FLINK-38284
Project: Flink
Issue Type: Improvement
Components: Connectors / FileSystem, FileSystems
Reporter: Jaehyun Kim
h3. Description
Apache Hadoop has merged [PR #7032|https://github.com/apache/hadoop/pull/7032]
and HADOOP-19262, upgrading wildfly-openssl to 2.1.6.Final to compatibility
with Java 17 and OpenSSL 3. This fix is planned to be included in the upcoming
Hadoop 3.4.2 release.
Currently, Flink sets in {{flink-fliesystems/pom.xml}} :
{code:java}
<fs.hadoopshaded.version>3.3.4</fs.hadoopshaded.version> {code}
which means modules like {{flink-azure-fs-hadoop.jar}} transitively include
{{wildfly-oepnssl:1.0.7:Final}} via {{{}hadoop-azure:3.3.4{}}}. This version is
not compatible with OpenSSL 3 and causes runtime issues on modern platforms.
h3. Impact and Scope
This issue originates in Apache Hadoop's {{hadoop-azure}} module, which
transitively includes an outdated version of {{{}wildfly-openssl{}}}. As a
result, all Flink modules depending on this (e.g.,
{{{}flink-azure-fs-hadoop{}}}) are affected.
Furthermore, other Flink filesystem connectors that rely on Hadoop (directly or
via {{{}flink-shaded-hadoop{}}}) may also benefit from this upgrade:
* {{flink-azure-fs-hadoop}}
* {{flink-gs-fs-hadoop}}
* {{{}flink-oss-fs-hadoop{}}}{{{}{}}}
* {{{}flink-s3-fs-hadoop{}}}{{{}{}}}
This change is particularly relevant for users running Flink on:
* {*}Java 17{*}, where {{X509V1CertImpl}} was removed from the JDK
* *OpenSSL 3.x systems* (e.g., RHEL 9), where older {{wildfly-openssl}}
versions fail to load
h3. Motivation
Upgrading to {{hadoop-azure:3.4.2}} will:
* Ensure compatibility with Java 17+ and OpenSSL 3
* Resolve {{ClassNotFoundException:
com.sun.security.cert.internal.x509.X509V1CertImpl}} errors on OpenSSL
1.1-based systems (e.g., RHEL 8.10)
* Align with Hadoop upstream fixes
* Avoid of performance-impacting workarounds like forcing
{{fs.azure.ssl.channel.mode=Default_JSSE}}
* Even when JSSE fallback avoids the crash, {*}it is not ideal for performance
and stability{*}.
Using native OpenSSL via JNI (as intended by {{{}wildfly-openssl{}}}) is
preferred in high-throughput or secure production environments.
h3. Proposed Plan
Once Hadoop 3.4.2 is officially released:
# Update {{fs.hadoopshaded.version}} to {{3.4.2}} in
{{flink-filesystems/pom.xml}}
#
# Verify and update NOTICE/LICENSE files as required
# Rebuild {{flink-azure-fs-hadoop}} to confirm correct shading of the updated
dependencies
# Ensure that native SSL initialization works in both OpenSSL 1.1 and 3
environments
# Optionally, update test coverage for ABFS + SSL
This ticket serves to track the upgrade preparation and corresponding work once
the upstream Hadoop release is available.
h3. Environment Affected
* Flink 1.19.0 - 2.1.0
* Java 17 (OracleJDK, OpenJDK, Amazon Corretto)
* RHEL 8.10 (OpenSSL 1.1.1) → native loads, causes error
{code:java}
[ERROR] org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error
occurred in the cluster entrypoint.java.util.concurrent.CompletionException:
java.lang.RuntimeException: java.lang.IllegalStateException:
javax.security.cert.CertificateException: Could not find class:
java.lang.ClassNotFoundException:
com/sun/security/cert/internal/x509/X509V1CertImpl{code}
* RHEL 9.3 (OpenSSL 3.x) → native fails, JSSE fallback
{code:java}
[DEBUG] org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory [] - Failed
to load OpenSSL. Falling back to the JSSE{code}
* ABFS with HA enabled ({{{}abfss://{}}})
h3. Workarounds Today
* Set {{fs.azure.ssl.channel.mode=Default_JSSE}} to disable native OpenSSL
* Avoid OpenSSL 1.1 platforms
* Remove the {{wildfly-openssl}} JAR from the opt plugin (not ideal)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
