[
https://issues.apache.org/jira/browse/FLINK-38284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016234#comment-18016234
]
Jaehyun Kim commented on FLINK-38284:
-------------------------------------
Hadoop 3.4.2 is not yet released.
Once it is officially available, I plan to follow up with a PR to update
`fs.hadoopshaded.version` and the relevant modules.
> Prepare to upgrade hadoop version to 3.4.2 across Flink's Hadoop-based FS
> connectors for OpenSSL 3 and Java 17 compatibility
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-38284
> URL: https://issues.apache.org/jira/browse/FLINK-38284
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
> Reporter: Jaehyun Kim
> Priority: Major
>
> h3. *Description*
> Apache Hadoop has merged [PR
> #7032|https://github.com/apache/hadoop/pull/7032] and HADOOP-19262, upgrading
> wildfly-openssl to 2.1.6.Final to compatibility with Java 17 and OpenSSL 3.
> This fix is planned to be included in the upcoming Hadoop 3.4.2 release.
> Currently, Flink sets in {{flink-fliesystems/pom.xml}} :
> {code:java}
> <fs.hadoopshaded.version>3.3.4</fs.hadoopshaded.version> {code}
> which means modules like {{flink-azure-fs-hadoop.jar}} transitively include
> {{wildfly-oepnssl:1.0.7:Final}} via {{{}hadoop-azure:3.3.4{}}}. This version
> is not compatible with OpenSSL 3 and causes runtime issues on modern
> platforms.
> h3. *Impact and Scope*
> This issue originates in Apache Hadoop's {{hadoop-azure}} module, which
> transitively includes an outdated version of {{{}wildfly-openssl{}}}. As a
> result, all Flink modules depending on this (e.g.,
> {{{}flink-azure-fs-hadoop{}}}) are affected.
> Furthermore, other Flink filesystem connectors that rely on Hadoop (directly
> or via {{{}flink-shaded-hadoop{}}}) may also benefit from this upgrade:
> * {{flink-azure-fs-hadoop}}
> * {{flink-gs-fs-hadoop}}
> * {{flink-oss-fs-hadoop}}
> * {{flink-s3-fs-hadoop}}
> This change is particularly relevant for users running Flink on:
> * {*}Java 17{*}, where {{X509V1CertImpl}} was removed from the JDK
> * *OpenSSL 3.x systems* (e.g., RHEL 9), where older {{wildfly-openssl}}
> versions fail to load
> h3. *Motivation*
> Upgrading to {{hadoop-azure:3.4.2}} will:
> * Ensure compatibility with Java 17+ and OpenSSL 3
> * Resolve {{ClassNotFoundException:
> com.sun.security.cert.internal.x509.X509V1CertImpl}} errors on OpenSSL
> 1.1-based systems (e.g., RHEL 8.10)
> * Align with Hadoop upstream fixes
> * Avoid of performance-impacting workarounds like forcing
> {{fs.azure.ssl.channel.mode=Default_JSSE}}
> * Even when JSSE fallback avoids the crash, {*}it is not ideal for
> performance and stability{*}.
> Using native OpenSSL via JNI (as intended by {{{}wildfly-openssl{}}}) is
> preferred in high-throughput or secure production environments.
> h3. *Proposed Plan*
> Once Hadoop 3.4.2 is officially released:
> # Update {{fs.hadoopshaded.version}} to {{3.4.2}} in
> {{flink-filesystems/pom.xml}}
> # Verify and update NOTICE/LICENSE files as required
> # Rebuild {{flink-azure-fs-hadoop}} to confirm correct shading of the
> updated dependencies
> # Ensure that native SSL initialization works in both OpenSSL 1.1 and 3
> environments
> # Optionally, update test coverage for ABFS + SSL
> This ticket serves to track the upgrade preparation and corresponding work
> once the upstream Hadoop release is available.
> h3. *Environment Affected*
> * Flink 1.19.0 - 2.1.0
> * Java 17 (OracleJDK, OpenJDK, Amazon Corretto)
> * RHEL 8.10 (OpenSSL 1.1.1) → native loads, causes error
> {code:java}
> [ERROR] org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error
> occurred in the cluster entrypoint.java.util.concurrent.CompletionException:
> java.lang.RuntimeException: java.lang.IllegalStateException:
> javax.security.cert.CertificateException: Could not find class:
> java.lang.ClassNotFoundException:
> com/sun/security/cert/internal/x509/X509V1CertImpl{code}
> * RHEL 9.3 (OpenSSL 3.x) → native fails, JSSE fallback
> {code:java}
> [DEBUG] org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory [] -
> Failed to load OpenSSL. Falling back to the JSSE{code}
> * ABFS with HA enabled ({{{}abfss://{}}})
> h3. *Workarounds Today*
> * Set {{fs.azure.ssl.channel.mode:Default_JSSE}} in {{config.yaml}} to
> disable native OpenSSL
> * Avoid OpenSSL 1.1 platforms
> * Remove the {{wildfly-openssl}} JAR from the opt plugin (not ideal)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
