[
https://issues.apache.org/jira/browse/HADOOP-19872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089353#comment-18089353
]
ASF GitHub Bot commented on HADOOP-19872:
-----------------------------------------
razvan commented on PR #8550:
URL: https://github.com/apache/hadoop/pull/8550#issuecomment-4717861495
Thanks for creating this PR. I can confirm that building on UBI 10 works.
My understanding of this PR is that it turns the functions using the engine
api into no-ops without replacement. Is this correct?
If this is true, how does this change impact functionality of existing
deployments? Is an upgrade of Hadoop clusters without some form of (meta)data
migration even possible?
I don't know enough about how this API is/was used, but I imagine that any
keys/certs/secrets or encrypted (meta)data generated previously will not be
usable anymore after the upgrade. Am I correct ?
> Build failure with OpenSSL 3 (UBI10) in native code due to removed ENGINE API
> -----------------------------------------------------------------------------
>
> Key: HADOOP-19872
> URL: https://issues.apache.org/jira/browse/HADOOP-19872
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, native
> Affects Versions: 3.5.0, 3.4.3, 3.6.0
> Environment: # OS: Red Hat UBI10 container image
> # OpenSSL: 3.x (from UBI10) (package: openssl-devel-1:3.5.1-7.el10_1.x86_64)
> # Java: 11
> Reporter: rami
> Priority: Major
> Labels: pull-request-available
>
>
> Building Hadoop Common native code against OpenSSL 3 fails on UBI10 container
> images because the code uses legacy ENGINE APIs that have been removed by
> RedHat from the openssl-devel package. See the output from the build logs
> below.
>
> I want to also mention that UBI10 doesn't provide a package for the boost
> libraries anymore (in the epel10 repositories) so if anyone looks into this
> issue, they have to also come up with a workaround/fix for this problem.
>
> This is the relevant extract from the build logs:
> ```
> 727.1 [WARNING] make[2]: Leaving directory
> '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
> 727.1 [WARNING] make[2]: Leaving directory
> '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
> 727.1 [WARNING] make[1]: Leaving directory
> '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:184:13:
> warning: 'loadSm4Ctr' defined but not used [-Wunused-function]
> 727.1 [WARNING] 184 | static void loadSm4Ctr(JNIEnv *env)
> 727.1 [WARNING] | ^~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:184:13:
> warning: 'loadSm4Ctr' defined but not used [-Wunused-function]
> 727.1 [WARNING] 184 | static void loadSm4Ctr(JNIEnv *env)
> 727.1 [WARNING] | ^~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'locks_setup':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33:
> warning: implicit declaration of function 'dlsym_CRYPTO_num_locks'; did you
> mean 'dlsym_CRYPTO_malloc'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 250 | lock_cs =
> dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() * \
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | dlsym_CRYPTO_malloc
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3:
> warning: implicit declaration of function 'dlsym_CRYPTO_set_id_callback';
> did you mean 'CRYPTO_set_id_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 257 | dlsym_CRYPTO_set_id_callback((unsigned long
> (*)())pthreads_thread_id);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_id_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3:
> warning: implicit declaration of function
> 'dlsym_CRYPTO_set_locking_callback'; did you mean
> 'CRYPTO_set_locking_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 258 | dlsym_CRYPTO_set_locking_callback((void
> (*)())pthreads_locking_callback);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_locking_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'openssl_rand_init':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> error: 'ENGINE_METHOD_RAND' undeclared (first use in this function)
> 727.1 [WARNING] 329 | rc = dlsym_ENGINE_set_default(eng,
> ENGINE_METHOD_RAND);
> 727.1 [WARNING] |
> ^~~~~~~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> note: each undeclared identifier is reported only once for each function it
> appears in
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> At top level:
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13:
> warning: 'locks_cleanup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 261 | static void locks_cleanup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13:
> warning: 'locks_setup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 247 | static void locks_setup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~
> 727.1 [WARNING] make[2]: *** [CMakeFiles/hadoop_static.dir/build.make:104:
> CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c.o]
> Error 1
> 727.1 [WARNING] make[2]: *** Waiting for unfinished jobs....
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'locks_setup':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33:
> warning: implicit declaration of function 'dlsym_CRYPTO_num_locks'; did you
> mean 'dlsym_CRYPTO_malloc'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 250 | lock_cs =
> dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() * \
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | dlsym_CRYPTO_malloc
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3:
> warning: implicit declaration of function 'dlsym_CRYPTO_set_id_callback';
> did you mean 'CRYPTO_set_id_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 257 | dlsym_CRYPTO_set_id_callback((unsigned long
> (*)())pthreads_thread_id);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_id_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3:
> warning: implicit declaration of function
> 'dlsym_CRYPTO_set_locking_callback'; did you mean
> 'CRYPTO_set_locking_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 258 | dlsym_CRYPTO_set_locking_callback((void
> (*)())pthreads_locking_callback);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_locking_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'openssl_rand_init':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> error: 'ENGINE_METHOD_RAND' undeclared (first use in this function)
> 727.1 [WARNING] 329 | rc = dlsym_ENGINE_set_default(eng,
> ENGINE_METHOD_RAND);
> 727.1 [WARNING] |
> ^~~~~~~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> note: each undeclared identifier is reported only once for each function it
> appears in
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> At top level:
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13:
> warning: 'locks_cleanup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 261 | static void locks_cleanup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13:
> warning: 'locks_setup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 247 | static void locks_setup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~
> 727.1 [WARNING] make[2]: *** [CMakeFiles/hadoop.dir/build.make:104:
> CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c.o]
> Error 1
> 727.1 [WARNING] make[2]: *** Waiting for unfinished jobs....
> 727.1 [WARNING] make[1]: *** [CMakeFiles/Makefile2:113:
> CMakeFiles/hadoop_static.dir/all] Error 2
> 727.1 [WARNING] make[1]: *** Waiting for unfinished jobs....
> 727.1 [WARNING] make[1]: *** [CMakeFiles/Makefile2:87:
> CMakeFiles/hadoop.dir/all] Error 2
> 727.1 [WARNING] make: *** [Makefile:91: all] Error 2
> 727.1 [INFO]
> ------------------------------------------------------------------------
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
