[
https://issues.apache.org/jira/browse/HADOOP-19872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089357#comment-18089357
]
ASF GitHub Bot commented on HADOOP-19872:
-----------------------------------------
rohit-kb commented on PR #8550:
URL: https://github.com/apache/hadoop/pull/8550#issuecomment-4718223686
Thanks for confirming UBI 10 build!
Regarding your concern, The ENGINE API change only affects how the random
bytes are produced internally (RDRAND via ENGINE vs RDRAND via OpenSSL's
built-in provider in version 3.x). The output is on par random bytes in both
cases.
Existing keys, certificates, and encrypted data should not be affected. Here
is the overall flow I believe:
Encryption:
1. OpensslSecureRandom generates 32 random bytes for the key.
2. Key is stored in KMS
3. File is encrypted with that key and written to HDFS.
Decryption:
1. Fetch the key from KMS
2. Use the key to decrypt the file
OpensslSecureRandom is never used in decryption. Our change only affects
how the random bytes are generated inside OpensslSecureRandom. Whether those
random bytes came from Intel RDRAND via the ENGINE path or via OpenSSL's
built-in provider.
> Build failure with OpenSSL 3 (UBI10) in native code due to removed ENGINE API
> -----------------------------------------------------------------------------
>
> Key: HADOOP-19872
> URL: https://issues.apache.org/jira/browse/HADOOP-19872
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, native
> Affects Versions: 3.5.0, 3.4.3, 3.6.0
> Environment: # OS: Red Hat UBI10 container image
> # OpenSSL: 3.x (from UBI10) (package: openssl-devel-1:3.5.1-7.el10_1.x86_64)
> # Java: 11
> Reporter: rami
> Priority: Major
> Labels: pull-request-available
>
>
> Building Hadoop Common native code against OpenSSL 3 fails on UBI10 container
> images because the code uses legacy ENGINE APIs that have been removed by
> RedHat from the openssl-devel package. See the output from the build logs
> below.
>
> I want to also mention that UBI10 doesn't provide a package for the boost
> libraries anymore (in the epel10 repositories) so if anyone looks into this
> issue, they have to also come up with a workaround/fix for this problem.
>
> This is the relevant extract from the build logs:
> ```
> 727.1 [WARNING] make[2]: Leaving directory
> '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
> 727.1 [WARNING] make[2]: Leaving directory
> '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
> 727.1 [WARNING] make[1]: Leaving directory
> '/build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/target/native'
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:184:13:
> warning: 'loadSm4Ctr' defined but not used [-Wunused-function]
> 727.1 [WARNING] 184 | static void loadSm4Ctr(JNIEnv *env)
> 727.1 [WARNING] | ^~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:184:13:
> warning: 'loadSm4Ctr' defined but not used [-Wunused-function]
> 727.1 [WARNING] 184 | static void loadSm4Ctr(JNIEnv *env)
> 727.1 [WARNING] | ^~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'locks_setup':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33:
> warning: implicit declaration of function 'dlsym_CRYPTO_num_locks'; did you
> mean 'dlsym_CRYPTO_malloc'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 250 | lock_cs =
> dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() * \
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | dlsym_CRYPTO_malloc
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3:
> warning: implicit declaration of function 'dlsym_CRYPTO_set_id_callback';
> did you mean 'CRYPTO_set_id_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 257 | dlsym_CRYPTO_set_id_callback((unsigned long
> (*)())pthreads_thread_id);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_id_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3:
> warning: implicit declaration of function
> 'dlsym_CRYPTO_set_locking_callback'; did you mean
> 'CRYPTO_set_locking_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 258 | dlsym_CRYPTO_set_locking_callback((void
> (*)())pthreads_locking_callback);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_locking_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'openssl_rand_init':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> error: 'ENGINE_METHOD_RAND' undeclared (first use in this function)
> 727.1 [WARNING] 329 | rc = dlsym_ENGINE_set_default(eng,
> ENGINE_METHOD_RAND);
> 727.1 [WARNING] |
> ^~~~~~~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> note: each undeclared identifier is reported only once for each function it
> appears in
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> At top level:
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13:
> warning: 'locks_cleanup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 261 | static void locks_cleanup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13:
> warning: 'locks_setup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 247 | static void locks_setup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~
> 727.1 [WARNING] make[2]: *** [CMakeFiles/hadoop_static.dir/build.make:104:
> CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c.o]
> Error 1
> 727.1 [WARNING] make[2]: *** Waiting for unfinished jobs....
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'locks_setup':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33:
> warning: implicit declaration of function 'dlsym_CRYPTO_num_locks'; did you
> mean 'dlsym_CRYPTO_malloc'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 250 | lock_cs =
> dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() * \
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | dlsym_CRYPTO_malloc
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3:
> warning: implicit declaration of function 'dlsym_CRYPTO_set_id_callback';
> did you mean 'CRYPTO_set_id_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 257 | dlsym_CRYPTO_set_id_callback((unsigned long
> (*)())pthreads_thread_id);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_id_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3:
> warning: implicit declaration of function
> 'dlsym_CRYPTO_set_locking_callback'; did you mean
> 'CRYPTO_set_locking_callback'? [-Wimplicit-function-declaration]
> 727.1 [WARNING] 258 | dlsym_CRYPTO_set_locking_callback((void
> (*)())pthreads_locking_callback);
> 727.1 [WARNING] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 727.1 [WARNING] | CRYPTO_set_locking_callback
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> In function 'openssl_rand_init':
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> error: 'ENGINE_METHOD_RAND' undeclared (first use in this function)
> 727.1 [WARNING] 329 | rc = dlsym_ENGINE_set_default(eng,
> ENGINE_METHOD_RAND);
> 727.1 [WARNING] |
> ^~~~~~~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:329:40:
> note: each undeclared identifier is reported only once for each function it
> appears in
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:
> At top level:
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13:
> warning: 'locks_cleanup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 261 | static void locks_cleanup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~~~
> 727.1 [WARNING]
> /build/src/hadoop/hadoop/patchable-work/worktree/3.4.2/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13:
> warning: 'locks_setup' defined but not used [-Wunused-function]
> 727.1 [WARNING] 247 | static void locks_setup(void)
> 727.1 [WARNING] | ^~~~~~~~~~~
> 727.1 [WARNING] make[2]: *** [CMakeFiles/hadoop.dir/build.make:104:
> CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c.o]
> Error 1
> 727.1 [WARNING] make[2]: *** Waiting for unfinished jobs....
> 727.1 [WARNING] make[1]: *** [CMakeFiles/Makefile2:113:
> CMakeFiles/hadoop_static.dir/all] Error 2
> 727.1 [WARNING] make[1]: *** Waiting for unfinished jobs....
> 727.1 [WARNING] make[1]: *** [CMakeFiles/Makefile2:87:
> CMakeFiles/hadoop.dir/all] Error 2
> 727.1 [WARNING] make: *** [Makefile:91: all] Error 2
> 727.1 [INFO]
> ------------------------------------------------------------------------
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
