[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angerszhu updated YARN-10495:
-----------------------------
Description:
In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on crypto
to container-executor, we meet a case that in our jenkins machine, we have
libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we don't
have libcrypto.so.1.0.0 but *libcrypto.so.1.1*
we build hadoop with
{code:java}
-Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
{code}
Under jenkins machine shared lib library pats /usr/lib/x86_64-linux-gun(where
is libcrypto)
{code:java}
-rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
-rw-r--r-- 1 root root 54550 Jun 18 2017 libcrypt.a
-rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so -> libcrypto.so.1.0.0
-rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
/lib/x86_64-linux-gnu/libcrypt.so.1
-rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
{code}
Under nodemanager shared lib library pats /usr/lib/x86_64-linux-gun(where is
libcrypto)
{code:java}
-rw-r--r-- 1 root root 55852 2�� 7 2019 libcrypt.a
-rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so -> libcrypto.so.1.1
-rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
-rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
/lib/x86_64-linux-gnu/libcrypt.so.1
-rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
{code}
We build container-executor with
The libcrypto.so 's version is not same case error when we start nodemanager
{code:java}
.. 3 more Caused by:
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
object file: No such file or directory at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
... 4 more Caused by: ExitCodeException exitCode=127:
/home/hadoop/hadoop/bin/container-executor: error while loading shared
libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or
directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
org.apache.hadoop.util.Shell.run(Shell.java:901) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
... 6 more
{code}
We should make RPATH of container-executor configurable to solve this problem
was:
In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on crypto
to container-executor, we meet a case that in our jenkins machine, we have
libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we don't
have libcrypto.so.1.0.0 but *libcrypto.so.1.1*
we build hadoop with
{code:java}
-Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
{code}
Under jenkins machine /usr/lib/x86_64-linux-gun
{code:java}
-rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
-rw-r--r-- 1 root root 54550 Jun 18 2017 libcrypt.a
-rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so -> libcrypto.so.1.0.0
-rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
/lib/x86_64-linux-gnu/libcrypt.so.1
-rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
{code}
Under nodemanager /usr/lib/x86_64-linux-gun
{code:java}
-rw-r--r-- 1 root root 55852 2�� 7 2019 libcrypt.a
-rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so -> libcrypto.so.1.1
-rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
-rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
/lib/x86_64-linux-gnu/libcrypt.so.1
-rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
{code}
The libcrypto.so 's version is not same case error when we start nodemanager
{code:java}
.. 3 more Caused by:
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
object file: No such file or directory at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
... 4 more Caused by: ExitCodeException exitCode=127:
/home/hadoop/hadoop/bin/container-executor: error while loading shared
libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or
directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
org.apache.hadoop.util.Shell.run(Shell.java:901) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
... 6 more
{code}
We should make RPATH of container-executor configurable to solve this problem
> make the rpath of container-executor configurable
> -------------------------------------------------
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: angerszhu
> Priority: Major
> Attachments: YARN-10495.001.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1*
> we build hadoop with
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library pats /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root 54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library pats /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
