[
https://issues.apache.org/jira/browse/YARN-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009286#comment-17009286
]
Siddharth Ahuja edited comment on YARN-10063 at 1/7/20 1:57 AM:
----------------------------------------------------------------
Hi [~pbacsko], thanks for your update. I see your point, however, if we make
\--https as optional, then, we still need to specify \--http there (even if
there are no additional details that need to be supplied). The reason why this
JIRA was actually created was because someone noticed "\--http" in the command
array but it wasn't clear when/how this got introduced because the usage output
from container-executor binary does not have this, e.g:
{code}
Full command array for failed execution:
[nice, -n, 0, /var/lib/yarn-ce/bin/container-executor, usr, usr, 1,
application_1576461726457_5994, container_e136_1576461726457_5994_01_001801,
/data07/yarn/nm/usercache/usr_ds_exec_hdp/appcache/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801,
/data09/yarn/nm/nmPrivate/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801/launch_container.sh,
/data08/yarn/nm/nmPrivate/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801/container_e136_1576461726457_5994_01_001801.tokens,
--http,
/data06/yarn/nm/nmPrivate/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801/container_e136_1576461726457_5994_01_001801.pid,
/data01/yarn/nm%/data02/yarn/nm%/data03/yarn/nm%/data04/yarn/nm%/data05/yarn/nm%/data06/yarn/nm%/data07/yarn/nm%/data08/yarn/nm%/data09/yarn/nm%/data10/yarn/nm%/data11/yarn/nm%/data12/yarn/nm,
/data01/yarn/container-logs%/data02/yarn/container-logs%/data03/yarn/container-logs%/data04/yarn/container-logs%/data05/yarn/container-logs%/data06/yarn/container-logs%/data07/yarn/container-logs%/data08/yarn/container-logs%/data09/yarn/container-logs%/data10/yarn/container-logs%/data11/yarn/container-logs%/data12/yarn/container-logs%/opt/yarn/container-logs,
cgroups=/var/lib/yarn-ce/cgroups/cpu/hadoop-yarn/container_e136_1576461726457_5994_01_001801/tasks]
{code}
As per:
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java#L127
and
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java#L131
we will either have --http or --https specified and if the latter is specified
then it will have keystore and truststore details.
Therefore, we could have something like this:
{code}
launch container: 1 appid containerid workdir container-script tokens
--http | --https keystorepath truststorepath pidfile nm-local-dirs nm-log-dirs
resources
launch docker container: 4 appid containerid workdir container-script
tokens --http | --https keystorepath truststorepath pidfile nm-local-dirs
nm-log-dirs docker-command-file resources
{code}
Please let me know if you are happy with this and I will provide another patch
based on the above. Thanks!
was (Author: sahuja):
Hi [~pbacsko], thanks for your update. I see your point, however, if we make
\--https as optional, then, we still need to specify \--http there (even if
there is no additional details that need to be supplied). The reason why this
JIRA was actually created was because someone noticed "\--http" in the command
array but it wasn't clear when/how this got introduced because the usage output
from container-executor binary does not have this, e.g:
{code}
Full command array for failed execution:
[nice, -n, 0, /var/lib/yarn-ce/bin/container-executor, usr, usr, 1,
application_1576461726457_5994, container_e136_1576461726457_5994_01_001801,
/data07/yarn/nm/usercache/usr_ds_exec_hdp/appcache/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801,
/data09/yarn/nm/nmPrivate/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801/launch_container.sh,
/data08/yarn/nm/nmPrivate/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801/container_e136_1576461726457_5994_01_001801.tokens,
--http,
/data06/yarn/nm/nmPrivate/application_1576461726457_5994/container_e136_1576461726457_5994_01_001801/container_e136_1576461726457_5994_01_001801.pid,
/data01/yarn/nm%/data02/yarn/nm%/data03/yarn/nm%/data04/yarn/nm%/data05/yarn/nm%/data06/yarn/nm%/data07/yarn/nm%/data08/yarn/nm%/data09/yarn/nm%/data10/yarn/nm%/data11/yarn/nm%/data12/yarn/nm,
/data01/yarn/container-logs%/data02/yarn/container-logs%/data03/yarn/container-logs%/data04/yarn/container-logs%/data05/yarn/container-logs%/data06/yarn/container-logs%/data07/yarn/container-logs%/data08/yarn/container-logs%/data09/yarn/container-logs%/data10/yarn/container-logs%/data11/yarn/container-logs%/data12/yarn/container-logs%/opt/yarn/container-logs,
cgroups=/var/lib/yarn-ce/cgroups/cpu/hadoop-yarn/container_e136_1576461726457_5994_01_001801/tasks]
{code}
As per:
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java#L127
and
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java#L131
we will either have --http or --https specified and if the latter is specified
then it will have keystore and truststore details.
Therefore, we could have something like this:
{code}
launch container: 1 appid containerid workdir container-script tokens
--http | --https keystorepath truststorepath pidfile nm-local-dirs nm-log-dirs
resources
launch docker container: 4 appid containerid workdir container-script
tokens --http | --https keystorepath truststorepath pidfile nm-local-dirs
nm-log-dirs docker-command-file resources
{code}
Please let me know if you are happy with this and I will provide another patch
based on the above. Thanks!
> Usage output of container-executor binary needs to include --http/--https
> argument
> ----------------------------------------------------------------------------------
>
> Key: YARN-10063
> URL: https://issues.apache.org/jira/browse/YARN-10063
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Siddharth Ahuja
> Assignee: Siddharth Ahuja
> Priority: Minor
> Attachments: YARN-10063.001.patch
>
>
> YARN-8448/YARN-6586 seems to have introduced a new option - "--http"
> (default) and "--https" that is possible to be passed in to the
> container-executor binary, see :
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L564
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L521
> however, the usage output seems to have missed this:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L74
> Raising this jira to improve this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
