[
https://issues.apache.org/jira/browse/YARN-9906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18046460#comment-18046460
]
ASF GitHub Bot commented on YARN-9906:
--------------------------------------
github-actions[bot] closed pull request #1659: YARN-9906. Fix Docker container
multi volume mounts bug
URL: https://github.com/apache/hadoop/pull/1659
> When setting multi volumes throurh the "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS"
> setting is not valid
> ---------------------------------------------------------------------------------------------------
>
> Key: YARN-9906
> URL: https://issues.apache.org/jira/browse/YARN-9906
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: lynn
> Priority: Major
> Labels: pull-request-available
> Attachments: docker_volume_mounts.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As
> [https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/DockerContainers.html#Application_Submission]
> described, when I set the item "{{YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" to
> multi volumes mounts, the value is a comma-separated list of mounts.}}
>
> {quote}vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker,
>
> YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro,/etc/group:/etc/group:ro;/etc/hadoop/conf:/etc/hadoop/conf"
> hadoop jar hadoop-examples.jar pi -Dyarn.app.mapreduce.am.env=$vars \
> -Dmapreduce.map.env=$vars -Dmapreduce.reduce.env=$vars 10 100{quote}
> I found the docker container can mount the first volume, so it can't be
> running successfully without report error!
> The code of
> [DockerLinuxContainerRuntime.java|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java]
> as follows:
> {quote}if (environment.containsKey(ENV_DOCKER_CONTAINER_MOUNTS)) {
> Matcher parsedMounts = USER_MOUNT_PATTERN.matcher(
> environment.get(ENV_DOCKER_CONTAINER_MOUNTS));
> if (!parsedMounts.find()) {
> throw new ContainerExecutionException(
> "Unable to parse user supplied mount list: "
> + environment.get(ENV_DOCKER_CONTAINER_MOUNTS));
> }{quote}
> The regex pattern is in
> [OCIContainerRuntime|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java]
> as follows
> {quote}static final Pattern USER_MOUNT_PATTERN = Pattern.compile(
> "(?<=^|,)([^:\\x00]+):([^:\\x00]+)" +
> "(:(r[ow]|(r[ow][+])?(r?shared|r?slave|r?private)))?(?:,|$)");{quote}
> it is seperated by comma indeed, but when i read the code of submit the jar
> to yarn , i find the code
> [Apps.java|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java]
> {quote}private static final Pattern VARVAL_SPLITTER = Pattern.compile(
> "(?<=^|,)" // preceded by ',' or line begin
> + '(' + Shell.ENV_NAME_REGEX + ')' // var group
> + '='
> + "([^,]*)" // val group
> );
> {quote}
> It is sepearted by comma as the same.
> So, I just modify the comma to semicolon(";") of the item
> "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
