Anne Isabelle Macedo created YARN-11811:
-------------------------------------------
Summary: YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS doesn't know how to
deal with leading quotes
Key: YARN-11811
URL: https://issues.apache.org/jira/browse/YARN-11811
Project: Hadoop YARN
Issue Type: Bug
Components: container-executor
Affects Versions: 3.4.0
Environment: We're using EMR emr-7.6.0. I know that there might be
differences between EMR and FOSS Apache Hadoop, but I believe the bug also
exists on the FOSS version.
Reporter: Anne Isabelle Macedo
TL;DR the regex for USER_MOUNT_PATTERN [1] doesn't trim leading quotes if they
don't get sanitized somehow from a spark-submit command.
Context: we're trying to use the YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS conf on a
Hadoop cluster provisioned by AWS EMR and, when the value for this variable is
provided with quotes (such as '/tmp:/src/container'), we get a cryptic error
message such as this one on container-launch:
{{Exception message: Mount must be a localized resource: '/tmp/}}
I don't really know if this is a failure on AWS api for sanitizing these
quotes, but it seems that, if they make their way to container launch, the
variable's content will be used with this leading quote - thus provoking some
unexpected behavior.
However, without these quotes, I see errors such as:
{{{}Shell error output: Configuration does not allow mount{}}}{{{}{}}}
with a path that wasn't the one I provided in the variable.
I checked on regex101.com and it seems that the USER_MOUNT_PATTERN won't
actually ignore quotes, meaning that if they make it to hadoop they will be
part of the regex match groups and variable's content.
[1]
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java#L74
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
