[
https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345671#comment-16345671
]
Zian Chen commented on YARN-7626:
---------------------------------
Hi [~miklos.szeg...@cloudera.com] , thank you so much for your comments, for
is_regex function, missing "through" for iteration and isUserMount boolean
value change, I'm totally agreed with your suggestions. I'll update the patch
according to your comments.
For those two contradictions, I would like to explain more in details, line 853
and line 925-930 are inside function check_mount_permitted, this function
accepts two params as input, permitted_mounts are mounts which are specified by
admin which can be permitted for checking the user mounts, and requested are
user mounts. in line
{code:java}
char *normalized_path = normalize_mount(requested, 0);{code}
we call function normalize_mount which will lead to line 853, in this function
we use "isUsermount" to distinguish the input mount is user mount or permitted
mount, here, in this case, it's user mount, then this part of the code will not
be executed for user mount,
{code:java}
// we only allow permitted mount to be REGEX, for permitted mount, we check
// if it's a valid REGEX return; for user mount, we need to strictly check
if (isUserMount != 0) {
if (is_regex(mount) == 0) {
return strdup(mount);
}
}
{code}
we will follow the original logic which checks if the user mount is a real
path, if not, verify if it's a validate volume name, then do proper actions and
return normalized_path for user mount.
Then line 925-930 is used to check if permitted mounts are a regular
expression. If it is, we need to use regex matching to see if current
permitted_mounts can be matched with normalized_path of the user mount, that's
what line 925-930 trying to do. Let me give you an example for this,
for example one of the permitted_mounts is "^/usr/local/.*$" which is a regex
and the user mount is "//usr/local/bin/", when we call function
check_mount_permitted, we will call function normalize_mount for user mount
"//usr/local/bin/", check if it's a real path, it is, then the rest of code
will normalize the user mount into "/usr/local/bin/" then return as
normalized_path, then we go back to function check_mount_permitted and loop
permitted_mounts, no other can match "/usr/local/bin/" except regex
"^/usr/local/.*$", and this is because we use line 925-930 check
permitted_mounts is a regex and it can match "/usr/local/bin/" using regex
matching.
However, function does not always get user mount as input, like when we call
function normalize_mounts in these two lines inside function add_mounts,
{code:java}
ret = normalize_mounts(permitted_ro_mounts, 1);
ret |= normalize_mounts(permitted_rw_mounts, 1);{code}
normalize_mount will get permitted_mounts which let the code logic enter into
this part,
{code:java}
// we only allow permitted mount to be REGEX, for permitted mount, we check
// if it's a valid REGEX return; for user mount, we need to strictly check
if (isUserMount != 0) {
if (is_regex(mount) == 0) {
return strdup(mount);
}
}{code}
in this situation, we only accept permitted_mounts as regex when it's not a
real path, otherwise, we don't allow it as a valid permitted_mount.
For the second contradiction, "this check should be before if
(strcmp(normalized_path, permitted_mounts[i]) == 0)", the order of the first
check and second check is not matter, because inside permitted_mounts array, we
have regex format permitted_mount as well as non-regex ones, we try to use
every possible permitted_mouunt to see if we could match current
normalized_path, no matter the permitted_mount is regex or non-regex, if it's
non-regex, we use strcmp to match, if it's regex, we use function
validate_volume_name_with_argument to match.
Hope my explanation helps with the understanding of this part of the code,
further comments are highly welcomed. Wangda Tan What's your opinion
> Allow regular expression matching in container-executor.cfg for devices and
> named docker volumes mount
> ------------------------------------------------------------------------------------------------------
>
> Key: YARN-7626
> URL: https://issues.apache.org/jira/browse/YARN-7626
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Zian Chen
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-7626.001.patch, YARN-7626.002.patch,
> YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch
>
>
> Currently when we config some of the GPU devices related fields (like ) in
> container-executor.cfg, these fields are generated based on different driver
> versions or GPU device names. We want to enable regular expression matching
> so that user don't need to manually set up these fields when config
> container-executor.cfg,
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
