[
https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508689#comment-15508689
]
Jian He commented on YARN-5621:
-------------------------------
Thanks for your explanation, some parts I'm missing:
bq. Both will start ContainerLocalizer instances
Is it starting both instances now? Not sure if I read the code wrong... It
seems not the case. Based on the code, if it's an already existing resource,
it will NOT start the ContainerLocalizer. e.g. below comments in
ResourceLocalizationService. Could you point me the right cod... maybe I'm
missing something.
{code}
/*
* Multiple containers will try to download the same resource. So the
* resource download should start only if
* 1) We can acquire a non blocking semaphore lock on resource
* 2) Resource is still in DOWNLOADING state
*/
{code}
bq. it may be used by both, concurrently.
This approach may not be easily worked for the new containers without
structural change, because for new containers, when localizer is started, the
work-dirs are not setup yet. It cannot create symlinks on localization for new
containers. The work-dirs are created later when launching the container.
bq. for services that upgrade over minutes/hours,
Not only for upgrades, it's also used by Tez for localizing resources on
demand.
I think I understand your approach now, basically,
To create the symlinks,
1. we start the localizer process,
2. send the symlinks over on localizer heartbeat,
3. localizer process create symlinks.
One question is, should we keep the localizer process around or terminates it
immediately after symlinks created. If we keep it around for certain time, we
need to add some life timeout for the localizer process. If it's immediately
terminated, that means every localize request by AM would spawn/kill a new
process, and I think that could churn over the machine resources.
> Support LinuxContainerExecutor to create symlinks for continuously localized
> resources
> --------------------------------------------------------------------------------------
>
> Key: YARN-5621
> URL: https://issues.apache.org/jira/browse/YARN-5621
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Jian He
> Assignee: Jian He
> Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch,
> YARN-5621.4.patch, YARN-5621.5.patch
>
>
> When new resources are localized, new symlink needs to be created for the
> localized resource. This is the change for the LinuxContainerExecutor to
> create the symlinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
