[
https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858564#comment-15858564
]
Hitesh Sharma commented on YARN-5501:
-------------------------------------
Hi [~jlowe],
First off all a big thanks for taking the time to look at the document and
sharing your thoughts. I appreciate it a lot.
bq. I am confused on how this will be used in practice. To me pre-initialized
containers means containers that have already started up with application- or
framework-specific resources localized, processes have been launched using
those resources, and potentially connections already negotiated to external
services. I'm not sure how YARN is supposed to know what mix of local
resources, users, and configs to use for preinitialized containers that will
get a good "hit rate" on container requests. Maybe I'm misunderstanding what
is really meant by "preinitialized," and some concrete, sample use cases with
detailed walkthroughs of how they work in practice would really help
crystallize the goals here.
Your understanding of pre-initialized containers is correct here. In the
proposed design YARN RM has the config to start pre-initialized containers and
this config is pretty much a launch context, which contains launch commands,
details of resources to localize, and we also provide the resource constraints
with which the container should be started. This configuration is currently
static, but in the future we intend to this to be pluggable, so we can extend
it to be dynamic and adjust based on cluster load.
The first use case happens to be a scenario where each of the container needs
to start some processes that take a lot of time to initialize (localization and
startup costs). YARN NM receives the config to start the pre-initialized
container (there is a dummy application that is associated with the pre-init
container for a specific application) and it follows the regular code paths for
a container which includes localizing resources and launching the container. As
you know, in YARN a container goes to RUNNING state once started, but a
pre-initialized container instead goes to PREINITIALIZED state (there are some
hooks which allow us to know that the container has initialized properly). From
this point the container is not different from a regular container as the
container monitor is overlooking it. The "Pool Manager" within YARN NM is used
to start the pre-initialized container and watches for container events like
stop in which case it simply tries to start it again. In other words at the
moment we simply use YARN RM to pick the nodes where pre-initialized container
should be started and let the "Pool Manager" in the NM manage the lifecycle of
the container.
When the AM for which we pre-initialized the container comes and asks for this
container then the "Container Manager" takes the pre-initialized container by
issuing a "detach" container event and "attaches" it to the application. We
added attachContainer and detachContainer events into ContainerExecutor which
allow us to define what they mean. As an example, in attachContainer we start a
new process within the cgroup of pre-initialized container. The PID to
container mapping within the ContainerExecutor is updated to reflect everything
accordingly (pre-initialized containers have a different container ID and
belong to a different application before they are taken up). As part of the
detachContainer all the resources associated with the pre-initialized container
are now associated with the new container and get cleaned up accordingly.
The other use case where we have prototyped container pooling is the scenario
where a container actually needs to be a Virtual Machine. Creation of VMs can
take a long time thus container pooling allows us to keep the empty VM shells
ready to go.
bq. Reusing containers across different applications is going to create some
interesting scenarios that don't exist today. For example, what does a
container ID for one of these looks like? How many things today assume that
all container IDs for an application are essentially prefixed by the
application ID? This would violate that assumption, unless we introduce some
sort of container ID aliasing where we create a "fake" container ID that maps
to the "real" ID of the reused container. It would be good to know how we're
going to treat container IDs and what applications will see when they get one
of these containers in response to their allocation request.
All pre-initialized containers belong to a specific application type. There is
a dummy application created to which the pre-initialized container are mapped.
As part of containerAttach and containerDetach event we disassociate the
containers between application. Specifically ContainerExecutor has a mapping of
container ID to PID file and as part of container detach we update this
mapping. For e.g. let's say the pre-init container had the ID container123 in
that case the mapping in executor would have
container123=../container123.pidfile, but as part of container attach and
detach we update this mapping so that it now looks like
newcontainer456=../container13.pidfile. The ContainerExecutors use this mapping
to locate the cgroup or Windows job object and thus all container events are
now issued on the pre-init container (i.e. the already started process).
bq. What happens for preinitialized container failures, both during application
execution and when idle? Do we let the application launch its own recovery
container, etc.
Eventually we want YARN RM should manage these things but for now in our PoC we
have "Pool Manager", which listens on these events, and simply keeps retrying.
At the moment we only use YARN RM to select nodes and pass config down.
bq. How does resource accounting/scheduling work with these containers? Are
they running in a dedicated queue? Can users go beyond their normal limits by
getting these containers outside of the app's queue? Will it look weird when
the user's queue isn't full yet we don't allow them any more containers because
they're already using the maximum number of preinitialized containers?
We need to figure this part out. One of the thoughts we have had is that
pre-init containers can be considered opportunistic which means they can get
killed in favor of other containers, but if they do get used then they take the
mantle of the new container.
bq. Could you elaborate on how the resizing works? How do the processes
running within the container being resized made aware of the new size
constraints? Today containers don't communicate with the NM directly, so I'm
not sure how the preinitialized containers are supposed to know they are
suddenly half the size or can now leverage more memory than they could before.
Without that communication channel it seems like we're either going to kill
processes that overflowed their lowered memory constraint or we're going to
waste cluster resources because the processes are still trying to fit within
the old memory size.
Strictly speaking we haven't prototyped this part, but the idea is to reuse
container allocation increase mechanisms. For e.g. if the pre-init container
was running with 2 core and 2GB then after "attach" it could have it's
resources increased to 4 core and 4GB. The resizing is simply at the job object
or cgroup level and we expect the application to have it's own communication
channel to talk with the processes that are started a priori.
bq. What are the security considerations? Are preinitialized containers tied
to a particular user? How are app-specific credentials conveyed to the
preinitialized container, and can credentials leak between apps that use the
same container?
We haven't prototyped this part as much. Currently we start the pre-init
containers by skipping some of the security checks done in the "Container
Manager". I think we can instead configure the user with which pre-init
containers should be started and then associate them with the actual
application.
bq. Is this some separate, new protocol for advertising or is this just simply
reporting the container is launched just like other container status today?
The RM already knows it sent the NM a command to launch the container, so it
seems this is just the NM reporting the state of the container is now launched
as it does for any other container start request today, but I wasn't sure if
that is what was meant here.
This is kind of reporting back to the RM that the pre-init container is ready.
We have been thinking of using [YARN-3926] to advertise the pre-init container
as resources so they can be requested by the AMs.
bq. I'm confused, I thought the preinitialized container is already launched,
but this talks about launching it after attach. Again a concrete use-case
walkthrough would help clarify what's really being proposed. If this is
primarily about reducing localization time instead of process startup after
localization then there are simpler approaches we can take.
You are correct that the pre-init containers are already started. But in our
scenarios a container can have multiple processes running within the same
cgroup or job object. With container pooling we start some of these processes a
priori and the remaining are started when the AM comes around asking for
containers. The containers for our application look for the processes started a
priori and communicate with them.
Again thanks a ton for the excellent feedback and look forward to discuss more.
We can refine some of the ideas here and make them more generic and useful to
the community.
> Container Pooling in YARN
> -------------------------
>
> Key: YARN-5501
> URL: https://issues.apache.org/jira/browse/YARN-5501
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Arun Suresh
> Assignee: Hitesh Sharma
> Attachments: Container Pooling - one pager.pdf
>
>
> This JIRA proposes a method for reducing the container launch latency in
> YARN. It introduces a notion of pooling *Unattached Pre-Initialized
> Containers*.
> Proposal in brief:
> * Have a *Pre-Initialized Container Factory* service within the NM to create
> these unattached containers.
> * The NM would then advertise these containers as special resource types
> (this should be possible via YARN-3926).
> * When a start container request is received by the node manager for
> launching a container requesting this specific type of resource, it will take
> one of these unattached pre-initialized containers from the pool, and use it
> to service the container request.
> * Once the request is complete, the pre-initialized container would be
> released and ready to serve another request.
> This capability would help reduce container launch latencies and thereby
> allow for development of more interactive applications on YARN.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
