[
https://issues.apache.org/jira/browse/YARN-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065369#comment-15065369
]
Steve Loughran commented on YARN-4470:
--------------------------------------
In SLIDER-787 we've already implemented AM upgrade. Specifically, we just have
the AM commit suicide and rely on AM restart to bring itself back up, getting
the list of containers back and then rebuilding our state. We also rely on the
RM to update the HDFS and other tokens as well as the AM/RM token.
As the NMs download the resources again, we pick up the new binaries. What we
can't do currently is (a) change AM resource requirements or (b) avoid that AM
restart being mistaken for a failure. YARN-3417 proposes a specific exit code
there.
Accordingly, I'm not convinced we need to do anything here other than treat a
specific AM failure exit code/reported exit as a "restart is not a failure"
It does require them AM to initiate the upgrade —but it needs to do this for
container upgrades anyway. Without the AM doing that part of the process, you'd
end up with the AM at, say, v1.3 and the containers at 1.2. The AM needs to
think about version mismatch in AM/container communications, and how to upgrade
the containers by selective restart.
the clients don't need to worry about handoff across versions provided they
don't cache URLs/IPC connections, but they need to recover those for AM
failover anyway. Same for containers, which need to cope with the AM coming up
somewhere else. We use the YARN-913 registry binding for that.
The main enhancements of this proposal there are (a) side-by-side startup &
handoff and (b) rollback. Rollback isn't necessarily something that an app can
easily do: what happens if the upgrade AM fails in "that short time period"
after changing some state in HDFS, ZK, the containers, etc: you may be able to
rollback the binaries, but the persistent state can have changed.
w.r.t side-by-side, again, there's that time window. In slider we build up our
internal state on a restart based on the containers we get in AM registration,
updating it as queued container failure events start coming in. We actually
have to synchronize the AM rebuild process so that container callbacks don't
come until that state has been rebuilt. If the AM came up alongside the
existing one, it'd get confused pretty fast in the presence of container
failures during this handoff period. Either it'd be told of them (state
current, new container requests triggered) or not told of them (state
inconsistent). You'd have to do a lot of work
To summarise: even if this feature existed I don't think we'd move slider to
it; all we'd like is the YARN-3417 exit code, the ability to restart in the
same container (==no queuing delay) and the ability to request expanded AM
resources. I could imagine actually separating the two: request a resize in the
AM container, then, once granted, triggering the restart. Otherwise, we've got
the complexity in the code for AM upgrades, with the hard part actually dealing
with AM restart midway through rolling container upgrade, and rollback of
container upgrades.
I think before trying to implement this feature, have a go at implementing
rolling upgrades in an existing app and see what's missing.
> Application Master in-place upgrade
> -----------------------------------
>
> Key: YARN-4470
> URL: https://issues.apache.org/jira/browse/YARN-4470
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
> Reporter: Giovanni Matteo Fumarola
> Assignee: Giovanni Matteo Fumarola
> Attachments: AM in-place upgrade design. rev1.pdf
>
>
> It would be nice if clients could ask for an AM in-place upgrade.
> It will give to YARN the possibility to upgrade the AM, without losing the
> work
> done within its containers. This allows to deploy bug-fixes and new versions
> of the AM incurring in long service downtimes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
