[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675521#comment-16675521 ] Sunil Govindan commented on YARN-7512: -- Hi [~csingh] [~eyang] cud u pls give a short write up as Release Notes which says a bit about this feature. you could edit this jira, and update ReleaseNotes section for same. Thank you. > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn-native-services >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Major > Attachments: _In-Place Upgrade of Long-Running Applications in > YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf, > _In-Place Upgrade of Long-Running Applications in YARN_v3.pdf > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675510#comment-16675510 ] Sunil Govindan commented on YARN-7512: -- No fix-version given the tasks spanned across releases > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn-native-services >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Major > Attachments: _In-Place Upgrade of Long-Running Applications in > YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf, > _In-Place Upgrade of Long-Running Applications in YARN_v3.pdf > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406898#comment-16406898 ] Chandni Singh commented on YARN-7512: - Updated the design document after offline discussion with [~leftnoteasy] and [~gsaha] > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Major > Fix For: yarn-native-services > > Attachments: _In-Place Upgrade of Long-Running Applications in > YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf, > _In-Place Upgrade of Long-Running Applications in YARN_v3.pdf > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401405#comment-16401405 ] Eric Yang commented on YARN-7512: - In YARN-8018, we discussed that when AM is online, ServiceClient sends JSON to AM, and AM commit the file with a new version number after completing operations. When AM is offline, ServiceClient checks for version number in client supplied JSON. ServiceClient will use client set version number. If version number doesn't exist, then ServiceClient use existing version from HDFS and increment by one. This approach will resolve the version number conflicts, and provide a way to let user reset the number, if hdfs copy is corrupted. > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Major > Fix For: yarn-native-services > > Attachments: _In-Place Upgrade of Long-Running Applications in > YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401323#comment-16401323
]
Chandni Singh commented on YARN-7512:
-
{quote}I was just questioning whether the version should appear in the file
name of the file that is saved to HDFS.
{quote}
Since we are not restricting upgrade while another upgrade is in progress,
there will be conflicts during writing to the existing file from rest service
while Service AM is reading the file to figure out which components need
upgrade. YARN-8018 has these changes to initializing service upgrade. Please,
let me know if there are problems with the approach on YARN-8018.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401298#comment-16401298
]
Billie Rinaldi commented on YARN-7512:
--
bq. if there are only two versions allowed at a time, perhaps we shouldn't
include the version number in the upgrade service json, instead calling it
something like {service_name}_upgrade.json or {service_name}_next.json
Sorry, this comment I made was really unclear. I completely agree that version
should appear as a field in the json, I was just questioning whether the
version should appear in the file name of the file that is saved to HDFS.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401158#comment-16401158
]
Chandni Singh commented on YARN-7512:
-
[~leftnoteasy]
Just some comments.
{quote}1. Talked to [~csingh], existing upgrade only supports artifact, and all
other fields will be silently ignored. Probably we should rename the "upgrade"
to something like "set-image"?
{quote}
Initially, I am focusing on upgrade based on just artifact change. The design
doc does include other fields. Once basic upgrade is in place, the plan was to
enhance the feature.
{quote}2. Regarding to existing interactions between user and native services
to do updates. IIUC, this is workflow proposed in the design doc, plz correct
me if I was wrong:
a. User upload a new upgrade spec to HDFS.
b. User call upgrade service to read the new spec.
c. User call upgrade component(s) to update components and instances.
d. User manually set finalize or YARN will automatically set finalize of
services.
{quote}
There are not 4 steps but 3 steps. a) and b) are part of initializing the
service upgrade.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401140#comment-16401140
]
Wangda Tan commented on YARN-7512:
--
Thanks [~csingh] for working on this design, a couple of questions/comments.
1) Regarding to upgrade, from my understanding, it will be more valueable to
upgrade only for component's artifact and component's replicas.
For other part, since we have environment subsistution, it is possible that a
change of global environment could cause all dependencies upgraded. For example:
{code:java}
Service:
name: my-service.
environments:
env1: value1
env2: value2
components:
component:
name: comp1
launch_command: "python $env1 ..."
artifact: $env2-x-y-z
component:
name: comp2
launch_command: "app $env2 ..."
{code}
Change of $env1/$env2 could cause all component needs to be upgraded, I'm not
sure if that is desired state.
Talked to [~csingh], existing upgrade only supports artifact, and all other
fields will be silently ignored. Probably we should rename the "upgrade" to
something like "set-image"?
2) Regarding to existing interactions between user and native services to do
updates. IIUC, this is workflow proposed in the design doc, plz correct me if I
was wrong:
a. User upload a new upgrade spec to HDFS.
b. User call upgrade service to read the new spec.
c. User call upgrade component(s) to update components and instances.
d. User manually set finalize or YARN will automatically set finalize of
services.
First of all, I'm not sure if changing service state to UPGRADING could bring
and benefits. IMHO, it is confusing to let user manual finalize an upgrade but
not all components are upgraded.
And a/b look like a burden to end user. If user needs to call c anyway, why we
need a/b.
My proposal is to keep APIs to only upgrade *artifact* of component and
instance of components:
{code:java}
Response upgradeComponentArtifact(ServiceName, ComponentName, NewArtifactName,
[LowerIndexBound, HigherIndexBound])
{code}
The {{[LowerIndexBound, HigherIndexBound]}} is useful when user want to verify
the artifact works instead of bring down all instances. By default we will
upgrade all instances of the component.
Once API server received this, this operation will be persisted, and component
transferred to UPGRADING. And once all requested instances upgraded,
component's state will be updated to READY again.
3) Several future requirement to consider:
a. What's the rollback story? It might be good to list history artifacts of
instances and timestamp. Existing design doesn't support rollback to a specific
version.
b. What's the AM restart story? From Chandni now AM doesn't persistent upgrade
status of containers. We may need to consider this.
Thoughts?
cc:[~gsaha]/[~billie.rinaldi].
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401089#comment-16401089
]
Chandni Singh commented on YARN-7512:
-
[~billie.rinaldi]
{quote}if there are only two versions allowed at a time, perhaps we shouldn't
include the version number in the upgrade service json, instead calling it
something like \{service_name}_upgrade.json or \{service_name}_next.json. That
would also have the advantage of not restricting the characters that can appear
in the version string.
{quote}
My thoughts regarding this:
# Users should be able to specify their versions of a service. The version
could be representative of the artifact of the application that they are
running as a service.
# We will allow abort/upgrade while another upgrade is in progress. I think it
is important to have a version field in service json to let the user know which
version of their service is running.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394304#comment-16394304
]
Chandni Singh commented on YARN-7512:
-
[~billie.rinaldi]
Regarding points 2 and 3:
In the ServiceClientAPI with (YARN-8018), I am going to add
{code:java}
public int actionUpgrade(Service service)
{code}
I was planning to modify the document but didn't do it yet.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394233#comment-16394233
]
Billie Rinaldi commented on YARN-7512:
--
A couple of late comments:
* If there are only two versions allowed at a time, perhaps we shouldn't
include the version number in the upgrade service json, instead calling it
something like \{service_name}_upgrade.json or \{service_name}_next.json. That
would also have the advantage of not restricting the characters that can appear
in the version string.
* Is the ServiceClient API section missing a method to specify the new Service?
* If there is only one Service json that can be upgraded to, it seems like we
should remove the version parameter from the actionUpgrade(String appName,
String version) method.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373578#comment-16373578 ] Chandni Singh commented on YARN-7512: - [~gsaha] and I worked on the design of in-place upgrade of long running service. The design is attached to this jira. Thanks to [~jianhe] , [~billie.rinaldi] and [~eyang] for reviewing it. > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Major > Fix For: yarn-native-services > > Attachments: _In-Place Upgrade of Long-Running Applications in > YARN.pdf > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297322#comment-16297322 ] Gour Saha commented on YARN-7512: - Needs the version field as per YARN-7523 Introduce description and version field in Service record > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Chandni Singh > Fix For: yarn-native-services > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284474#comment-16284474 ] Gour Saha commented on YARN-7512: - Need to use YARN feature YARN-4726 - Allocation reuse for application upgrades > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
