[jira] [Created] (YARN-8034) Clarification on preferredHost request with relaxedLocality
Jagadish created YARN-8034: -- Summary: Clarification on preferredHost request with relaxedLocality Key: YARN-8034 URL: https://issues.apache.org/jira/browse/YARN-8034 Project: Hadoop YARN Issue Type: Bug Reporter: Jagadish I work on Apache Samza, a stateful stream-processing framework that leverages Yarn for resource management. The Samza AM requests resources on specific hosts to schedule stateful jobs. We set relaxLocality = true in these requests we make to Yarn. Often we have observed that we don't get containers on the hosts that we requested them on and the Yarn RM returns containers on arbitrary hosts. Do you know what the behavior of the FairScheduler/CapacityScheduler is when setting "relaxLocality = true".I did play around by setting a high value for yarn.scheduler.capacity.node-locality-delay but it did not seem to matter. However, when setting relaxLocality = false, we get resources on the exact hosts we requested on. The behavior I want from Yarn is "Honor locality to the best possible extent and only return a container on an arbitrary host if the requested host is down". Is there a way to accomplish this? If you can point me to the Scheduler code, I'm happy to look at it as well. For context, we have continuous scheduling enabled in our clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401405#comment-16401405 ] Eric Yang commented on YARN-7512: - In YARN-8018, we discussed that when AM is online, ServiceClient sends JSON to AM, and AM commit the file with a new version number after completing operations. When AM is offline, ServiceClient checks for version number in client supplied JSON. ServiceClient will use client set version number. If version number doesn't exist, then ServiceClient use existing version from HDFS and increment by one. This approach will resolve the version number conflicts, and provide a way to let user reset the number, if hdfs copy is corrupted. > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Major > Fix For: yarn-native-services > > Attachments: _In-Place Upgrade of Long-Running Applications in > YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8002) Support NOT_SELF and ALL namespace types for allocation tag
[
https://issues.apache.org/jira/browse/YARN-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401385#comment-16401385
]
Weiwei Yang commented on YARN-8002:
---
Hi [~leftnoteasy], can you please take a look at the latest patch? Thanks
> Support NOT_SELF and ALL namespace types for allocation tag
> ---
>
> Key: YARN-8002
> URL: https://issues.apache.org/jira/browse/YARN-8002
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8002.001.patch, YARN-8002.002.patch,
> YARN-8002.003.patch, YARN-8002.004.patch
>
>
> This is a continua task after YARN-7972, YARN-7972 adds support to specify
> tags with namespace SELF and APP_ID, like following
> * self/
> * app-id//
> this task is to track the work to support 2 of remaining namespace types
> *NOT_SELF* & *ALL* (we'll support app-label later),
> * not-self/
> * all/
> this will require a bit refactoring in {{AllocationTagsManager}} as it needs
> to do some proper aggregation on tags for multiple apps.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
[
https://issues.apache.org/jira/browse/YARN-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401384#comment-16401384
]
genericqa commented on YARN-8032:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
31s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-8032 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914806/YARN-8032.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 73caa00316b9 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4bf6220 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19991/testReport/ |
| Max. process+thread count | 606 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19991/console |
| Powered by |
[jira] [Updated] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7654:
Attachment: YARN-7654.003.patch
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401357#comment-16401357
]
Eric Yang commented on YARN-7654:
-
- Fixed the concurrency issue, and added exit code check. The new execv
approach will work for existing mode as well as entry_point mode. Fixing
add_to_buffer mechanism next. I think I will keep test cases using char*
buffer, and having a intermediate concatenation function to convert char** to
char* to avoid unnecessary test change.
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7221) Add security check for privileged docker container
[ https://issues.apache.org/jira/browse/YARN-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401370#comment-16401370 ] Eric Yang commented on YARN-7221: - [~ebadger] Good points, I will update the patch to reflect the required changes. Thank you for the review. > Add security check for privileged docker container > -- > > Key: YARN-7221 > URL: https://issues.apache.org/jira/browse/YARN-7221 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0, 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7221.001.patch, YARN-7221.002.patch, > YARN-7221.003.patch, YARN-7221.004.patch, YARN-7221.005.patch, > YARN-7221.006.patch > > > When a docker is running with privileges, majority of the use case is to have > some program running with root then drop privileges to another user. i.e. > httpd to start with privileged and bind to port 80, then drop privileges to > www user. > # We should add security check for submitting users, to verify they have > "sudo" access to run privileged container. > # We should remove --user=uid:gid for privileged containers. > > Docker can be launched with --privileged=true, and --user=uid:gid flag. With > this parameter combinations, user will not have access to become root user. > All docker exec command will be drop to uid:gid user to run instead of > granting privileges. User can gain root privileges if container file system > contains files that give user extra power, but this type of image is > considered as dangerous. Non-privileged user can launch container with > special bits to acquire same level of root power. Hence, we lose control of > which image should be run with --privileges, and who have sudo rights to use > privileged container images. As the result, we should check for sudo access > then decide to parameterize --privileged=true OR --user=uid:gid. This will > avoid leading developer down the wrong path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401367#comment-16401367 ] Eric Yang commented on YARN-8018: - [~csingh] Yes, I am in favor of option 1 and agree that offline mode would take over priority over AM mode. > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.wip.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
[ https://issues.apache.org/jira/browse/YARN-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8032: Attachment: YARN-8032.002.patch > Yarn service should expose failuresValidityInterval to users and use it for > launching containers > > > Key: YARN-8032 > URL: https://issues.apache.org/jira/browse/YARN-8032 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8032.001.patch, YARN-8032.002.patch > > > With YARN-5015 the support for sliding window retry policy was added. Yarn > service should expose it via the api for the users to take advantage of it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401323#comment-16401323
]
Chandni Singh commented on YARN-7512:
-
{quote}I was just questioning whether the version should appear in the file
name of the file that is saved to HDFS.
{quote}
Since we are not restricting upgrade while another upgrade is in progress,
there will be conflicts during writing to the existing file from rest service
while Service AM is reading the file to figure out which components need
upgrade. YARN-8018 has these changes to initializing service upgrade. Please,
let me know if there are problems with the approach on YARN-8018.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
[
https://issues.apache.org/jira/browse/YARN-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401321#comment-16401321
]
genericqa commented on YARN-8032:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 1 new + 46 unchanged - 0 fixed = 47 total (was 46) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
40s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-8032 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914797/YARN-8032.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c0cfe73a11b4 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4bf6220 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19990/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19990/testReport/ |
| Max.
[jira] [Commented] (YARN-7523) Introduce description and version field in Service record
[ https://issues.apache.org/jira/browse/YARN-7523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401316#comment-16401316 ] Sergey Shelukhin commented on YARN-7523: I understand YS is not released yet, so technically nobody should be using it... but for future is it possible to make these changes in backward compatible manner to avoid everyone having to update after every release (or after every patch in this case, if dogfooding YS)? cc [~gsaha] > Introduce description and version field in Service record > - > > Key: YARN-7523 > URL: https://issues.apache.org/jira/browse/YARN-7523 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Critical > Fix For: 3.1.0 > > Attachments: YARN-7523.001.patch, YARN-7523.002.patch, > YARN-7523.003.patch, YARN-7523.004.patch > > > YARN-7512 would need version field in Service record. It would be good to > introduce a description field also to allow service owners to capture some > details which can be used to display in Service catalog as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401312#comment-16401312 ] Billie Rinaldi commented on YARN-8018: -- I agree, it makes sense for the AM to write the service definition during flex. I guess the only thing to look into would be whether we are currently supporting flex while the service is stopped. In that case, we might want to allow the client to persist the service definition if the service is stopped, but not if the AM is running. > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.wip.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401298#comment-16401298
]
Billie Rinaldi commented on YARN-7512:
--
bq. if there are only two versions allowed at a time, perhaps we shouldn't
include the version number in the upgrade service json, instead calling it
something like {service_name}_upgrade.json or {service_name}_next.json
Sorry, this comment I made was really unclear. I completely agree that version
should appear as a field in the json, I was just questioning whether the
version should appear in the file name of the file that is saved to HDFS.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
[ https://issues.apache.org/jira/browse/YARN-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401276#comment-16401276 ] Chandni Singh commented on YARN-8032: - [~gsaha] [~billie.rinaldi] [~leftnoteasy] It is a simple patch. Could you please help review? > Yarn service should expose failuresValidityInterval to users and use it for > launching containers > > > Key: YARN-8032 > URL: https://issues.apache.org/jira/browse/YARN-8032 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8032.001.patch > > > With YARN-5015 the support for sliding window retry policy was added. Yarn > service should expose it via the api for the users to take advantage of it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
[ https://issues.apache.org/jira/browse/YARN-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8032: Attachment: YARN-8032.001.patch > Yarn service should expose failuresValidityInterval to users and use it for > launching containers > > > Key: YARN-8032 > URL: https://issues.apache.org/jira/browse/YARN-8032 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8032.001.patch > > > With YARN-5015 the support for sliding window retry policy was added. Yarn > service should expose it via the api for the users to take advantage of it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
[ https://issues.apache.org/jira/browse/YARN-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8032: Description: With YARN-5015 the support for sliding window retry policy was added. Yarn service should expose it via the api for the users to take advantage of it. > Yarn service should expose failuresValidityInterval to users and use it for > launching containers > > > Key: YARN-8032 > URL: https://issues.apache.org/jira/browse/YARN-8032 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > > With YARN-5015 the support for sliding window retry policy was added. Yarn > service should expose it via the api for the users to take advantage of it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8033) CLI Integration with NodeAttributesManagerImpl
[ https://issues.apache.org/jira/browse/YARN-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-8033: Attachment: YARN-8033-YARN-3409.WIP.patch > CLI Integration with NodeAttributesManagerImpl > -- > > Key: YARN-8033 > URL: https://issues.apache.org/jira/browse/YARN-8033 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Attachments: YARN-8033-YARN-3409.WIP.patch > > > In continuation with YARN-6856 & YARN-6858, need to integrate CLI to the > NodeAttributeManager to complete the functionality of Centralised mapping of > Node Attributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8033) CLI Integration with NodeAttributesManagerImpl
Naganarasimha G R created YARN-8033: --- Summary: CLI Integration with NodeAttributesManagerImpl Key: YARN-8033 URL: https://issues.apache.org/jira/browse/YARN-8033 Project: Hadoop YARN Issue Type: Sub-task Reporter: Naganarasimha G R Assignee: Naganarasimha G R In continuation with YARN-6856 & YARN-6858, need to integrate CLI to the NodeAttributeManager to complete the functionality of Centralised mapping of Node Attributes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
Chandni Singh created YARN-8032: --- Summary: Yarn service should expose failuresValidityInterval to users and use it for launching containers Key: YARN-8032 URL: https://issues.apache.org/jira/browse/YARN-8032 Project: Hadoop YARN Issue Type: Bug Reporter: Chandni Singh Assignee: Chandni Singh -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8016) Refine PlacementRule interface and add a app-name queue mapping rule as an example
[ https://issues.apache.org/jira/browse/YARN-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401206#comment-16401206 ] Zian Chen commented on YARN-8016: - [~yufeigu] , thank you Yufei, I'll work on refactoring the patch. > Refine PlacementRule interface and add a app-name queue mapping rule as an > example > -- > > Key: YARN-8016 > URL: https://issues.apache.org/jira/browse/YARN-8016 > Project: Hadoop YARN > Issue Type: Task >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Attachments: YARN-8016.001.patch > > > After YARN-3635/YARN-6689, PlacementRule becomes a common interface which can > be used by scheduler and can be dynamically updated by scheduler according to > configs. There're some other works. > - There's no way to initialize PlacementRule. > - No example of PlacementRule except the user-group mapping one. > This JIRA is targeted to refine PlacementRule interfaces and add another > PlacementRule example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401192#comment-16401192
]
Eric Yang commented on YARN-7654:
-
[~jlowe] Thank you for the review. This patch requires YARN-7221 to apply. As
you can see the struggle with flipping code for execv, this is the reason that
this patch takes a long time to develop. If we want to separate execv call
from getting a working version, then patch 001 would be reasonable to commit.
However, my current workspace is deeply committed to get execv working and I
don't want to get another CVE to add to my list of Hadoop security problems.
Therefore, I am likely to stay on course to finish execv version. I also found
that we only did docker inspect once to get pid. There is a fairly high chance
that docker inspect isn't ready and no pid would be produced. I have refactor
the code to loop through docker inspect n times before giving up. This
property needs to be configurable or detect the first child process has exited.
Docker image download can take sometime, and the inspect command might need a
lot of retires before giving up. Shane provided some good technique to check
proc in existing code. I should be able to reuse it for detection part.
{quote}
Why does the code not check for an execv error and instead exits with a success
code? It should log the strerror of the execv failure and exit with a failure
code instead.
{quote}
Good point. This will be fixed in the next version. Thanks again.
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8006) Make Hbase-2 profile as default for YARN-7055 branch
[ https://issues.apache.org/jira/browse/YARN-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401167#comment-16401167 ] Haibo Chen commented on YARN-8006: -- Talked with [~rkanter] about this issue. He suggested us to check if yetus/maven does something funny (because of the hbase version change) by going ahead commit this patch and submitting a dummy patch to trigger unit tests. Taking his idea, I tried that locally with dev-support/bin/test-patch. The unit tests did not fail with NoSuchMethodError any more (failed with the filter-related unit tests which are expected). I'm +1 on committing YARN-8006-YARN-7055.001.patch with local yetus test even though the jenkins job is failing. > Make Hbase-2 profile as default for YARN-7055 branch > > > Key: YARN-8006 > URL: https://issues.apache.org/jira/browse/YARN-8006 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-8006-YARN-7055.001.patch, yetus-run.tar.gz > > > In last weekly call folks discussed that we should have separate branch with > hbase-2 as profile by default. Trunk default profile is hbase-1 which runs > all the tests under hbase-1 profile. But for hbase-2 profile tests are not > running. > As per the discussion, lets keep YARN-7055 branch for hbase-2 profile as > default. Any server side patches can be given to this branch as well which > runs tests for hbase-2 profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401162#comment-16401162 ] Chandni Singh commented on YARN-8018: - [~eyang], are you in favor of approach 1, that is, we move the existing code that overwrites service definition during flex to AM? I think that is a better approach as well. > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.wip.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7221) Add security check for privileged docker container
[
https://issues.apache.org/jira/browse/YARN-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401161#comment-16401161
]
Eric Badger commented on YARN-7221:
---
bq. Eric Badger Are you running sudo -U ebadger -n -l docker as root user or
yarn? When container-executor runs this check, it is using root privileges to
check, thus password prompt is omitted. The sudo session doesn't exist beyond
the check.
I was running as ebadger, which is why I saw that. Makes sense on why I was
seeing what I saw.
{noformat}
+ char tmpl[] = "id -G -n %s";
+ char buffer[4096];
+ if (fork()==0) {
+char *cmd = (char *) alloc_and_clear_memory(strlen(tmpl) + strlen(user),
sizeof(char));
+sprintf(cmd, tmpl, user);
{noformat}
Is there a reason for tmpl? It doesn't seem to be necessary here. We can just
put it into the sprintf. And even more, is there a reason we can't use
{{getgroups()}} instead of calling {{id}}? Seems unnecessary to shell out the
call to {{id}} that we can do in C land.
{noformat}
+if (fp == NULL) {
+ exit(127);
+}
{noformat}
Missing a free for {{cmd}} here.
{noformat}
+if (strcmp(token, "root")==0 || strcmp(token, "docker")==0) {
+ pclose(fp);
+ free(cmd);
{noformat}
Missing a free for {{token}} here.
{noformat}
+wait();
+if (WIFEXITED(statval)) {
+ if (WEXITSTATUS(statval)==0) {
+return 1;
+ }
+}
+ }
+ return 0;
{noformat}
Since returning 1 is "success" in this case, I think a comment might be useful.
Just a simple "//success" or something like that, since returning 1 usually
implies failure when the only options are 0 and 1.
> Add security check for privileged docker container
> --
>
> Key: YARN-7221
> URL: https://issues.apache.org/jira/browse/YARN-7221
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7221.001.patch, YARN-7221.002.patch,
> YARN-7221.003.patch, YARN-7221.004.patch, YARN-7221.005.patch,
> YARN-7221.006.patch
>
>
> When a docker is running with privileges, majority of the use case is to have
> some program running with root then drop privileges to another user. i.e.
> httpd to start with privileged and bind to port 80, then drop privileges to
> www user.
> # We should add security check for submitting users, to verify they have
> "sudo" access to run privileged container.
> # We should remove --user=uid:gid for privileged containers.
>
> Docker can be launched with --privileged=true, and --user=uid:gid flag. With
> this parameter combinations, user will not have access to become root user.
> All docker exec command will be drop to uid:gid user to run instead of
> granting privileges. User can gain root privileges if container file system
> contains files that give user extra power, but this type of image is
> considered as dangerous. Non-privileged user can launch container with
> special bits to acquire same level of root power. Hence, we lose control of
> which image should be run with --privileges, and who have sudo rights to use
> privileged container images. As the result, we should check for sudo access
> then decide to parameterize --privileged=true OR --user=uid:gid. This will
> avoid leading developer down the wrong path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401158#comment-16401158
]
Chandni Singh commented on YARN-7512:
-
[~leftnoteasy]
Just some comments.
{quote}1. Talked to [~csingh], existing upgrade only supports artifact, and all
other fields will be silently ignored. Probably we should rename the "upgrade"
to something like "set-image"?
{quote}
Initially, I am focusing on upgrade based on just artifact change. The design
doc does include other fields. Once basic upgrade is in place, the plan was to
enhance the feature.
{quote}2. Regarding to existing interactions between user and native services
to do updates. IIUC, this is workflow proposed in the design doc, plz correct
me if I was wrong:
a. User upload a new upgrade spec to HDFS.
b. User call upgrade service to read the new spec.
c. User call upgrade component(s) to update components and instances.
d. User manually set finalize or YARN will automatically set finalize of
services.
{quote}
There are not 4 steps but 3 steps. a) and b) are part of initializing the
service upgrade.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7933) [atsv2 read acls] Add TimelineWriter#writeDomain
[ https://issues.apache.org/jira/browse/YARN-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401145#comment-16401145 ] Haibo Chen commented on YARN-7933: -- Is there a doc from ATSv1 (or 1.5) that describes the domain ACL model? > [atsv2 read acls] Add TimelineWriter#writeDomain > - > > Key: YARN-7933 > URL: https://issues.apache.org/jira/browse/YARN-7933 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-7933.01.patch, YARN-7933.02.patch > > > > Add an API TimelineWriter#writeDomain for writing the domain info -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401140#comment-16401140
]
Wangda Tan commented on YARN-7512:
--
Thanks [~csingh] for working on this design, a couple of questions/comments.
1) Regarding to upgrade, from my understanding, it will be more valueable to
upgrade only for component's artifact and component's replicas.
For other part, since we have environment subsistution, it is possible that a
change of global environment could cause all dependencies upgraded. For example:
{code:java}
Service:
name: my-service.
environments:
env1: value1
env2: value2
components:
component:
name: comp1
launch_command: "python $env1 ..."
artifact: $env2-x-y-z
component:
name: comp2
launch_command: "app $env2 ..."
{code}
Change of $env1/$env2 could cause all component needs to be upgraded, I'm not
sure if that is desired state.
Talked to [~csingh], existing upgrade only supports artifact, and all other
fields will be silently ignored. Probably we should rename the "upgrade" to
something like "set-image"?
2) Regarding to existing interactions between user and native services to do
updates. IIUC, this is workflow proposed in the design doc, plz correct me if I
was wrong:
a. User upload a new upgrade spec to HDFS.
b. User call upgrade service to read the new spec.
c. User call upgrade component(s) to update components and instances.
d. User manually set finalize or YARN will automatically set finalize of
services.
First of all, I'm not sure if changing service state to UPGRADING could bring
and benefits. IMHO, it is confusing to let user manual finalize an upgrade but
not all components are upgraded.
And a/b look like a burden to end user. If user needs to call c anyway, why we
need a/b.
My proposal is to keep APIs to only upgrade *artifact* of component and
instance of components:
{code:java}
Response upgradeComponentArtifact(ServiceName, ComponentName, NewArtifactName,
[LowerIndexBound, HigherIndexBound])
{code}
The {{[LowerIndexBound, HigherIndexBound]}} is useful when user want to verify
the artifact works instead of bring down all instances. By default we will
upgrade all instances of the component.
Once API server received this, this operation will be persisted, and component
transferred to UPGRADING. And once all requested instances upgraded,
component's state will be updated to READY again.
3) Several future requirement to consider:
a. What's the rollback story? It might be good to list history artifacts of
instances and timestamp. Existing design doesn't support rollback to a specific
version.
b. What's the AM restart story? From Chandni now AM doesn't persistent upgrade
status of containers. We may need to consider this.
Thoughts?
cc:[~gsaha]/[~billie.rinaldi].
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7952) RM should be able to recover log aggregation status after restart/fail-over
[ https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401137#comment-16401137 ] Hudson commented on YARN-7952: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13846 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13846/]) YARN-7952. RM should be able to recover log aggregation status after (wangda: rev 4bf622043f034835d65ff2a4785b9b06d0ef1fd2) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/logaggregation/tracker/TestNMLogAggregationStatusTracker.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/logaggregation/tracker/NMLogAggregationStatusTracker.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStartedEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > RM should be able to recover log aggregation status after restart/fail-over > --- > > Key: YARN-7952 > URL: https://issues.apache.org/jira/browse/YARN-7952 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-7952-poc.patch, YARN-7952.1.patch, > YARN-7952.2.patch, YARN-7952.3.patch, YARN-7952.3.patch, YARN-7952.5.patch, > YARN-7952.6.patch, YARN-7952.7.patch, YARN-7952.8.patch, YARN-7952.9.patch > > > Right now, the NM would send its own log aggregation status to RM > periodically to RM. And RM would aggregate the status for each application, > but it will not generate the final status until a client call(from web ui or > cli) trigger it. But RM never persists the log aggregation status. So, when > RM restarts/fails over, the log aggregation status will become “NOT_STARTED”. > This is confusing, maybe we should change it to “NOT_AVAILABLE” (will create > a separate ticket for this). Anyway, we need to persist the log aggregation > status for the future use. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401130#comment-16401130 ] Eric Yang commented on YARN-8018: - [~csingh] RM rest API is multi-threaded. There is no guarantee that request will be handled via ServiceClient in ordered fashion. It would be best to have a memory queue of size of n (default to 1) in AM for queuing the requests, and commit service definition after flex is completed. If AM goes down, then AM recovery will use the last copy on HDFS. User will need to resubmit the upgrade request again if there was something in the memory queue prior to AM crash. This will also prevent getting stuck in queued request. This also reduce complexity of logic that we run in RM. > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.wip.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8021) The -addToClusterNodeLabels command doesn't provides error message for invalid node ID
[
https://issues.apache.org/jira/browse/YARN-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401127#comment-16401127
]
Naganarasimha G R edited comment on YARN-8021 at 3/15/18 9:19 PM:
--
[~GergelyNovak] , we can use "failOnUnknownNodes" option to ensure that the
command fails if the node is invalid. we had not taken the approach which you
had mentioned because it will cause a incompatible change and earlier we wanted
to support to map a label even when the node is down or yet to be added to the
system.
Will close this Jira, if this option satisfies your requirement !
was (Author: naganarasimha):
[~GergelyNovak] , we can use "failOnUnknownNodes" option to ensure that the
command fails if the node is invalid. we had not taken the approach which you
had mentioned because it will cause a incompatible change and earlier we wanted
to support to map a label even when the node is down or yet to be added to the
system.
> The -addToClusterNodeLabels command doesn't provides error message for
> invalid node ID
> --
>
> Key: YARN-8021
> URL: https://issues.apache.org/jira/browse/YARN-8021
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Gergely Novák
>Priority: Major
>
> {noformat}
> yarn rmadmin -replaceLabelsOnNode "invalid-node-id:1234=node_label_x"
> {noformat}
> The above command doesn't return anything aside from the usual "Connecting to
> ResourceManager at..." message. It also doesn't print anything for a
> successful node label replacement, but it does throw an error for most of the
> other bad inputs (invalid port, invalid node label, invalid format).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8021) The -addToClusterNodeLabels command doesn't provides error message for invalid node ID
[
https://issues.apache.org/jira/browse/YARN-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401127#comment-16401127
]
Naganarasimha G R commented on YARN-8021:
-
[~GergelyNovak] , we can use "failOnUnknownNodes" option to ensure that the
command fails if the node is invalid. we had not taken the approach which you
had mentioned because it will cause a incompatible change and earlier we wanted
to support to map a label even when the node is down or yet to be added to the
system.
> The -addToClusterNodeLabels command doesn't provides error message for
> invalid node ID
> --
>
> Key: YARN-8021
> URL: https://issues.apache.org/jira/browse/YARN-8021
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Gergely Novák
>Priority: Major
>
> {noformat}
> yarn rmadmin -replaceLabelsOnNode "invalid-node-id:1234=node_label_x"
> {noformat}
> The above command doesn't return anything aside from the usual "Connecting to
> ResourceManager at..." message. It also doesn't print anything for a
> successful node label replacement, but it does throw an error for most of the
> other bad inputs (invalid port, invalid node label, invalid format).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8006) Make Hbase-2 profile as default for YARN-7055 branch
[ https://issues.apache.org/jira/browse/YARN-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8006: - Attachment: (was: YARN-8006-YARN-8006.00.patch) > Make Hbase-2 profile as default for YARN-7055 branch > > > Key: YARN-8006 > URL: https://issues.apache.org/jira/browse/YARN-8006 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-8006-YARN-7055.001.patch, yetus-run.tar.gz > > > In last weekly call folks discussed that we should have separate branch with > hbase-2 as profile by default. Trunk default profile is hbase-1 which runs > all the tests under hbase-1 profile. But for hbase-2 profile tests are not > running. > As per the discussion, lets keep YARN-7055 branch for hbase-2 profile as > default. Any server side patches can be given to this branch as well which > runs tests for hbase-2 profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8006) Make Hbase-2 profile as default for YARN-7055 branch
[ https://issues.apache.org/jira/browse/YARN-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8006: - Attachment: (was: YARN-8006-YARN-8006.01.patch) > Make Hbase-2 profile as default for YARN-7055 branch > > > Key: YARN-8006 > URL: https://issues.apache.org/jira/browse/YARN-8006 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-8006-YARN-7055.001.patch, yetus-run.tar.gz > > > In last weekly call folks discussed that we should have separate branch with > hbase-2 as profile by default. Trunk default profile is hbase-1 which runs > all the tests under hbase-1 profile. But for hbase-2 profile tests are not > running. > As per the discussion, lets keep YARN-7055 branch for hbase-2 profile as > default. Any server side patches can be given to this branch as well which > runs tests for hbase-2 profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401051#comment-16401051 ] Chandni Singh edited comment on YARN-8018 at 3/15/18 8:58 PM: -- I see an issue with the finalization of upgrade. In my change I introduced event based state modification to service state. So during finalize the service master overwrites the service definition when handling the restart event. However, when flex is trigger, the service definition is overwritten from the rest server. So service definition is modified by different hosts. To solve this, we have 2 approaches: # move overwriting of service definition to AM during flex. Otherwise, this is a potential problem during processing concurrent flex requests when there are multiple rest services? # I change my patch to get rid of event base state handling of service. I added it because I think this makes handling of auto-finalize much easier and less prone to bugs. Any thoughts? was (Author: csingh): I see an issue with the finalization of upgrade. In my change I introduced event based state modification to service state. So during finalize the service master overwrites the service definition when handling the restart event. However, when flex is trigger, the service definition is overwritten from the rest server. So service definition is modified by different hosts. To solve this, we have 2 approaches: # move overwriting of service definition to AM during flex. # I change my patch to get rid of event base state handling of service. I added it because I think this makes handling of auto-finalize much easier and less prone to bugs. Any thoughts? > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.wip.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7512) Support service upgrade via YARN Service API and CLI
[
https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401089#comment-16401089
]
Chandni Singh commented on YARN-7512:
-
[~billie.rinaldi]
{quote}if there are only two versions allowed at a time, perhaps we shouldn't
include the version number in the upgrade service json, instead calling it
something like \{service_name}_upgrade.json or \{service_name}_next.json. That
would also have the advantage of not restricting the characters that can appear
in the version string.
{quote}
My thoughts regarding this:
# Users should be able to specify their versions of a service. The version
could be representative of the artifact of the application that they are
running as a service.
# We will allow abort/upgrade while another upgrade is in progress. I think it
is important to have a version field in service json to let the user know which
version of their service is running.
> Support service upgrade via YARN Service API and CLI
>
>
> Key: YARN-7512
> URL: https://issues.apache.org/jira/browse/YARN-7512
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Gour Saha
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
> Attachments: _In-Place Upgrade of Long-Running Applications in
> YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf
>
>
> YARN Service API and CLI needs to support service (and containers) upgrade in
> line with what Slider supported in SLIDER-787
> (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-7512: Attachment: _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Gour Saha >Assignee: Chandni Singh >Priority: Major > Fix For: yarn-native-services > > Attachments: _In-Place Upgrade of Long-Running Applications in > YARN_v1.pdf, _In-Place Upgrade of Long-Running Applications in YARN_v2.pdf > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401051#comment-16401051 ] Chandni Singh commented on YARN-8018: - I see an issue with the finalization of upgrade. In my change I introduced event based state modification to service state. So during finalize the service master overwrites the service definition when handling the restart event. However, when flex is trigger, the service definition is overwritten from the rest server. So service definition is modified by different hosts. To solve this, we have 2 approaches: # move overwriting of service definition to AM during flex. # I change my patch to get rid of event base state handling of service. I added it because I think this makes handling of auto-finalize much easier and less prone to bugs. Any thoughts? > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.wip.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[
https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400958#comment-16400958
]
genericqa commented on YARN-7581:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-7581 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914758/YARN-7581.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2ec339397616 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1976e00 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19989/testReport/ |
| Max. process+thread count | 440 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
|
| Console
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400857#comment-16400857 ] Haibo Chen commented on YARN-7581: -- 04 patch to address the checkstyle issue. > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581.00.patch, YARN-7581.01.patch, > YARN-7581.02.patch, YARN-7581.03.patch, YARN-7581.04.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-7581: - Attachment: YARN-7581.04.patch > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581.00.patch, YARN-7581.01.patch, > YARN-7581.02.patch, YARN-7581.03.patch, YARN-7581.04.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[
https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400832#comment-16400832
]
genericqa commented on YARN-7581:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 9s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 31s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-7581 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914739/YARN-7581.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3f19f3823e16 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5e013d5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19988/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client.txt
|
| Test Results |
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[
https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400777#comment-16400777
]
Chandni Singh commented on YARN-8018:
-
{quote}I think the version number would be better to be a integer and
auto-incremented when actionUpgrade is triggered
{quote}
There are some conflicting views about this. Should I create another Jira and
we can discuss it over there?
> Yarn service: Add support for initiating service upgrade
>
>
> Key: YARN-8018
> URL: https://issues.apache.org/jira/browse/YARN-8018
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-8018.wip.patch
>
>
> Add support for initiating service upgrade which includes the following main
> changes:
> # Service API to initiate upgrade
> # Persist service version on hdfs
> # Start the upgraded version of service
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8018) Yarn service: Add support for initiating service upgrade
[
https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400777#comment-16400777
]
Chandni Singh edited comment on YARN-8018 at 3/15/18 5:30 PM:
--
{quote}I think the version number would be better to be a integer and
auto-incremented when actionUpgrade is triggered
{quote}
There are some conflicting views about this. Should I create another Jira for
this and we can discuss it over there?
was (Author: csingh):
{quote}I think the version number would be better to be a integer and
auto-incremented when actionUpgrade is triggered
{quote}
There are some conflicting views about this. Should I create another Jira and
we can discuss it over there?
> Yarn service: Add support for initiating service upgrade
>
>
> Key: YARN-8018
> URL: https://issues.apache.org/jira/browse/YARN-8018
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-8018.wip.patch
>
>
> Add support for initiating service upgrade which includes the following main
> changes:
> # Service API to initiate upgrade
> # Persist service version on hdfs
> # Start the upgraded version of service
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400745#comment-16400745 ] Haibo Chen commented on YARN-7581: -- Updated the patch to address Rohith's comments. > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581.00.patch, YARN-7581.01.patch, > YARN-7581.02.patch, YARN-7581.03.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-7581: - Attachment: YARN-7581.03.patch > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581.00.patch, YARN-7581.01.patch, > YARN-7581.02.patch, YARN-7581.03.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400691#comment-16400691 ] Eric Yang commented on YARN-8018: - The current approach looks good. Unit tests ran fine. We need to add to hadoop-yarn-services-api to have a REST API and hadoop-yarn-client CLI to perform upgrade. I think the version number would be better to be a integer and auto-incremented when actionUpgrade is triggered. > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.wip.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400676#comment-16400676
]
Jason Lowe commented on YARN-7654:
--
Thanks for the patch! Switching the code to execv docker instead of popen
docker is going to be a fairly extensive change that isn't directly tied to
supporting an entry point. Would it make more sense to track the execv effort
as a separate JIRA?
What base should be used for the patch? I couldn't apply it to trunk,
branch-3.1, or branch-3.0.
bq. I am currently running into a problem that after docker is started, the
existing logic for detecting docker exit is not working.
I couldn't tell from the patch what the problem would be. If I can get the
patch applied and some more details on how it behaves when it doesn't work I
may be able to help debug.
bq. I am concerned that we are using popen for docker launch for non-entry
point version, it actually spawn a shell underneath.
Yes, popen opens up the possibility of the shell doing us "favors" that we
really do not want. It would be safer to avoid it.
bq. We might need to convert the output buffer from char array to char pointer
array to avoid chopping strings in the middle.
This is a necessity. By serializing multiple values into a String and then
needing to rediscover the individual values later, we're opening up the
possibility that the arguments will be mishandled based on the contents of the
arguments accidentally looking like separator characters. That's one of the
many dangers of running the shell instead of execv directly which we're trying
to avoid. If we're going to be serious about leveraging execv for security and
correctness in light of untrusted data in the arguments then building up the
command line for execv shouldn't be serialzing and deserializing through String.
As a bonus for going through execv directly, the whole quoting and appending
and sanitizing argument logic seems unnecessary at that point. We can just
pass these values straight through to the docker command for interpretation by
Docker rather than worrying about how to quote them properly so the shell
doesn't butcher them on the way through.
Why does the code not check for an execv error and instead exits with a success
code? It should log the strerror of the execv failure and exit with a failure
code instead.
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8006) Make Hbase-2 profile as default for YARN-7055 branch
[ https://issues.apache.org/jira/browse/YARN-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400641#comment-16400641 ] Haibo Chen commented on YARN-8006: -- I was able to get 'dev-support/bin/test-patch YARN-8006-YARN-7055.01.patch' to execute my local copy of yetus by setting YETUS_HOME. From there, I enabled maven debug messages at compile. Attached is the all the artifacts generated by my local yetus run. My own analysis of the artifacts is: 1) timelineservice-hbase-client is compiled against hbase-client-2.0.0-beta1 and hbase-client.2.0.0-beta jar is provided in classpath at unit test execution time. 2) there is no mention of hbase 1.2.6 after the patch is applied. You can use yetus-console.txt as an index to see what yetus did at each stage and where to find the output file for each stage. The output files are bundled in yetus-output.tar.gz. Also attached are surefire-report and the jars that are said by maven to be loaded at unit test time. Depending on how important YARN-7055 is to us, I wonder if we want to try to remove the conditional-compilation related profile in maven pom.xml, if that is acceptable. > Make Hbase-2 profile as default for YARN-7055 branch > > > Key: YARN-8006 > URL: https://issues.apache.org/jira/browse/YARN-8006 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-8006-YARN-7055.001.patch, > YARN-8006-YARN-8006.00.patch, YARN-8006-YARN-8006.01.patch, yetus-run.tar.gz > > > In last weekly call folks discussed that we should have separate branch with > hbase-2 as profile by default. Trunk default profile is hbase-1 which runs > all the tests under hbase-1 profile. But for hbase-2 profile tests are not > running. > As per the discussion, lets keep YARN-7055 branch for hbase-2 profile as > default. Any server side patches can be given to this branch as well which > runs tests for hbase-2 profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8006) Make Hbase-2 profile as default for YARN-7055 branch
[ https://issues.apache.org/jira/browse/YARN-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8006: - Attachment: yetus-run.tar.gz > Make Hbase-2 profile as default for YARN-7055 branch > > > Key: YARN-8006 > URL: https://issues.apache.org/jira/browse/YARN-8006 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-8006-YARN-7055.001.patch, > YARN-8006-YARN-8006.00.patch, YARN-8006-YARN-8006.01.patch, yetus-run.tar.gz > > > In last weekly call folks discussed that we should have separate branch with > hbase-2 as profile by default. Trunk default profile is hbase-1 which runs > all the tests under hbase-1 profile. But for hbase-2 profile tests are not > running. > As per the discussion, lets keep YARN-7055 branch for hbase-2 profile as > default. Any server side patches can be given to this branch as well which > runs tests for hbase-2 profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6936) [Atsv2] Retrospect storing entities into sub application table from client perspective
[ https://issues.apache.org/jira/browse/YARN-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S reassigned YARN-6936: --- Assignee: Rohith Sharma K S > [Atsv2] Retrospect storing entities into sub application table from client > perspective > -- > > Key: YARN-6936 > URL: https://issues.apache.org/jira/browse/YARN-6936 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-6936.000.patch > > > Currently YARN-6734 stores entities into sub application table only if doAs > user and submitted users are different. This holds good for Tez kind of use > cases. But AM runs as same as submitted user like MR also need to store > entities in sub application table so that it could read entities without > application id. > This would be a point of concern later stages when ATSv2 is deployed into > production. This JIRA is to retrospect decision of storing entities into sub > application table based on client side configuration driven rather than user > driven. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2571) RM to support YARN registry
[ https://issues.apache.org/jira/browse/YARN-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400255#comment-16400255 ] ASF GitHub Bot commented on YARN-2571: -- Github user steveloughran closed the pull request at: https://github.com/apache/hadoop/pull/66 > RM to support YARN registry > > > Key: YARN-2571 > URL: https://issues.apache.org/jira/browse/YARN-2571 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: oct16-hard > Attachments: YARN-2571-001.patch, YARN-2571-002.patch, > YARN-2571-003.patch, YARN-2571-005.patch, YARN-2571-007.patch, > YARN-2571-008.patch, YARN-2571-009.patch, YARN-2571-010.patch, > YARN-2571-012.patch, YARN-2571-013.patch, YARN-2571-015.patch, > YARN-2571-016.patch, YARN-2571-017.patch > > > The RM needs to (optionally) integrate with the YARN registry: > # startup: create the /services and /users paths with system ACLs (yarn, hdfs > principals) > # app-launch: create the user directory /users/$username with the relevant > permissions (CRD) for them to create subnodes. > # attempt, container, app completion: remove service records with the > matching persistence and ID -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8031) NodeManager will fail to start if cpu subsystem is already mounted
JayceAu created YARN-8031: - Summary: NodeManager will fail to start if cpu subsystem is already mounted Key: YARN-8031 URL: https://issues.apache.org/jira/browse/YARN-8031 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.5.0 Reporter: JayceAu Attachments: image-2018-03-15-14-47-30-583.png if *yarn.nodemanager.linux-container-executor.cgroups.mount* is set to true and cpu subsystem is not yet mounted, NodeManager will mount the cpu subsystem and then create the control group whose default name is *hadoop-yarn* if the mount step is successful. This procedure works well if cpu subsystem is not yet mounted. However, under some situation cpu subsystem is already mounted before NodeManager starts and NodeManager will fail to start because of no write permission to the *hadoop-yarn* path . For example: # in OS that use systemd such as centos7 will have cpu subsystem mounted by default on machine startup # some deamon whose start order is more precedent than NodeManager may also rely on the mounted state of cpu subsystem. In our production environment, we limit the cpu usage of the monitoring and control agent, which starts on reboot In order to solve this problem, container-executor must be able to create the control group *hadoop-yarn* if mounting controller is successful or this controller is already mounted. Besides, if cpu subsystem is used in combination with other subsystem and it's already mounted, container-executor should use the latest mount point of cpu subsystem instread of the one provided by NodeManager. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7905) Parent directory permission incorrect during public localization
[
https://issues.apache.org/jira/browse/YARN-7905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400040#comment-16400040
]
Bibin A Chundatt commented on YARN-7905:
Minor comment from my side
{{getMockedResource}} path used need not be testcase related.
{code}
2593Path newpath = new Path(
2594System.getProperty("test.build.data") +
"/testPublicCacheDirPermission",
2595name);
{code}
Others looks good to me.
> Parent directory permission incorrect during public localization
> -
>
> Key: YARN-7905
> URL: https://issues.apache.org/jira/browse/YARN-7905
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bilwa S T
>Priority: Critical
> Attachments: YARN-7905-001.patch, YARN-7905-002.patch,
> YARN-7905-003.patch, YARN-7905-004.patch, YARN-7905-005.patch
>
>
> Similar to YARN-6708 during public localization also we have to take care for
> parent directory if the umask is 027 during node manager start up.
> /filecache/0/200
> the directory permission of /filecache/0 is 750. Which cause
> application failure
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7952) RM should be able to recover log aggregation status after restart/fail-over
[ https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399973#comment-16399973 ] Xuan Gong commented on YARN-7952: - findbug warning and testcase failure are not related > RM should be able to recover log aggregation status after restart/fail-over > --- > > Key: YARN-7952 > URL: https://issues.apache.org/jira/browse/YARN-7952 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Attachments: YARN-7952-poc.patch, YARN-7952.1.patch, > YARN-7952.2.patch, YARN-7952.3.patch, YARN-7952.3.patch, YARN-7952.5.patch, > YARN-7952.6.patch, YARN-7952.7.patch, YARN-7952.8.patch, YARN-7952.9.patch > > > Right now, the NM would send its own log aggregation status to RM > periodically to RM. And RM would aggregate the status for each application, > but it will not generate the final status until a client call(from web ui or > cli) trigger it. But RM never persists the log aggregation status. So, when > RM restarts/fails over, the log aggregation status will become “NOT_STARTED”. > This is confusing, maybe we should change it to “NOT_AVAILABLE” (will create > a separate ticket for this). Anyway, we need to persist the log aggregation > status for the future use. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
