[jira] [Commented] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288522#comment-15288522
]
Varun Saxena commented on YARN-5093:
Yes. Write it out only if its > 0.
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4887) AM-RM protocol changes for supporting delta protocol
[
https://issues.apache.org/jira/browse/YARN-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288519#comment-15288519
]
Hadoop QA commented on YARN-4887:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 15s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api generated
10 new + 5406 unchanged - 0 fixed = 5416 total (was 5406) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804574/YARN-4887-v1.patch |
| JIRA Issue | YARN-4887 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 9abeebb6afd2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8a9ecb7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| javadoc | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api:
https://builds.apache.org/job/PreCom
[jira] [Commented] (YARN-5107) TestContainerMetrics fails
[
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288517#comment-15288517
]
Hadoop QA commented on YARN-5107:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 17s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804542/YARN-5107.01.patch |
| JIRA Issue | YARN-5107 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8127ec64e278 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8a9ecb7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/11522/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11522/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: YARN-5107.01.patch
>
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.022
[jira] [Commented] (YARN-5096) timelinereader has a lot of logging that's not useful
[
https://issues.apache.org/jira/browse/YARN-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288490#comment-15288490
]
Varun Saxena commented on YARN-5096:
+1. Will commit it shortly.
> timelinereader has a lot of logging that's not useful
> -
>
> Key: YARN-5096
> URL: https://issues.apache.org/jira/browse/YARN-5096
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5096-YARN-2928.01.patch
>
>
> After running about a dozen or so requests, the timelinereader log is filled
> with the following logging entries:
> {noformat}
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> {noformat}
> There were some ~ 3,000 such logging entries. It's too excessive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-5038) [YARN-3368] Application and Container pages shows wrong values when RM is stopped
[ https://issues.apache.org/jira/browse/YARN-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reopened YARN-5038: -- > [YARN-3368] Application and Container pages shows wrong values when RM is > stopped > - > > Key: YARN-5038 > URL: https://issues.apache.org/jira/browse/YARN-5038 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G > Fix For: YARN-3368 > > > Few minor issues to fix. > - In Applications page, "Running Container" is shows as -1 when app is > finished. > - In container page, "Finished Time" is showing 1970 as date by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5000) [YARN-3368] App attempt page is not loading when timeline server is not started
[ https://issues.apache.org/jira/browse/YARN-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288470#comment-15288470 ] Wangda Tan commented on YARN-5000: -- Committed with YARN-4515. (72151c24452cfef250f7ce6f457dec6ada2cbd26). Thanks [~sunilg]! > [YARN-3368] App attempt page is not loading when timeline server is not > started > --- > > Key: YARN-5000 > URL: https://issues.apache.org/jira/browse/YARN-5000 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G > Fix For: YARN-3368 > > Attachments: 0001-YARN-5000.patch, > AppFinishedAndNoTimelineServer.png, AppRunningAndNoTimelineServer.png, > AppRunningAndNoTimelineServer_v2.png, YARN-5000-YARN-3368.1.patch, > YARN-5000-YARN-3368.2.patch, YARN-5000-YARN-3368.3.patch, > YARN-5000-YARN-3368.4.patch, YARN-5000-YARN-3368.5.patch, screenshot-1.png > > > If timeline server is not started, app attempt page is not getting loaded. > In new web-ui, yarnContainer route is tightly coupled with both RM and > Timeline server. And if one of server is not up, page will not load. If > timeline server is not up, container information from RM is to be displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5038) [YARN-3368] Application and Container pages shows wrong values when RM is stopped
[ https://issues.apache.org/jira/browse/YARN-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved YARN-5038. -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: YARN-3368 Committed with YARN-4515. (72151c24452cfef250f7ce6f457dec6ada2cbd26). Thanks [~sunilg]! > [YARN-3368] Application and Container pages shows wrong values when RM is > stopped > - > > Key: YARN-5038 > URL: https://issues.apache.org/jira/browse/YARN-5038 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G > Fix For: YARN-3368 > > > Few minor issues to fix. > - In Applications page, "Running Container" is shows as -1 when app is > finished. > - In container page, "Finished Time" is showing 1970 as date by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4515) [YARN-3368] Support hosting web UI framework inside YARN RM
[ https://issues.apache.org/jira/browse/YARN-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288465#comment-15288465 ] Wangda Tan commented on YARN-4515: -- Thanks [~sunilg], latest patch looks good to me, committing. > [YARN-3368] Support hosting web UI framework inside YARN RM > --- > > Key: YARN-4515 > URL: https://issues.apache.org/jira/browse/YARN-4515 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Sunil G > Attachments: 0001-YARN-4515.patch, YARN-4515-YARN-3368.1.patch, > YARN-4515-YARN-3368.2.patch, YARN-4515-YARN-3368.3.patch, > YARN-4515-YARN-3368.4.patch, YARN-4515-YARN-3368.5.patch, > YARN-4515-YARN-3368.6.patch, YARN-4515-YARN-3368.7.patch, > preliminary-YARN-4515-host_rm_web_ui_v2.patch > > > Currently it can be only launched outside of YARN, we should make it runnable > inside YARN for easier testing and we should have a configuration to > enable/disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5108) TestFileSystemApplicationHistoryStore fails
[
https://issues.apache.org/jira/browse/YARN-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288462#comment-15288462
]
Brahma Reddy Battula commented on YARN-5108:
It's dupe of HADOOP-13140..
> TestFileSystemApplicationHistoryStore fails
> ---
>
> Key: YARN-5108
> URL: https://issues.apache.org/jira/browse/YARN-5108
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Xuan Gong
>
> Unit test failure is found in other JIRA Jenkins test (YARN-5100):
> {noformat}
> Running
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore
> Tests run: 8, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 20.188 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore
> testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
> Time elapsed: 0.054 sec <<< ERROR!
> java.lang.NullPointerException: null
> at java.util.TreeMap.getEntry(TreeMap.java:347)
> at java.util.TreeMap.get(TreeMap.java:278)
> at
> org.apache.hadoop.fs.GlobalStorageStatistics.put(GlobalStorageStatistics.java:73)
> at org.apache.hadoop.fs.FileSystem.getStatistics(FileSystem.java:3598)
> at org.apache.hadoop.fs.FileSystem.initialize(FileSystem.java:214)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:101)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.setup(TestFileSystemApplicationHistoryStore.java:64)
> testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
> Time elapsed: 0.054 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.tearDown(TestFileSystemApplicationHistoryStore.java:89)
> testInitNonExistingWorkingDirectoryInSafeMode(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
> Time elapsed: 0.517 sec <<< FAILURE!
> org.mockito.exceptions.verification.WantedButNotInvoked:
> Wanted but not invoked:
> rawLocalFileSystem.isDirectory();
> -> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5096) timelinereader has a lot of logging that's not useful
[
https://issues.apache.org/jira/browse/YARN-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-5096:
--
Attachment: YARN-5096-YARN-2928.01.patch
Posted the patch. Could you please take a quick look and commit this? Thanks!
> timelinereader has a lot of logging that's not useful
> -
>
> Key: YARN-5096
> URL: https://issues.apache.org/jira/browse/YARN-5096
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5096-YARN-2928.01.patch
>
>
> After running about a dozen or so requests, the timelinereader log is filled
> with the following logging entries:
> {noformat}
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> {noformat}
> There were some ~ 3,000 such logging entries. It's too excessive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5096) timelinereader has a lot of logging that's not useful
[
https://issues.apache.org/jira/browse/YARN-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-5096:
--
Description:
After running about a dozen or so requests, the timelinereader log is filled
with the following logging entries:
{noformat}
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
{noformat}
There were some ~ 3,000 such logging entries. It's too excessive.
was:
After running about a dozen or so requests, the timelinereader log is filled
with the following logging entries:
{noformat}
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
2016-05-16 15:59:13,364 INFO
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: null
prefix was specified; returning all columns
{noformat}
There were some ~ 3,000 such logging entries. It's too excessive.
Also, when I requested YARN_CONTAINER with fields=ALL, I see the following logs:
{noformat}
WARN
org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
incorrectly formatted column name: it will be discarded
{noformat}
> timelinereader has a lot of logging that's not useful
> -
>
> Key: YARN-5096
> URL: https://issues.apache.org/jira/browse/YARN-5096
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Labels: yarn-2928-1st-milestone
>
> After running about a dozen or so requests, the timelinereader log is filled
> with the following logging entries:
> {noformat}
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn
[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288444#comment-15288444
]
Sangjin Lee commented on YARN-5109:
---
For example, code that creates the column name for events:
{code}
byte[] eventTs =
Bytes.toBytes(TimelineStorageUtils.invertLong(eventTimestamp));
...
byte[] compoundColumnQualifierBytes =
EntityColumnPrefix.EVENT.
getCompoundColQualBytes(eventId, eventTs, null);
...
public static byte[] getCompoundColumnQualifierBytes(String qualifier,
byte[]...components) {
byte[] colQualBytes = Bytes.toBytes(Separator.VALUES.encode(qualifier));
for (int i = 0; i < components.length; i++) {
colQualBytes = Separator.VALUES.join(colQualBytes, components[i]);
}
return colQualBytes;
}
{code}
The {{getCompoundColumnQualifierBytes()}} method uses the bytes from the
timestamp as is without any encoding for VALUES ({{\x3d}}).
I believe a similar issue exists with row keys. In most cases, long's are
passed to the row key without any encoding for QUALIFIERS. If any of the byte
values happens to be QUALIFIERS ({{\x21}}), it will cause the row key parsing
to fail.
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5109) timestamps are stored unencoded causing parse errors
Sangjin Lee created YARN-5109:
-
Summary: timestamps are stored unencoded causing parse errors
Key: YARN-5109
URL: https://issues.apache.org/jira/browse/YARN-5109
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: YARN-2928
Reporter: Sangjin Lee
Priority: Blocker
When we store timestamps (for example as part of the row key or part of the
column name for an event), the bytes are used as is without any encoding. If
the byte value happens to contain a separator character we use (e.g. "!" or
"="), it causes a parse failure when we read it.
I came across this while looking into this error in the timeline reader:
{noformat}
2016-05-17 21:28:38,643 WARN
org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
incorrectly formatted column name: it will be discarded
{noformat}
I traced the data that was causing this, and the column name (for the event)
was the following:
{noformat}
i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
{noformat}
Note that the column name is supposed to be of the format (event
id)=(timestamp)=(event info key). However, observe the timestamp portion:
{noformat}
\x7F\xFF\xFE\xABDY=\x99
{noformat}
The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288193#comment-15288193 ] Allen Wittenauer commented on YARN-4757: Also, why is this a YARN JIRA? It seems pretty logical that there are other services that could benefit from discovery never mind HDFS-10421. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288191#comment-15288191 ] Allen Wittenauer commented on YARN-4757: So rather than add these capabilities to an existing, tested solution, we're going to build our own > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4844) Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource
[ https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4844: - Attachment: YARN-4844.9.branch Attached ver.9 patch, rebased to latest trunk/branch-2 and fixed build failures. > Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource > > > Key: YARN-4844 > URL: https://issues.apache.org/jira/browse/YARN-4844 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-4844.1.patch, YARN-4844.2.patch, YARN-4844.3.patch, > YARN-4844.4.patch, YARN-4844.5.patch, YARN-4844.6.patch, YARN-4844.7.patch, > YARN-4844.8.branch-2.patch, YARN-4844.8.patch, YARN-4844.9.branch, > YARN-4844.9.branch-2.patch > > > We use int32 for memory now, if a cluster has 10k nodes, each node has 210G > memory, we will get a negative total cluster memory. > And another case that easier overflows int32 is: we added all pending > resources of running apps to cluster's total pending resources. If a > problematic app requires too much resources (let's say 1M+ containers, each > of them has 3G containers), int32 will be not enough. > Even if we can cap each app's pending request, we cannot handle the case that > there're many running apps, each of them has capped but still significant > numbers of pending resources. > So we may possibly need to add getMemoryLong/getVirtualCoreLong to > o.a.h.y.api.records.Resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4844) Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource
[ https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4844: - Attachment: YARN-4844.9.branch-2.patch > Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource > > > Key: YARN-4844 > URL: https://issues.apache.org/jira/browse/YARN-4844 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-4844.1.patch, YARN-4844.2.patch, YARN-4844.3.patch, > YARN-4844.4.patch, YARN-4844.5.patch, YARN-4844.6.patch, YARN-4844.7.patch, > YARN-4844.8.branch-2.patch, YARN-4844.8.patch, YARN-4844.9.branch-2.patch > > > We use int32 for memory now, if a cluster has 10k nodes, each node has 210G > memory, we will get a negative total cluster memory. > And another case that easier overflows int32 is: we added all pending > resources of running apps to cluster's total pending resources. If a > problematic app requires too much resources (let's say 1M+ containers, each > of them has 3G containers), int32 will be not enough. > Even if we can cap each app's pending request, we cannot handle the case that > there're many running apps, each of them has capped but still significant > numbers of pending resources. > So we may possibly need to add getMemoryLong/getVirtualCoreLong to > o.a.h.y.api.records.Resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2882) Add an OPPORTUNISTIC ExecutionType
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-2882: -- Issue Type: Sub-task (was: New Feature) Parent: YARN-2877 > Add an OPPORTUNISTIC ExecutionType > -- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 3.0.0-alpha1 > > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, > YARN-2882-yarn-2877.004.patch, YARN-2882.005.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4913) Yarn logs should take a -out option to write to a directory
[ https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288120#comment-15288120 ] Varun Vasudev commented on YARN-4913: - After some offline discussion with [~xgong] and [~djp] it looks like the bug I'm seeing is specific to my setup. +1 for the latest patch. I'll commit it tomorrow if no one objects. > Yarn logs should take a -out option to write to a directory > --- > > Key: YARN-4913 > URL: https://issues.apache.org/jira/browse/YARN-4913 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4913.1.patch, YARN-4913.2.patch, YARN-4913.3.patch, > YARN-4913.4.patch, YARN-4913.5.1.patch, YARN-4913.5.patch, YARN-4913.6.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5090) Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster
[
https://issues.apache.org/jira/browse/YARN-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288113#comment-15288113
]
Hudson commented on YARN-5090:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9812 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9812/])
YARN-5090. Add End-to-End test-cases for DistributedScheduling using (arun
suresh: rev 8a9ecb7584b529be1de3a6e8850041e211c88ebc)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMProxy.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestDistributedSchedulingService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestDistributedScheduling.java
> Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster
> --
>
> Key: YARN-5090
> URL: https://issues.apache.org/jira/browse/YARN-5090
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5090.001.patch, YARN-5090.002.patch,
> YARN-5090.002.patch, mvn_test.txt, mvn_test_2.txt
>
>
> Validates End2End Distributed Scheduling flow which includes
> # the AM specifying *OPPORTUNISTIC* containers in its resource requests
> (YARN-2882, YARN-4335)
> # the Request intercepted by the {{AMRMProxyService}} running on the NM and
> handled by the {{LocalScheduler}} RequestInterceptor (YARN-2884, YARN-2885)
> # use of the {{DistributedSchedulingProtocol}} to talk to the
> {{DistributedSchedulingService}} running on the RM to obtain list of lightly
> loaded Nodes (YARN-2885, YARN-4412)
> # use of the {{OpportunisticContainerAllocator}} to create Container tokens
> which are returned to the AM (YARN-2885)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4887) AM-RM protocol changes for supporting delta protocol
[ https://issues.apache.org/jira/browse/YARN-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-4887: - Attachment: YARN-4887-v1.patch Attaching a patch with the proposed API changes. The patch does not have a test case as it changes only API records which are already automatically covered by _TestPBImplRecords_ > AM-RM protocol changes for supporting delta protocol > > > Key: YARN-4887 > URL: https://issues.apache.org/jira/browse/YARN-4887 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-4887-v1.patch > > > YARN-4879 proposes the addition of a simple delta allocate protocol. This > JIRA is to track the changes in AM-RM protocol to accomplish it. The crux is > the addition of ID field in ResourceRequest and Container. The detailed > proposal is in the parent JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5107) TestContainerMetrics fails
[
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288102#comment-15288102
]
Hadoop QA commented on YARN-5107:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 12s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804542/YARN-5107.01.patch |
| JIRA Issue | YARN-5107 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 87e62b2d16d5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7154ace |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/11519/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11519/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: YARN-5107.01.patch
>
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.022
[jira] [Updated] (YARN-5090) Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster
[
https://issues.apache.org/jira/browse/YARN-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5090:
--
Parent Issue: YARN-2877 (was: YARN-4742)
> Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster
> --
>
> Key: YARN-5090
> URL: https://issues.apache.org/jira/browse/YARN-5090
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5090.001.patch, YARN-5090.002.patch,
> YARN-5090.002.patch, mvn_test.txt, mvn_test_2.txt
>
>
> Validates End2End Distributed Scheduling flow which includes
> # the AM specifying *OPPORTUNISTIC* containers in its resource requests
> (YARN-2882, YARN-4335)
> # the Request intercepted by the {{AMRMProxyService}} running on the NM and
> handled by the {{LocalScheduler}} RequestInterceptor (YARN-2884, YARN-2885)
> # use of the {{DistributedSchedulingProtocol}} to talk to the
> {{DistributedSchedulingService}} running on the RM to obtain list of lightly
> loaded Nodes (YARN-2885, YARN-4412)
> # use of the {{OpportunisticContainerAllocator}} to create Container tokens
> which are returned to the AM (YARN-2885)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5090) Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster
[
https://issues.apache.org/jira/browse/YARN-5090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288073#comment-15288073
]
Arun Suresh commented on YARN-5090:
---
Thanks again for the review [~leftnoteasy].
Committing this now..
> Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster
> --
>
> Key: YARN-5090
> URL: https://issues.apache.org/jira/browse/YARN-5090
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5090.001.patch, YARN-5090.002.patch,
> YARN-5090.002.patch, mvn_test.txt, mvn_test_2.txt
>
>
> Validates End2End Distributed Scheduling flow which includes
> # the AM specifying *OPPORTUNISTIC* containers in its resource requests
> (YARN-2882, YARN-4335)
> # the Request intercepted by the {{AMRMProxyService}} running on the NM and
> handled by the {{LocalScheduler}} RequestInterceptor (YARN-2884, YARN-2885)
> # use of the {{DistributedSchedulingProtocol}} to talk to the
> {{DistributedSchedulingService}} running on the RM to obtain list of lightly
> loaded Nodes (YARN-2885, YARN-4412)
> # use of the {{OpportunisticContainerAllocator}} to create Container tokens
> which are returned to the AM (YARN-2885)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2877) Extend YARN to support distributed scheduling
[ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288066#comment-15288066 ] Konstantinos Karanasos commented on YARN-2877: -- Marked the JIRA as resolved and added a release note. Thank you all for the invaluable feedback, the contributions and the extensive code reviews! Among all the people that contributed, I would like to particularly call out [~asuresh], [~curino], [~chris.douglas], [~subru], [~kishorch], [~kasha], [~leftnoteasy], [~jianhe], and [~sriramsrao]. > Extend YARN to support distributed scheduling > - > > Key: YARN-2877 > URL: https://issues.apache.org/jira/browse/YARN-2877 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos > Attachments: distributed-scheduling-design-doc_v1.pdf > > > This is an umbrella JIRA that proposes to extend YARN to support distributed > scheduling. Briefly, some of the motivations for distributed scheduling are > the following: > 1. Improve cluster utilization by opportunistically executing tasks otherwise > idle resources on individual machines. > 2. Reduce allocation latency. Tasks where the scheduling time dominates > (i.e., task execution time is much less compared to the time required for > obtaining a container from the RM). > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-2877) Extend YARN to support distributed scheduling
[ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos resolved YARN-2877. -- Resolution: Resolved Hadoop Flags: Reviewed Release Note: With this JIRA we are introducing distributed scheduling in YARN. In particular, we make the following contributions: - Introduce the notion of container types. GUARANTEED containers follow the semantics of the existing YARN containers. OPPORTUNISTIC ones can be seen as lower priority containers, and can be preempted in order to make space for GUARANTEED containers to run. - Queuing of tasks at the NMs. This enables us to send more containers in an NM than its available resources. At the moment we are allowing queuing of OPPORTUNISTIC containers. Once resources become available at the NM, such containers can immediately start their execution. - Introduce the AMRMProxy. This is a service running at each node, intercepting the requests between the AM and the RM. It is instrumental for both distributed scheduling and YARN Federation (YARN-2915). - Enable distributed scheduling. To minimize their allocation latency, OPPORTUNISTIC containers are dispatched immediately to NMs in a distributed fashion by using the AMRMProxy of the node where the corresponding AM resides, without needing to go through the ResourceManager. All the functionality introduced in this JIRA is disabled by default, so it will not affect the behavior of existing applications. We have introduced parameters in YarnConfiguration to enable NM queuing (yarn.nodemanager.container-queuing-enabled), distributed scheduling (yarn.distributed-scheduling.enabled) and the AMRMProxy service (yarn.nodemanager.amrmproxy.enable). AMs currently need to specify the type of container to be requested for each task. We are in the process of adding in the MapReduce AM the ability to randomly request OPPORTUNISTIC containers for a specified percentage of a job's tasks, so that users can experiment with the new features. Target Version/s: 3.0.0-alpha1 > Extend YARN to support distributed scheduling > - > > Key: YARN-2877 > URL: https://issues.apache.org/jira/browse/YARN-2877 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos > Attachments: distributed-scheduling-design-doc_v1.pdf > > > This is an umbrella JIRA that proposes to extend YARN to support distributed > scheduling. Briefly, some of the motivations for distributed scheduling are > the following: > 1. Improve cluster utilization by opportunistically executing tasks otherwise > idle resources on individual machines. > 2. Reduce allocation latency. Tasks where the scheduling time dominates > (i.e., task execution time is much less compared to the time required for > obtaining a container from the RM). > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4308) ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats
[
https://issues.apache.org/jira/browse/YARN-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287992#comment-15287992
]
Naganarasimha G R commented on YARN-4308:
-
[~sunilg], As most of us agree that metrics is not actually much useful, So can
you update the patch with updating javadocs and approp comment, and also i am
fine with info logging as its once per container so not much of an impact ?
> ContainersAggregated CPU resource utilization reports negative usage in first
> few heartbeats
>
>
> Key: YARN-4308
> URL: https://issues.apache.org/jira/browse/YARN-4308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4308.patch, 0002-YARN-4308.patch
>
>
> NodeManager reports ContainerAggregated CPU resource utilization as -ve value
> in first few heartbeats cycles. I added a new debug print and received below
> values from heartbeats.
> {noformat}
> INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> ContainersResource Utilization : CpuTrackerUsagePercent : -1.0
> INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:ContainersResource
> Utilization : CpuTrackerUsagePercent : 198.94598
> {noformat}
> Its better we send 0 as CPU usage rather than sending a negative values in
> heartbeats eventhough its happening in only first few heartbeats.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5108) TestFileSystemApplicationHistoryStore fails
[
https://issues.apache.org/jira/browse/YARN-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-5108:
-
Description:
Unit test failure is found in other JIRA Jenkins test (YARN-5100):
{noformat}
Running
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore
Tests run: 8, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 20.188 sec <<<
FAILURE! - in
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore
testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
Time elapsed: 0.054 sec <<< ERROR!
java.lang.NullPointerException: null
at java.util.TreeMap.getEntry(TreeMap.java:347)
at java.util.TreeMap.get(TreeMap.java:278)
at
org.apache.hadoop.fs.GlobalStorageStatistics.put(GlobalStorageStatistics.java:73)
at org.apache.hadoop.fs.FileSystem.getStatistics(FileSystem.java:3598)
at org.apache.hadoop.fs.FileSystem.initialize(FileSystem.java:214)
at
org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:101)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.setup(TestFileSystemApplicationHistoryStore.java:64)
testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
Time elapsed: 0.054 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.tearDown(TestFileSystemApplicationHistoryStore.java:89)
testInitNonExistingWorkingDirectoryInSafeMode(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
Time elapsed: 0.517 sec <<< FAILURE!
org.mockito.exceptions.verification.WantedButNotInvoked:
Wanted but not invoked:
rawLocalFileSystem.isDirectory();
-> at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304)
However, there were other interactions with this mock:
-> at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304)
{noformat}
was:
Running
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore
Tests run: 8, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 20.188 sec <<<
FAILURE! - in
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore
testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
Time elapsed: 0.054 sec <<< ERROR!
java.lang.NullPointerException: null
at java.util.TreeMap.getEntry(TreeMap.java:347)
at java.util.TreeMap.get(TreeMap.java:278)
at
org.apache.hadoop.fs.GlobalStorageStatistics.put(GlobalStorageStatistics.java:73)
at org.apache.hadoop.fs.FileSystem.getStatistics(FileSystem.java:3598)
at org.apache.hadoop.fs.FileSystem.initialize(FileSystem.java:214)
at
org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:101)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.setup(TestFileSystemApplicationHistoryStore.java:64)
testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
Time elapsed: 0.054 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.tearDown(TestFileSystemApplicationHistoryStore.java:89)
testInitNonExistingWorkingDirectoryInSafeMode(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore)
Time elapsed: 0.517 sec <<< FAILURE!
org.mockito.exceptions.verification.WantedButNotInvoked:
Wanted but not invoked:
rawLocalFileSystem.isDirectory();
-> at
org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHis
[jira] [Commented] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287986#comment-15287986
]
Sangjin Lee commented on YARN-5093:
---
The fix could be not writing the created column (and any other column that has
a similar issue) if the value is "empty"? cc [~vrushalic]
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5096) timelinereader has a lot of logging that's not useful
[
https://issues.apache.org/jira/browse/YARN-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee reassigned YARN-5096:
-
Assignee: Sangjin Lee
> timelinereader has a lot of logging that's not useful
> -
>
> Key: YARN-5096
> URL: https://issues.apache.org/jira/browse/YARN-5096
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Labels: yarn-2928-1st-milestone
>
> After running about a dozen or so requests, the timelinereader log is filled
> with the following logging entries:
> {noformat}
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> 2016-05-16 15:59:13,364 INFO
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper:
> null prefix was specified; returning all columns
> {noformat}
> There were some ~ 3,000 such logging entries. It's too excessive.
> Also, when I requested YARN_CONTAINER with fields=ALL, I see the following
> logs:
> {noformat}
> WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5100) The YarnApplicationState is always running in ATS no matter the application is running or finishes.
[ https://issues.apache.org/jira/browse/YARN-5100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287968#comment-15287968 ] Xuan Gong commented on YARN-5100: - [~djp] The testcase failure is not related. Create https://issues.apache.org/jira/browse/YARN-5108 to track that > The YarnApplicationState is always running in ATS no matter the application > is running or finishes. > --- > > Key: YARN-5100 > URL: https://issues.apache.org/jira/browse/YARN-5100 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Attachments: YARN-5100.1.patch, YARN-5100.2.patch > > > After YARN-5029, we add one more event : APP_STATE_UPDATE event which is used > by RM to sync up the App state between Timeline Server. But when we get > appReport from TimelineServer, we parse all events with timestamp descending > order (Finish event-->App State update event --> create event) which causes > this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5108) TestFileSystemApplicationHistoryStore fails
Xuan Gong created YARN-5108: --- Summary: TestFileSystemApplicationHistoryStore fails Key: YARN-5108 URL: https://issues.apache.org/jira/browse/YARN-5108 Project: Hadoop YARN Issue Type: Test Reporter: Xuan Gong Running org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore Tests run: 8, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 20.188 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore) Time elapsed: 0.054 sec <<< ERROR! java.lang.NullPointerException: null at java.util.TreeMap.getEntry(TreeMap.java:347) at java.util.TreeMap.get(TreeMap.java:278) at org.apache.hadoop.fs.GlobalStorageStatistics.put(GlobalStorageStatistics.java:73) at org.apache.hadoop.fs.FileSystem.getStatistics(FileSystem.java:3598) at org.apache.hadoop.fs.FileSystem.initialize(FileSystem.java:214) at org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:101) at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70) at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.setup(TestFileSystemApplicationHistoryStore.java:64) testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore) Time elapsed: 0.054 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.tearDown(TestFileSystemApplicationHistoryStore.java:89) testInitNonExistingWorkingDirectoryInSafeMode(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore) Time elapsed: 0.517 sec <<< FAILURE! org.mockito.exceptions.verification.WantedButNotInvoked: Wanted but not invoked: rawLocalFileSystem.isDirectory(); -> at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304) However, there were other interactions with this mock: -> at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70) at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1151) Ability to configure auxiliary services from HDFS-based JAR files
[ https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287966#comment-15287966 ] Hitesh Shah commented on YARN-1151: --- For paths, the impl should do auto-resolving i.e. use default fs if no fs specified, support both file:// and hdfs:// which is likely to happen especially if we have a mix of jars coming in via rpms vs hdfs-based cache. > Ability to configure auxiliary services from HDFS-based JAR files > - > > Key: YARN-1151 > URL: https://issues.apache.org/jira/browse/YARN-1151 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.1.0-beta, 2.9.0 >Reporter: john lilley >Assignee: Xuan Gong > Labels: auxiliary-service, yarn > Attachments: YARN-1151.1.patch > > > I would like to install an auxiliary service in Hadoop YARN without actually > installing files/services on every node in the system. Discussions on the > user@ list indicate that this is not easily done. The reason we want an > auxiliary service is that our application has some persistent-data components > that are not appropriate for HDFS. In fact, they are somewhat analogous to > the mapper output of MapReduce's shuffle, which is what led me to > auxiliary-services in the first place. It would be much easier if we could > just place our service's JARs in HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1151) Ability to configure auxiliary services from HDFS-based JAR files
[ https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287963#comment-15287963 ] Hitesh Shah commented on YARN-1151: --- How can one specify an archive? or would only simple files be supported? Additionally, would a fat jar ( jar-with-dependencies ) work out of the box? > Ability to configure auxiliary services from HDFS-based JAR files > - > > Key: YARN-1151 > URL: https://issues.apache.org/jira/browse/YARN-1151 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.1.0-beta, 2.9.0 >Reporter: john lilley >Assignee: Xuan Gong > Labels: auxiliary-service, yarn > Attachments: YARN-1151.1.patch > > > I would like to install an auxiliary service in Hadoop YARN without actually > installing files/services on every node in the system. Discussions on the > user@ list indicate that this is not easily done. The reason we want an > auxiliary service is that our application has some persistent-data components > that are not appropriate for HDFS. In fact, they are somewhat analogous to > the mapper output of MapReduce's shuffle, which is what led me to > auxiliary-services in the first place. It would be much easier if we could > just place our service's JARs in HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4844) Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource
[
https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287942#comment-15287942
]
Hadoop QA commented on YARN-4844:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 55 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 26s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 16s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 25s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
45s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
21s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 37s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 25s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 4s
{color} | {color:red} root in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 4s {color} |
{color:red} root in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 4s {color}
| {color:red} root in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 43s
{color} | {color:red} root in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 43s {color} |
{color:red} root in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 43s {color}
| {color:red} root in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 15s
{color} | {color:red} root: patch generated 187 new + 4208 unchanged - 113
fixed = 4395 total (was 4321) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 16s
{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {
[jira] [Assigned] (YARN-3621) FairScheduler doesn't count AM vcores towards max-share
[ https://issues.apache.org/jira/browse/YARN-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu reassigned YARN-3621: -- Assignee: Yufei Gu > FairScheduler doesn't count AM vcores towards max-share > --- > > Key: YARN-3621 > URL: https://issues.apache.org/jira/browse/YARN-3621 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > > FairScheduler seems to not count AM vcores towards max-vcores. On a queue > with maxVcores set to 1, I am able to run a sleep job. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4975) Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues
[
https://issues.apache.org/jira/browse/YARN-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated YARN-4975:
-
Assignee: Yufei Gu
> Fair Scheduler: exception thrown when a parent queue marked 'parent' has
> configured child queues
>
>
> Key: YARN-4975
> URL: https://issues.apache.org/jira/browse/YARN-4975
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
>
> We upgraded our clusters to 2.7.2 from 2.4.1 and saw the following exception
> in RM logs :
> {code}
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Both and type="parent" found for queue root.adhoc which is
> unsupported
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:519)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:352)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1440)
> {code}
> From the exception, it looks like we've configured 'reservation', but we've
> not. The issue is that AllocationFileLoaderService#loadQueue assumes that a
> parent queue marked as 'type=parent' cannot have configured child queues.
> That can be a problem in cases where we mark a queue as 'parent' which has no
> configured child queues to start with, but we can add child queues later on.
> Also the exception message is kind of misleading since we haven't configured
> 'reservation'.
> How to reproduce:
> Run fair scheduler with following queue config:
> {code}
>
> 10
> 300
>
> 3
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3621) FairScheduler doesn't count AM vcores towards max-share
[ https://issues.apache.org/jira/browse/YARN-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu resolved YARN-3621. Resolution: Duplicate > FairScheduler doesn't count AM vcores towards max-share > --- > > Key: YARN-3621 > URL: https://issues.apache.org/jira/browse/YARN-3621 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > > FairScheduler seems to not count AM vcores towards max-vcores. On a queue > with maxVcores set to 1, I am able to run a sleep job. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5106: - Assignee: Yufei Gu > Provide a builder interface for FairScheduler allocations for use in tests > -- > > Key: YARN-5106 > URL: https://issues.apache.org/jira/browse/YARN-5106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > Labels: newbie++ > > Most, if not all, fair scheduler tests create an allocations XML file. Having > a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3890) FairScheduler should show the scheduler health metrics similar to ones added in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-3890: - Assignee: Yufei Gu > FairScheduler should show the scheduler health metrics similar to ones added > in CapacityScheduler > - > > Key: YARN-3890 > URL: https://issues.apache.org/jira/browse/YARN-3890 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Yufei Gu > > We should add information displayed in YARN-3293 in FairScheduler as well > possibly sharing the implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4832) NM side resource value should get updated if change applied in RM side
[
https://issues.apache.org/jira/browse/YARN-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287908#comment-15287908
]
Hadoop QA commented on YARN-4832:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: patch
generated 1 new + 129 unchanged - 1 fixed = 130 total (was 130) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 27s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdk1.8.0_91
with JDK v1.8.0_91 generated 22 new + 1060 unchanged - 0 fixed = 1082 total
(was 1060) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-server-common in the patch passed with JDK
v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 38s
{color} | {color:green} hadoop-yarn-server-nodemanager in the p
[jira] [Commented] (YARN-1942) Many of ConverterUtils methods need to have public interfaces
[
https://issues.apache.org/jira/browse/YARN-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287894#comment-15287894
]
Hadoop QA commented on YARN-1942:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 37s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 14s
{color} | {color:red} root generated 60 new + 672 unchanged - 0 fixed = 732
total (was 672) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 52s
{color} | {color:red} root: patch generated 77 new + 2419 unchanged - 20 fixed
= 2496 total (was 2439) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
57s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 11 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 36s {color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 17s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 2s {color} |
{color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 12s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 23s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 47s {color}
| {color:red} hadoop-yarn-server-timeline-pluginstorage in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 8s {color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s
{color} | {colo
[jira] [Commented] (YARN-5107) TestContainerMetrics fails
[
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287879#comment-15287879
]
Akira AJISAKA commented on YARN-5107:
-
This issue seems to be related to JDK8 because the error does not occur in
branch-2.
> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: YARN-5107.01.patch
>
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.022 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsFlow(TestContainerMetrics.java:53)
> testContainerMetricsLimit(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.001 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsLimit(TestContainerMetrics.java:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287866#comment-15287866
]
Yufei Gu commented on YARN-5104:
The failed test is unrelated.
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5104.001.patch
>
>
> TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
> and handled.
> YARN-4916 did handle this exception OK:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw
> {code}
> java.io.IOException: Failed on local exception: java.net.SocketException:
> Invalid argument; Host Details : local host is: "xxx"; destination host is:
> "1234":0; "
> {code}
> It failed YARN-4916 with following message.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287864#comment-15287864
]
Sangjin Lee commented on YARN-5093:
---
Yes, I think you're right. I initially thought this might be a UI issue, but
the data also seems to be overwritten as 0. From the application table:
{noformat}
yarn_cluster!sjlee!Sleep
job!\x7F\xFF\xFE\xAB?\x98I\xF7!\x7F\xFF\xFE\xAB?\x9A\xE6~\x7F\xFF\
column=i:created_time, timestamp=146352718068400,
value=\x00\x00\x00\x00\x00\x00\x00\x00
{noformat}
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5107) TestContainerMetrics fails
[
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated YARN-5107:
Attachment: YARN-5107.01.patch
> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
> Attachments: YARN-5107.01.patch
>
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.022 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsFlow(TestContainerMetrics.java:53)
> testContainerMetricsLimit(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.001 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsLimit(TestContainerMetrics.java:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5107) TestContainerMetrics fails
[
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned YARN-5107:
---
Assignee: Akira AJISAKA
> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: YARN-5107.01.patch
>
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.022 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsFlow(TestContainerMetrics.java:53)
> testContainerMetricsLimit(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.001 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsLimit(TestContainerMetrics.java:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-5093:
--
Description:
When querying the REST API, I find that the created time value is returned as
"0" for most of the output. It includes:
- flow activity and flow runs in the flow activity page
- apps in the application page
- entities in the entity page
For example, in the flow activity page,
{noformat}
{
metrics: [ ],
events: [ ],
id: "yarn_cluster/146335680/sjlee@ds-date",
type: "YARN_FLOW_ACTIVITY",
createdtime: 0,
flowruns: [
{
metrics: [ ],
events: [ ],
id: "sjlee@ds-date/1463435661428",
type: "YARN_FLOW_RUN",
createdtime: 0,
info: {
SYSTEM_INFO_FLOW_VERSION: "1",
SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
SYSTEM_INFO_FLOW_NAME: "ds-date",
SYSTEM_INFO_USER: "sjlee"
},
isrelatedto: { },
relatesto: { }
}
],
info: {
SYSTEM_INFO_CLUSTER: "yarn_cluster",
UID: "yarn_cluster!sjlee!ds-date",
SYSTEM_INFO_FLOW_NAME: "ds-date",
SYSTEM_INFO_DATE: 146335680,
SYSTEM_INFO_USER: "sjlee"
},
isrelatedto: { },
relatesto: { }
}
{noformat}
The only page that appears to show the proper created time value is the flow
run page.
was:
When querying the REST API, I find that the created time value is returned as
"0" for most of the output. It includes:
- flow activity and flow runs in the flow activity page
- apps in the application page
- entities in the entity page
For example, in the flow activity page,
{noformat}
{
metrics: [ ],
events: [ ],
id: "yarn_cluster/146335680/sjlee@ds-date",
type: "YARN_FLOW_ACTIVITY",
createdtime: 0,
flowruns: [
{
metrics: [ ],
events: [ ],
id: "sjlee@ds-date/1463435661428",
type: "YARN_FLOW_RUN",
createdtime: 0,
info: {
SYSTEM_INFO_FLOW_VERSION: "1",
SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
SYSTEM_INFO_FLOW_NAME: "ds-date",
SYSTEM_INFO_USER: "sjlee"
},
isrelatedto: { },
relatesto: { }
}
],
info: {
SYSTEM_INFO_CLUSTER: "yarn_cluster",
UID: "yarn_cluster!sjlee!ds-date",
SYSTEM_INFO_FLOW_NAME: "ds-date",
SYSTEM_INFO_DATE: 146335680,
SYSTEM_INFO_USER: "sjlee"
},
isrelatedto: { },
relatesto: { }
}
{noformat}
The only page that appears to show the proper created time value is the flow
run page. I think the data exists in the storage but is not populated in the UI.
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5107) TestContainerMetrics fails
[
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated YARN-5107:
Description:
{noformat}
testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
Time elapsed: 0.022 sec <<< ERROR!
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
TestContainerMetrics cannot be returned by register()
register() should return MetricsSink
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsFlow(TestContainerMetrics.java:53)
testContainerMetricsLimit(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
Time elapsed: 0.001 sec <<< ERROR!
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
TestContainerMetrics cannot be returned by register()
register() should return MetricsSink
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsLimit(TestContainerMetrics.java:93)
{noformat}
> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.022 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsFlow(TestContainerMetrics.java:53)
> testContainerMetricsLimit(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
> Time elapsed: 0.001 sec <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsLimit(TestContainerMetrics.java:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5107) TestContainerMetrics fails
Akira AJISAKA created YARN-5107: --- Summary: TestContainerMetrics fails Key: YARN-5107 URL: https://issues.apache.org/jira/browse/YARN-5107 Project: Hadoop YARN Issue Type: Bug Components: test Reporter: Akira AJISAKA -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3344) procfs stat file is not in the expected format warning
[
https://issues.apache.org/jira/browse/YARN-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287859#comment-15287859
]
Hadoop QA commented on YARN-3344:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
patch generated 0 new + 142 unchanged - 19 fixed = 142 total (was 161) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804538/YARN-3344.09.patch |
| JIRA Issue | YARN-3344 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 501bb78d2c06 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 16c07cc |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/11518/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11518/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> procfs stat file is not in the expected format warning
> --
>
> Key: YARN-3344
> URL: https://issues.apache.org/jira/browse/YARN-3344
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Jon Bringhurst
>Assignee: Akira AJISAKA
> Attachments: YARN-3344-trunk.005.patch, YARN-3344.06.patch,
> YARN-3344.07.patch, YARN-3344.08.patch, Y
[jira] [Commented] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287838#comment-15287838
]
Hadoop QA commented on YARN-5104:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 16s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804526/YARN-5104.001.patch |
| JIRA Issue | YARN-5104 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1619673e27b2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0c6726e |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/11515/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/11515/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/11515/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11515/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> TestNMProxy.t
[jira] [Updated] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[ https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-5089: -- Summary: Improve "yarn log" command-line "logFiles" option to support regex (was: Improve Yarn log Command line "logFiles" option to support Java regex ) > Improve "yarn log" command-line "logFiles" option to support regex > --- > > Key: YARN-5089 > URL: https://issues.apache.org/jira/browse/YARN-5089 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5089.1.patch > > > Right now, we only support matching the exact file name. We need to support > Java regex as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5047) Refactor nodeUpdate across schedulers
[ https://issues.apache.org/jira/browse/YARN-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-5047: --- Summary: Refactor nodeUpdate across schedulers (was: Refactor nodeUpdate() from FairScheduler, CapacityScheduler, and FifoScheduler) > Refactor nodeUpdate across schedulers > - > > Key: YARN-5047 > URL: https://issues.apache.org/jira/browse/YARN-5047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, scheduler >Affects Versions: 3.0.0-alpha1 >Reporter: Ray Chiang >Assignee: Ray Chiang > Attachments: YARN-5047.001.patch, YARN-5047.002.patch > > > FairScheduler#nodeUpdate() and CapacityScheduler#nodeUpdate() have a lot of > commonality in their code. See about refactoring the common parts into > AbstractYARNScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3344) procfs stat file is not in the expected format warning
[
https://issues.apache.org/jira/browse/YARN-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287828#comment-15287828
]
Daniel Templeton commented on YARN-3344:
LGTM. +1 (non-binding)
> procfs stat file is not in the expected format warning
> --
>
> Key: YARN-3344
> URL: https://issues.apache.org/jira/browse/YARN-3344
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Jon Bringhurst
>Assignee: Akira AJISAKA
> Attachments: YARN-3344-trunk.005.patch, YARN-3344.06.patch,
> YARN-3344.07.patch, YARN-3344.08.patch, YARN-3344.09.patch
>
>
> Although this doesn't appear to be causing any functional issues, it is
> spamming our log files quite a bit. :)
> It appears that the regex in ProcfsBasedProcessTree doesn't work for all
> /proc//stat files.
> Here's the error I'm seeing:
> {noformat}
> "source_host": "asdf",
> "method": "constructProcessInfo",
> "level": "WARN",
> "message": "Unexpected: procfs stat file is not in the expected format
> for process with pid 6953"
> "file": "ProcfsBasedProcessTree.java",
> "line_number": "514",
> "class": "org.apache.hadoop.yarn.util.ProcfsBasedProcessTree",
> {noformat}
> And here's the basic info on process with pid 6953:
> {noformat}
> [asdf ~]$ cat /proc/6953/stat
> 6953 (python2.6 /expo) S 1871 1871 1871 0 -1 4202496 9364 1080 0 0 25 3 0 0
> 20 0 1 0 144918696 205295616 5856 18446744073709551615 1 1 0 0 0 0 0 16781312
> 2 18446744073709551615 0 0 17 13 0 0 0 0 0
> [asdf ~]$ ps aux|grep 6953
> root 6953 0.0 0.0 200484 23424 ?S21:44 0:00 python2.6
> /export/apps/salt/minion-scripts/module-sync.py
> jbringhu 13481 0.0 0.0 105312 872 pts/0S+ 22:13 0:00 grep -i 6953
> [asdf ~]$
> {noformat}
> This is using 2.6.32-431.11.2.el6.x86_64 in RHEL 6.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3344) procfs stat file is not in the expected format warning
[
https://issues.apache.org/jira/browse/YARN-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated YARN-3344:
Attachment: YARN-3344.09.patch
Fixed checkstyle issues. Thanks!
> procfs stat file is not in the expected format warning
> --
>
> Key: YARN-3344
> URL: https://issues.apache.org/jira/browse/YARN-3344
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Jon Bringhurst
>Assignee: Akira AJISAKA
> Attachments: YARN-3344-trunk.005.patch, YARN-3344.06.patch,
> YARN-3344.07.patch, YARN-3344.08.patch, YARN-3344.09.patch
>
>
> Although this doesn't appear to be causing any functional issues, it is
> spamming our log files quite a bit. :)
> It appears that the regex in ProcfsBasedProcessTree doesn't work for all
> /proc//stat files.
> Here's the error I'm seeing:
> {noformat}
> "source_host": "asdf",
> "method": "constructProcessInfo",
> "level": "WARN",
> "message": "Unexpected: procfs stat file is not in the expected format
> for process with pid 6953"
> "file": "ProcfsBasedProcessTree.java",
> "line_number": "514",
> "class": "org.apache.hadoop.yarn.util.ProcfsBasedProcessTree",
> {noformat}
> And here's the basic info on process with pid 6953:
> {noformat}
> [asdf ~]$ cat /proc/6953/stat
> 6953 (python2.6 /expo) S 1871 1871 1871 0 -1 4202496 9364 1080 0 0 25 3 0 0
> 20 0 1 0 144918696 205295616 5856 18446744073709551615 1 1 0 0 0 0 0 16781312
> 2 18446744073709551615 0 0 17 13 0 0 0 0 0
> [asdf ~]$ ps aux|grep 6953
> root 6953 0.0 0.0 200484 23424 ?S21:44 0:00 python2.6
> /export/apps/salt/minion-scripts/module-sync.py
> jbringhu 13481 0.0 0.0 105312 872 pts/0S+ 22:13 0:00 grep -i 6953
> [asdf ~]$
> {noformat}
> This is using 2.6.32-431.11.2.el6.x86_64 in RHEL 6.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5077) Fix FSLeafQueue#getFairShare() for queues with weight 0.0
[
https://issues.apache.org/jira/browse/YARN-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287817#comment-15287817
]
Karthik Kambatla commented on YARN-5077:
This has been a long standing inconvenience. Thanks for working on this.
High-level comment: IIUC, we want to consider queues with zero weight only when
computing instantaneous fairshare. And, IIRR, only active apps are passed to
compute-instantaneous-shares. So, we probably don't have to check if a queue is
active. That said when computing instantaneous fairshares, we could check if
any of the queues have a non-zero weight.
Other minor comments on the patch:
# Instead of double negation in the variable name, can we pass
{{forceWeightToOne}} to {{ComputeShares#computeShare}} and {{allWeightsZero}}
to {{resourceUsedWithWeightToResourceRatio}}?
# The method to check weights itself could be {{areAllWeightsZero}}
The test is pretty neat. I cringe every time I see the xml form of the
FairScheduler allocations file in tests, but we already have many of them.
Filed YARN-5016 for that.
> Fix FSLeafQueue#getFairShare() for queues with weight 0.0
> -
>
> Key: YARN-5077
> URL: https://issues.apache.org/jira/browse/YARN-5077
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5077.001.patch, YARN-5077.002.patch
>
>
> 1) When a queue's weight is set to 0.0, FSLeafQueue#getFairShare() returns
>
> 2) When a queue's weight is nonzero, FSLeafQueue#getFairShare() returns
>
> In case 1), that means no container ever gets allocated for an AM because
> from the viewpoint of the RM, there is never any headroom to allocate a
> container on that queue.
> For example, we have a pool with the following weights:
> - root.dev 0.0
> - root.product 1.0
> The root.dev is a best effort pool and should only get resources if
> root.product is not running. In our tests, with no jobs running under
> root.product, jobs started in root.dev queue stay stuck in ACCEPT phase and
> never start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5106) Provide a builder interface for FairScheduler allocations for use in tests
Karthik Kambatla created YARN-5106: -- Summary: Provide a builder interface for FairScheduler allocations for use in tests Key: YARN-5106 URL: https://issues.apache.org/jira/browse/YARN-5106 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.8.0 Reporter: Karthik Kambatla Most, if not all, fair scheduler tests create an allocations XML file. Having a helper class that potentially uses a builder would make the tests cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287799#comment-15287799
]
Varun Saxena commented on YARN-5093:
Checked the code. The issue with created time in case of application and entity
is that in HBaseTimelineWriterImpl#storeInfo, we are blindly updating created
time without any check.
We are neither checking for any event, nor for created time value.
If created time is not populated, it will be by default 0. Whenever any entity
is published, created time column is also updated, which means on events after
application created event, it will be updated with 0.
As we take the latest version of column value, 0 is returned in response.
This should be the issue. But I will verify it once I start testing as well.
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page. I think the data exists in the storage but is not populated in the
> UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3344) procfs stat file is not in the expected format warning
[
https://issues.apache.org/jira/browse/YARN-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287790#comment-15287790
]
Daniel Templeton commented on YARN-3344:
See
https://builds.apache.org/job/PreCommit-YARN-Build/11493/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
> procfs stat file is not in the expected format warning
> --
>
> Key: YARN-3344
> URL: https://issues.apache.org/jira/browse/YARN-3344
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Jon Bringhurst
>Assignee: Akira AJISAKA
> Attachments: YARN-3344-trunk.005.patch, YARN-3344.06.patch,
> YARN-3344.07.patch, YARN-3344.08.patch
>
>
> Although this doesn't appear to be causing any functional issues, it is
> spamming our log files quite a bit. :)
> It appears that the regex in ProcfsBasedProcessTree doesn't work for all
> /proc//stat files.
> Here's the error I'm seeing:
> {noformat}
> "source_host": "asdf",
> "method": "constructProcessInfo",
> "level": "WARN",
> "message": "Unexpected: procfs stat file is not in the expected format
> for process with pid 6953"
> "file": "ProcfsBasedProcessTree.java",
> "line_number": "514",
> "class": "org.apache.hadoop.yarn.util.ProcfsBasedProcessTree",
> {noformat}
> And here's the basic info on process with pid 6953:
> {noformat}
> [asdf ~]$ cat /proc/6953/stat
> 6953 (python2.6 /expo) S 1871 1871 1871 0 -1 4202496 9364 1080 0 0 25 3 0 0
> 20 0 1 0 144918696 205295616 5856 18446744073709551615 1 1 0 0 0 0 0 16781312
> 2 18446744073709551615 0 0 17 13 0 0 0 0 0
> [asdf ~]$ ps aux|grep 6953
> root 6953 0.0 0.0 200484 23424 ?S21:44 0:00 python2.6
> /export/apps/salt/minion-scripts/module-sync.py
> jbringhu 13481 0.0 0.0 105312 872 pts/0S+ 22:13 0:00 grep -i 6953
> [asdf ~]$
> {noformat}
> This is using 2.6.32-431.11.2.el6.x86_64 in RHEL 6.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena reassigned YARN-5093:
--
Assignee: Varun Saxena
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page. I think the data exists in the storage but is not populated in the
> UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4212) FairScheduler: Parent queues with 'Fair' policy should compute shares of all resources for its children during a recompute
[
https://issues.apache.org/jira/browse/YARN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287789#comment-15287789
]
Hadoop QA commented on YARN-4212:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} YARN-4212 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12764429/YARN-4212.1.patch |
| JIRA Issue | YARN-4212 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11517/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> FairScheduler: Parent queues with 'Fair' policy should compute shares of all
> resources for its children during a recompute
> --
>
> Key: YARN-4212
> URL: https://issues.apache.org/jira/browse/YARN-4212
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Yufei Gu
> Labels: fairscheduler
> Attachments: YARN-4212.1.patch
>
>
> The Fair Scheduler, while performing a {{recomputeShares()}} during an
> {{update()}} call, uses the parent queues policy to distribute shares to its
> children.
> If the parent queues policy is 'fair', it only computes weight for memory and
> sets the vcores fair share of its children to 0.
> Assuming a situation where we have 1 parent queue with policy 'fair' and
> multiple leaf queues with policy 'drf', Any app submitted to the child queues
> with vcore requirement > 1 will always be above fairshare, since during the
> recomputeShare process, the child queues were all assigned 0 for fairshare
> vcores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4457) Cleanup unchecked types for EventHandler
[
https://issues.apache.org/jira/browse/YARN-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287787#comment-15287787
]
Hadoop QA commented on YARN-4457:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 30s
{color} | {color:red} root generated 4 new + 667 unchanged - 5 fixed = 671
total (was 672) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 0s
{color} | {color:red} root: patch generated 8 new + 1377 unchanged - 9 fixed =
1385 total (was 1386) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 25s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 54s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 10s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 26s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 110m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804512/YARN-4457.005.patch |
| JIRA Issue | YARN-4457 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| un
[jira] [Assigned] (YARN-4212) FairScheduler: Parent queues with 'Fair' policy should compute shares of all resources for its children during a recompute
[
https://issues.apache.org/jira/browse/YARN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu reassigned YARN-4212:
--
Assignee: Yufei Gu (was: Arun Suresh)
> FairScheduler: Parent queues with 'Fair' policy should compute shares of all
> resources for its children during a recompute
> --
>
> Key: YARN-4212
> URL: https://issues.apache.org/jira/browse/YARN-4212
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Yufei Gu
> Labels: fairscheduler
> Attachments: YARN-4212.1.patch
>
>
> The Fair Scheduler, while performing a {{recomputeShares()}} during an
> {{update()}} call, uses the parent queues policy to distribute shares to its
> children.
> If the parent queues policy is 'fair', it only computes weight for memory and
> sets the vcores fair share of its children to 0.
> Assuming a situation where we have 1 parent queue with policy 'fair' and
> multiple leaf queues with policy 'drf', Any app submitted to the child queues
> with vcore requirement > 1 will always be above fairshare, since during the
> recomputeShare process, the child queues were all assigned 0 for fairshare
> vcores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4212) FairScheduler: Parent queues with 'Fair' policy should compute shares of all resources for its children during a recompute
[
https://issues.apache.org/jira/browse/YARN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287784#comment-15287784
]
Yufei Gu commented on YARN-4212:
Thanks [~asuresh].
> FairScheduler: Parent queues with 'Fair' policy should compute shares of all
> resources for its children during a recompute
> --
>
> Key: YARN-4212
> URL: https://issues.apache.org/jira/browse/YARN-4212
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Labels: fairscheduler
> Attachments: YARN-4212.1.patch
>
>
> The Fair Scheduler, while performing a {{recomputeShares()}} during an
> {{update()}} call, uses the parent queues policy to distribute shares to its
> children.
> If the parent queues policy is 'fair', it only computes weight for memory and
> sets the vcores fair share of its children to 0.
> Assuming a situation where we have 1 parent queue with policy 'fair' and
> multiple leaf queues with policy 'drf', Any app submitted to the child queues
> with vcore requirement > 1 will always be above fairshare, since during the
> recomputeShare process, the child queues were all assigned 0 for fairshare
> vcores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4844) Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource
[
https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287785#comment-15287785
]
Hadoop QA commented on YARN-4844:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 55 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 28s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m 2s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 8m 45s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 8m 45s {color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 45s {color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m 33s
{color} | {color:red} root: patch generated 192 new + 4323 unchanged - 117
fixed = 4515 total (was 4440) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m 4s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 48s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 21s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 51s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 58s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 48s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 46s {color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color}
[jira] [Commented] (YARN-3344) procfs stat file is not in the expected format warning
[
https://issues.apache.org/jira/browse/YARN-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287779#comment-15287779
]
Akira AJISAKA commented on YARN-3344:
-
bq. Clean up your whitespace issues
Is there any trailing whitespaces in my patch? Jenkins says the patch has no
whitespace issues.
> procfs stat file is not in the expected format warning
> --
>
> Key: YARN-3344
> URL: https://issues.apache.org/jira/browse/YARN-3344
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Jon Bringhurst
>Assignee: Akira AJISAKA
> Attachments: YARN-3344-trunk.005.patch, YARN-3344.06.patch,
> YARN-3344.07.patch, YARN-3344.08.patch
>
>
> Although this doesn't appear to be causing any functional issues, it is
> spamming our log files quite a bit. :)
> It appears that the regex in ProcfsBasedProcessTree doesn't work for all
> /proc//stat files.
> Here's the error I'm seeing:
> {noformat}
> "source_host": "asdf",
> "method": "constructProcessInfo",
> "level": "WARN",
> "message": "Unexpected: procfs stat file is not in the expected format
> for process with pid 6953"
> "file": "ProcfsBasedProcessTree.java",
> "line_number": "514",
> "class": "org.apache.hadoop.yarn.util.ProcfsBasedProcessTree",
> {noformat}
> And here's the basic info on process with pid 6953:
> {noformat}
> [asdf ~]$ cat /proc/6953/stat
> 6953 (python2.6 /expo) S 1871 1871 1871 0 -1 4202496 9364 1080 0 0 25 3 0 0
> 20 0 1 0 144918696 205295616 5856 18446744073709551615 1 1 0 0 0 0 0 16781312
> 2 18446744073709551615 0 0 17 13 0 0 0 0 0
> [asdf ~]$ ps aux|grep 6953
> root 6953 0.0 0.0 200484 23424 ?S21:44 0:00 python2.6
> /export/apps/salt/minion-scripts/module-sync.py
> jbringhu 13481 0.0 0.0 105312 872 pts/0S+ 22:13 0:00 grep -i 6953
> [asdf ~]$
> {noformat}
> This is using 2.6.32-431.11.2.el6.x86_64 in RHEL 6.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4212) FairScheduler: Parent queues with 'Fair' policy should compute shares of all resources for its children during a recompute
[
https://issues.apache.org/jira/browse/YARN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287778#comment-15287778
]
Arun Suresh commented on YARN-4212:
---
Sure... feel free to take over..
> FairScheduler: Parent queues with 'Fair' policy should compute shares of all
> resources for its children during a recompute
> --
>
> Key: YARN-4212
> URL: https://issues.apache.org/jira/browse/YARN-4212
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Labels: fairscheduler
> Attachments: YARN-4212.1.patch
>
>
> The Fair Scheduler, while performing a {{recomputeShares()}} during an
> {{update()}} call, uses the parent queues policy to distribute shares to its
> children.
> If the parent queues policy is 'fair', it only computes weight for memory and
> sets the vcores fair share of its children to 0.
> Assuming a situation where we have 1 parent queue with policy 'fair' and
> multiple leaf queues with policy 'drf', Any app submitted to the child queues
> with vcore requirement > 1 will always be above fairshare, since during the
> recomputeShare process, the child queues were all assigned 0 for fairshare
> vcores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4212) FairScheduler: Parent queues with 'Fair' policy should compute shares of all resources for its children during a recompute
[
https://issues.apache.org/jira/browse/YARN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287774#comment-15287774
]
Yufei Gu commented on YARN-4212:
Hi [~asuresh], are you still working on this? If not, can I take it over?
> FairScheduler: Parent queues with 'Fair' policy should compute shares of all
> resources for its children during a recompute
> --
>
> Key: YARN-4212
> URL: https://issues.apache.org/jira/browse/YARN-4212
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Labels: fairscheduler
> Attachments: YARN-4212.1.patch
>
>
> The Fair Scheduler, while performing a {{recomputeShares()}} during an
> {{update()}} call, uses the parent queues policy to distribute shares to its
> children.
> If the parent queues policy is 'fair', it only computes weight for memory and
> sets the vcores fair share of its children to 0.
> Assuming a situation where we have 1 parent queue with policy 'fair' and
> multiple leaf queues with policy 'drf', Any app submitted to the child queues
> with vcore requirement > 1 will always be above fairshare, since during the
> recomputeShare process, the child queues were all assigned 0 for fairshare
> vcores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4308) ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats
[
https://issues.apache.org/jira/browse/YARN-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287771#comment-15287771
]
Karthik Kambatla commented on YARN-4308:
Given we expect to see -1 exactly once, do we really need metrics for this?
Just logging it at INFO level seems innocuous, no?
> ContainersAggregated CPU resource utilization reports negative usage in first
> few heartbeats
>
>
> Key: YARN-4308
> URL: https://issues.apache.org/jira/browse/YARN-4308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4308.patch, 0002-YARN-4308.patch
>
>
> NodeManager reports ContainerAggregated CPU resource utilization as -ve value
> in first few heartbeats cycles. I added a new debug print and received below
> values from heartbeats.
> {noformat}
> INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> ContainersResource Utilization : CpuTrackerUsagePercent : -1.0
> INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:ContainersResource
> Utilization : CpuTrackerUsagePercent : 198.94598
> {noformat}
> Its better we send 0 as CPU usage rather than sending a negative values in
> heartbeats eventhough its happening in only first few heartbeats.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4832) NM side resource value should get updated if change applied in RM side
[ https://issues.apache.org/jira/browse/YARN-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287769#comment-15287769 ] Hudson commented on YARN-4832: -- SUCCESS: Integrated in Hadoop-trunk-Commit #9808 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9808/]) Addendum patch for YARN-4832. Contributed by Junping Du (jianhe: rev 0c6726e20d9503589b21123f30757ddfbd405dde) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto > NM side resource value should get updated if change applied in RM side > -- > > Key: YARN-4832 > URL: https://issues.apache.org/jira/browse/YARN-4832 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Fix For: 2.8.0 > > Attachments: YARN-4832-addendum.patch, YARN-4832-branch-2.patch, > YARN-4832-demo.patch, YARN-4832-v2.patch, YARN-4832-v3.patch, YARN-4832.patch > > > Now, if we execute CLI to update node (single or multiple) resource in RM > side, NM will not receive any notification. It doesn't affect resource > scheduling but will make resource usage metrics reported by NM a bit weird. > We should sync up new resource between RM and NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4457) Cleanup unchecked types for EventHandler
[
https://issues.apache.org/jira/browse/YARN-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287757#comment-15287757
]
Karthik Kambatla commented on YARN-4457:
Thanks for taking the time to clean these up.
One rather high-level question: since all meaningful events have a type,
shouldn't the dispatcher itself be generic? Are there dispatchers that serve
events with closest common ancestor being only Event and no other subtype? If
we do that, would we be able to typify these EventHandlers better than just
?
> Cleanup unchecked types for EventHandler
>
>
> Key: YARN-4457
> URL: https://issues.apache.org/jira/browse/YARN-4457
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4457.001.patch, YARN-4457.002.patch,
> YARN-4457.003.patch, YARN-4457.004.patch, YARN-4457.005.patch
>
>
> The EventHandler class is often used in an untyped context resulting in a
> bunch of warnings about unchecked usage. The culprit is the
> {{Dispatcher.getHandler()}} method. Fixing the typing on the method to
> return {{EventHandler}} instead of {{EventHandler}} clears up the
> errors and doesn't not introduce any incompatible changes. In the case that
> some code does:
> {code}
> EventHandler h = dispatcher.getHandler();
> {code}
> it will still work and will issue a compiler warning about raw types. There
> are, however, no instances of this issue in the current source base.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4958) The file localization process should allow for wildcards to reduce the application footprint in the state store
[
https://issues.apache.org/jira/browse/YARN-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287735#comment-15287735
]
Daniel Templeton commented on YARN-4958:
The test failures are unrelated. I filed MAPREDUCE-6702 to track.
> The file localization process should allow for wildcards to reduce the
> application footprint in the state store
> ---
>
> Key: YARN-4958
> URL: https://issues.apache.org/jira/browse/YARN-4958
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-4958.001.patch, YARN-4958.002.patch,
> YARN-4958.003.patch
>
>
> When using the -libjars option to add classes to the classpath, every library
> so added is explicitly listed in the {{ContainerLaunchContext}}'s local
> resources even though they're all uploaded to the same directory in HDFS.
> When using tools like Crunch without an uber JAR or when trying to take
> advantage of the shared cache, the number of libraries can be quite large.
> We've seen many cases where we had to turn down the max number of
> applications to prevent ZK from running out of heap because of the size of
> the state store entries.
> Rather than listing all files independently, this JIRA proposes to have the
> NM allow wildcards in the resource localization paths. Specifically, we
> propose to allow a path to have a final component (name) set to "*", which is
> interpreted by the NM as "download the full directory and link to every file
> in it from the job's working directory." This behavior is the same as the
> current behavior when using -libjars, but avoids explicitly listing every
> file.
> This JIRA does not attempt to provide more general purpose wildcards, such as
> "\*.jar" or "file\*", as having multiple entries for a single directory
> presents numerous logistical issues.
> This JIRA also does not attempt to integrate with the shared cache. That
> work will be left to a future JIRA. Specifically, this JIRA only applies
> when a full directory is uploaded. Currently the shared cache does not
> handle directory uploads.
> This JIRA proposes to allow for wildcards both in the internal processing of
> the -libjars switch and in paths added through the {{Job}} and
> {{DistributedCache}} classes.
> The proposed approach is to treat a path, "dir/\*", as "dir" for purposes of
> all file verification and localization. In the final step, the NM will query
> the localized directory to get a list of the files in "dir" such that each
> can be linked from the job's working directory. Since $PWD/\* is always
> included on the classpath, all JAR files in "dir" will be in the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4832) NM side resource value should get updated if change applied in RM side
[
https://issues.apache.org/jira/browse/YARN-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287733#comment-15287733
]
Hadoop QA commented on YARN-4832:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} YARN-4832 does not apply to branch-2. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804521/YARN-4832-branch-2.patch
|
| JIRA Issue | YARN-4832 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11513/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> NM side resource value should get updated if change applied in RM side
> --
>
> Key: YARN-4832
> URL: https://issues.apache.org/jira/browse/YARN-4832
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
> Attachments: YARN-4832-addendum.patch, YARN-4832-branch-2.patch,
> YARN-4832-demo.patch, YARN-4832-v2.patch, YARN-4832-v3.patch, YARN-4832.patch
>
>
> Now, if we execute CLI to update node (single or multiple) resource in RM
> side, NM will not receive any notification. It doesn't affect resource
> scheduling but will make resource usage metrics reported by NM a bit weird.
> We should sync up new resource between RM and NM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5105) entire time series is returned for YARN container system metrics (CPU and memory)
[
https://issues.apache.org/jira/browse/YARN-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena reassigned YARN-5105:
--
Assignee: Varun Saxena
> entire time series is returned for YARN container system metrics (CPU and
> memory)
> -
>
> Key: YARN-5105
> URL: https://issues.apache.org/jira/browse/YARN-5105
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
>
> I see that the entire time series of the CPU and memory metrics are returned
> for the YARN containers REST query. This has a potential of bloating the
> output big time.
> {noformat}
> "metrics": [
> {
> "type": "TIME_SERIES",
> "id": "MEMORY",
> "values":
> {
> "1463518173363": 407539712,
> "1463518170347": 407539712,
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287708#comment-15287708
]
Varun Saxena commented on YARN-5093:
[~sjlee0],
For flow activities we do not populate created time. The top of the day
timestamp is carried in INFO field (SYSTEM_INFO_DATE: 146335680). Because
flow activity does not indicate flow created time.
And while setting flow runs within flow activity, we do not set created time
for these flow runs either. Reason being RUN_ID column in flow activity table
does not carry the min start time info associated with a flow run.
Created time should appear in app and entity query though. Will check on that.
Let me see if this comes up for me.
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Priority: Critical
> Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page. I think the data exists in the storage but is not populated in the
> UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4844) Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource
[
https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-4844:
-
Attachment: YARN-4844.8.branch-2.patch
Uploaded patch for branch-2: {{YARN-4844.8.branch-2.patch}}
> Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource
>
>
> Key: YARN-4844
> URL: https://issues.apache.org/jira/browse/YARN-4844
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-4844.1.patch, YARN-4844.2.patch, YARN-4844.3.patch,
> YARN-4844.4.patch, YARN-4844.5.patch, YARN-4844.6.patch, YARN-4844.7.patch,
> YARN-4844.8.branch-2.patch, YARN-4844.8.patch
>
>
> We use int32 for memory now, if a cluster has 10k nodes, each node has 210G
> memory, we will get a negative total cluster memory.
> And another case that easier overflows int32 is: we added all pending
> resources of running apps to cluster's total pending resources. If a
> problematic app requires too much resources (let's say 1M+ containers, each
> of them has 3G containers), int32 will be not enough.
> Even if we can cap each app's pending request, we cannot handle the case that
> there're many running apps, each of them has capped but still significant
> numbers of pending resources.
> So we may possibly need to add getMemoryLong/getVirtualCoreLong to
> o.a.h.y.api.records.Resource.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4494) Recover completed apps asynchronously
[ https://issues.apache.org/jira/browse/YARN-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287681#comment-15287681 ] Karthik Kambatla commented on YARN-4494: [~sunilg] - can you elaborate on the on-demand fast recovery? Are you suggesting we recover apps lazily when there is a request for them? If yes, do we recover everything when someone requests all apps? How about apps that match a specific category? I don't remember if I filed a JIRA, but would it help to simplify the recovery of completed applications to the extent possible so this is not an issue? > Recover completed apps asynchronously > - > > Key: YARN-4494 > URL: https://issues.apache.org/jira/browse/YARN-4494 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Jun Gong >Assignee: Jun Gong > > With RM HA enabled, when recovering apps, recover completed apps > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Attachment: YARN-5104.001.patch
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5104.001.patch
>
>
> TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
> and handled.
> YARN-4916 did handle this exception OK:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw
> {code}
> java.io.IOException: Failed on local exception: java.net.SocketException:
> Invalid argument; Host Details : local host is: "xxx"; destination host is:
> "1234":0; "
> {code}
> It failed YARN-4916 with following message.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.
[
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287674#comment-15287674
]
Karthik Kambatla commented on YARN-4464:
[~jianhe] - do you think it is okay to lower this all the way to 0? Personally,
I think setting it to 0 might be better so users immediately realize they don't
see any completed apps after a restart.
Can we update the default value in yarn-default.xml to match this? I am also
pleasantly surprised that none of our recovery tests assumed this is not zero :)
> default value of yarn.resourcemanager.state-store.max-completed-applications
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
> Issue Type: Wish
> Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow. I have waited about 20min.
> realize missing
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.
> need to change lower value or document notice on [RM Restart
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4002) make ResourceTrackerService.nodeHeartbeat more concurrent
[ https://issues.apache.org/jira/browse/YARN-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287663#comment-15287663 ] Jian He commented on YARN-4002: --- [~rohithsharma], could you fix the findbugs warning ? also, setExcludesFile, setIncludesFile, updateFileNames need not acquire the writelock or can they just be removed ? > make ResourceTrackerService.nodeHeartbeat more concurrent > - > > Key: YARN-4002 > URL: https://issues.apache.org/jira/browse/YARN-4002 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Hong Zhiguo >Assignee: Hong Zhiguo >Priority: Critical > Attachments: 0001-YARN-4002.patch, YARN-4002-lockless-read.patch, > YARN-4002-rwlock-v2.patch, YARN-4002-rwlock-v2.patch, > YARN-4002-rwlock-v3-rebase.patch, YARN-4002-rwlock-v3.patch, > YARN-4002-rwlock-v4.patch, YARN-4002-rwlock-v5.patch, YARN-4002-rwlock.patch, > YARN-4002-v0.patch > > > We have multiple RPC threads to handle NodeHeartbeatRequest from NMs. By > design the method ResourceTrackerService.nodeHeartbeat should be concurrent > enough to scale for large clusters. > But we have a "BIG" lock in NodesListManager.isValidNode which I think it's > unnecessary. > First, the fields "includes" and "excludes" of HostsFileReader are only > updated on "refresh nodes". All RPC threads handling node heartbeats are > only readers. So RWLock could be used to alow concurrent access by RPC > threads. > Second, since he fields "includes" and "excludes" of HostsFileReader are > always updated by "reference assignment", which is atomic in Java, the reader > side lock could just be skipped. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4866) FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy
[ https://issues.apache.org/jira/browse/YARN-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287660#comment-15287660 ] Karthik Kambatla commented on YARN-4866: Thanks for working on this, Yufei. Comments on the patch: # In addition to checking the cluster resources, should we also check against queue's max resources for this queue and any parents? For the latter, a helper method might be needed. # Nit: Can we avoid naming the variables with a "not" for readability? For instance, may be we could use overMaxAMShareLimit instead of notOverLimit and exhaustsVcores instead of notTakeAllVCore and negate the expressions as necessary? # Nit: Also, do we really need the second boolean. How about modifying the newly added if condition to if (!overMaxAMShareLimit && policy is not DRF) and just override overMaxAMShareLimit? The method itself could return !overMaxAMShareLimit? # Nit: After the changes, if the code needs more explanation, should we add a comment before the if condition so it is clear why the second check? > FairScheduler: AMs can consume all vcores leading to a livelock when using > FAIR policy > -- > > Key: YARN-4866 > URL: https://issues.apache.org/jira/browse/YARN-4866 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > Attachments: YARN-4866.001.patch, YARN-4866.002.patch, > YARN-4866.003.patch, YARN-4866.004.patch > > > The maxAMShare uses the queue's policy for enforcing limits. When using FAIR > policy, this considers only memory. If there are fewer vcores on the cluster, > the AMs can end up taking all the vcores leading to a livelock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Description:
TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
and handled.
YARN-4916 did handle this exception OK:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw
{code}
java.io.IOException: Failed on local exception: java.net.SocketException:
Invalid argument; Host Details : local host is: "xxx"; destination host is:
"1234":0; "
{code}
It failed YARN-4916 with following message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
was:
TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
and handled.
YARN-4916 did handle this exception OK:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw
{code}
java.io.IOException: Failed on local exception: java.net.SocketException:
Invalid argument; Host Details : local host is: "xxx"; destination host is:
"1234":0; "
{code}, which failed YARN-4916 with following message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
> and handled.
> YARN-4916 did handle this exception OK:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw
> {code}
> java.io.IOException: Failed on local exception: java.net.SocketException:
> Invalid argument; Host Details : local host is: "xxx"; destination host is:
> "1234":0; "
> {code}
> It failed YARN-4916 with following message.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Description:
TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
and handled.
YARN-4916 did handle this exception OK:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw
{code}
java.io.IOException: Failed on local exception: java.net.SocketException:
Invalid argument; Host Details : local host is: "xxx"; destination host is:
"1234":0; "
{code}, which failed YARN-4916 with following message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
was:
TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
and handled.
YARN-4916 did handle this exception OK:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916 with following
message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
> and handled.
> YARN-4916 did handle this exception OK:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw
> {code}
> java.io.IOException: Failed on local exception: java.net.SocketException:
> Invalid argument; Host Details : local host is: "xxx"; destination host is:
> "1234":0; "
> {code}, which failed YARN-4916 with following message.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Description:
TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
and handled.
YARN-4916 did handle this exception OK:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916 with following
message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
was:
YARN-4916 did solve this exception:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916 with following
message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> TestNMProxy.testNMProxyRPCRetry throws an exception and expected to be caught
> and handled.
> YARN-4916 did handle this exception OK:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw "java.io.IOException: Failed on local exception:
> java.net.SocketException: Invalid argument; Host Details : local host is:
> "xxx"; destination host is: "1234":0; ", which failed YARN-4916 with
> following message.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5105) entire time series is returned for YARN container system metrics (CPU and memory)
[
https://issues.apache.org/jira/browse/YARN-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287624#comment-15287624
]
Varun Saxena commented on YARN-5105:
How do we handle it ? Have an explicit limit from user or have an internal
limit or a combination of both (a default limit if no limit from user is
specified) ?
Or go the time range way. Had raised YARN-4455 earlier wherein the idea was to
trim the metric result by time range.
> entire time series is returned for YARN container system metrics (CPU and
> memory)
> -
>
> Key: YARN-5105
> URL: https://issues.apache.org/jira/browse/YARN-5105
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
> Labels: yarn-2928-1st-milestone
>
> I see that the entire time series of the CPU and memory metrics are returned
> for the YARN containers REST query. This has a potential of bloating the
> output big time.
> {noformat}
> "metrics": [
> {
> "type": "TIME_SERIES",
> "id": "MEMORY",
> "values":
> {
> "1463518173363": 407539712,
> "1463518170347": 407539712,
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Description:
YARN-4916 did solve this exception:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916 with following
message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
was:
YARN-4916 did solve this problem:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916 with following
message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> YARN-4916 did solve this exception:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw "java.io.IOException: Failed on local exception:
> java.net.SocketException: Invalid argument; Host Details : local host is:
> "xxx"; destination host is: "1234":0; ", which failed YARN-4916 with
> following message.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Description:
YARN-4916 did solve this problem:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916 with following
message.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
was:
YARN-4916 did solve this problem:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> YARN-4916 did solve this problem:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw "java.io.IOException: Failed on local exception:
> java.net.SocketException: Invalid argument; Host Details : local host is:
> "xxx"; destination host is: "1234":0; ", which failed YARN-4916 with
> following message.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Description:
YARN-4916 did solve this problem:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; ", which failed YARN-4916.
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
was:
YARN-4916 did solve this problem:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; "
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> YARN-4916 did solve this problem:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw "java.io.IOException: Failed on local exception:
> java.net.SocketException: Invalid argument; Host Details : local host is:
> "xxx"; destination host is: "1234":0; ", which failed YARN-4916.
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5104) TestNMProxy.testNMProxyRPCRetry failed
[
https://issues.apache.org/jira/browse/YARN-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-5104:
---
Description:
YARN-4916 did solve this problem:
{code}
java.net.BindException: Problem binding to [xxx] java.net.BindException: Can't
assign requested address; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
But it can also throw "java.io.IOException: Failed on local exception:
java.net.SocketException: Invalid argument; Host Details : local host is:
"xxx"; destination host is: "1234":0; "
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
was:
{code}
Error Message:
null
Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
{code}
> TestNMProxy.testNMProxyRPCRetry failed
> --
>
> Key: YARN-5104
> URL: https://issues.apache.org/jira/browse/YARN-5104
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> YARN-4916 did solve this problem:
> {code}
> java.net.BindException: Problem binding to [xxx] java.net.BindException:
> Can't assign requested address; For more details see:
> http://wiki.apache.org/hadoop/BindException
> {code}
> But it can also throw "java.io.IOException: Failed on local exception:
> java.net.SocketException: Invalid argument; Host Details : local host is:
> "xxx"; destination host is: "1234":0; "
> {code}
> Error Message:
> null
> Stack Trace:
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestNMProxy.testNMProxyRPCRetry(TestNMProxy.java:192)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4832) NM side resource value should get updated if change applied in RM side
[ https://issues.apache.org/jira/browse/YARN-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-4832: - Attachment: YARN-4832-branch-2.patch put a patch against branch-2 also. > NM side resource value should get updated if change applied in RM side > -- > > Key: YARN-4832 > URL: https://issues.apache.org/jira/browse/YARN-4832 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Attachments: YARN-4832-addendum.patch, YARN-4832-branch-2.patch, > YARN-4832-demo.patch, YARN-4832-v2.patch, YARN-4832-v3.patch, YARN-4832.patch > > > Now, if we execute CLI to update node (single or multiple) resource in RM > side, NM will not receive any notification. It doesn't affect resource > scheduling but will make resource usage metrics reported by NM a bit weird. > We should sync up new resource between RM and NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5100) The YarnApplicationState is always running in ATS no matter the application is running or finishes.
[ https://issues.apache.org/jira/browse/YARN-5100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287596#comment-15287596 ] Junping Du commented on YARN-5100: -- bq. This would affect YARN_APPLICATION_UPDATED event as well, but I have created a separate jira : https://issues.apache.org/jira/browse/YARN-5101 to check that issue. I see. Tracking this issue in another jira sounds good to me, and we can discuss if any other consideration should be taken. Remind by check-style report, the method of convertToApplicationReport() should be refactored to something shorter and more readable. Given this is a legacy issue, we can file a separated JIRA to fix this. [~xgong], does unit test failure related to your patch here? If not, do we have JIRA to track the test issue? > The YarnApplicationState is always running in ATS no matter the application > is running or finishes. > --- > > Key: YARN-5100 > URL: https://issues.apache.org/jira/browse/YARN-5100 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Attachments: YARN-5100.1.patch, YARN-5100.2.patch > > > After YARN-5029, we add one more event : APP_STATE_UPDATE event which is used > by RM to sync up the App state between Timeline Server. But when we get > appReport from TimelineServer, we parse all events with timestamp descending > order (Finish event-->App State update event --> create event) which causes > this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5098) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token
[
https://issues.apache.org/jira/browse/YARN-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287561#comment-15287561
]
Jian He commented on YARN-5098:
---
bq. the app needs to be submitted without an HDFS token so the RM will acquire
and manage it directly on the app's behalf
Btw, this is not necessary, RM will try to get the token on app's behalf if the
token is going to expire, regardless whether the app provided the token or not
in the first place.
I debugged this, with YARN-2704, in normal case, RM should get the new token
and distribute it to NM if the token is going to expire. The problem here is
that RM gets shutdown for a long time during which the token expired. After RM
restart, RM tries to recover the app and renew the token. Obviously the renew
will fail because the token is expired, and so the log aggregation failed when
the app completed.
One solution in my mind is to let RM request a new token and distribute it to
NM, if the token renewal fails on app recovery. Right now the failure is just
ignored and continue.
> Yarn Application log Aggreagation fails due to NM can not get correct HDFS
> delegation token
> ---
>
> Key: YARN-5098
> URL: https://issues.apache.org/jira/browse/YARN-5098
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Yesha Vora
>
> Environment : HA cluster
> Yarn application logs for long running application could not be gathered
> because Nodemanager failed to talk to HDFS with below error.
> {code}
> 2016-05-16 18:18:28,533 INFO logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:finishLogAggregation(555)) - Application just
> finished : application_1463170334122_0002
> 2016-05-16 18:18:28,545 WARN ipc.Client (Client.java:run(705)) - Exception
> encountered while connecting to the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 171 for hrt_qa) can't be found in cache
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:583)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:398)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:752)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:748)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:747)
> at
> org.apache.hadoop.ipc.Client$Connection.access$3100(Client.java:398)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1597)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at org.apache.hadoop.ipc.Client.call(Client.java:1386)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:240)
> at com.sun.proxy.$Proxy83.getServerDefaults(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getServerDefaults(ClientNamenodeProtocolTranslatorPB.java:282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy84.getServerDefaults(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getServerDefaults(DFSClient.java:1018)
> at org.apache.hadoop.fs.Hdfs.getServerDefaults(Hdfs.java:156)
> at
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:550)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:687)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5098) Yarn Application log Aggreagation fails due to NM can not get correct HDFS delegation token
[
https://issues.apache.org/jira/browse/YARN-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He reassigned YARN-5098:
-
Assignee: Jian He
> Yarn Application log Aggreagation fails due to NM can not get correct HDFS
> delegation token
> ---
>
> Key: YARN-5098
> URL: https://issues.apache.org/jira/browse/YARN-5098
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Yesha Vora
>Assignee: Jian He
>
> Environment : HA cluster
> Yarn application logs for long running application could not be gathered
> because Nodemanager failed to talk to HDFS with below error.
> {code}
> 2016-05-16 18:18:28,533 INFO logaggregation.AppLogAggregatorImpl
> (AppLogAggregatorImpl.java:finishLogAggregation(555)) - Application just
> finished : application_1463170334122_0002
> 2016-05-16 18:18:28,545 WARN ipc.Client (Client.java:run(705)) - Exception
> encountered while connecting to the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 171 for hrt_qa) can't be found in cache
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:583)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:398)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:752)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:748)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:747)
> at
> org.apache.hadoop.ipc.Client$Connection.access$3100(Client.java:398)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1597)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at org.apache.hadoop.ipc.Client.call(Client.java:1386)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:240)
> at com.sun.proxy.$Proxy83.getServerDefaults(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getServerDefaults(ClientNamenodeProtocolTranslatorPB.java:282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy84.getServerDefaults(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getServerDefaults(DFSClient.java:1018)
> at org.apache.hadoop.fs.Hdfs.getServerDefaults(Hdfs.java:156)
> at
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:550)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:687)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5018) Online aggregation logic should not run immediately after collectors got started
[ https://issues.apache.org/jira/browse/YARN-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5018: -- Labels: yarn-2928-1st-milestone (was: ) > Online aggregation logic should not run immediately after collectors got > started > > > Key: YARN-5018 > URL: https://issues.apache.org/jira/browse/YARN-5018 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Labels: yarn-2928-1st-milestone > > In app level collector, we launch the aggregation logic immediately after the > collector got started. However, at this time, important context data has yet > to be published to the container. Also, if the aggregation result is empty, > we do not need to publish them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5018) Online aggregation logic should not run immediately after collectors got started
[ https://issues.apache.org/jira/browse/YARN-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287551#comment-15287551 ] Li Lu commented on YARN-5018: - Agree that we should mark this as a merge blocker. For the missing CPU and memory data, I suspect there are some wire-up issues. I'll look into them after I finish some fixes in ATS v1.5. > Online aggregation logic should not run immediately after collectors got > started > > > Key: YARN-5018 > URL: https://issues.apache.org/jira/browse/YARN-5018 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > > In app level collector, we launch the aggregation logic immediately after the > collector got started. However, at this time, important context data has yet > to be published to the container. Also, if the aggregation result is empty, > we do not need to publish them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5105) entire time series is returned for YARN container system metrics (CPU and memory)
Sangjin Lee created YARN-5105:
-
Summary: entire time series is returned for YARN container system
metrics (CPU and memory)
Key: YARN-5105
URL: https://issues.apache.org/jira/browse/YARN-5105
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Affects Versions: YARN-2928
Reporter: Sangjin Lee
I see that the entire time series of the CPU and memory metrics are returned
for the YARN containers REST query. This has a potential of bloating the output
big time.
{noformat}
"metrics": [
{
"type": "TIME_SERIES",
"id": "MEMORY",
"values":
{
"1463518173363": 407539712,
"1463518170347": 407539712,
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4832) NM side resource value should get updated if change applied in RM side
[ https://issues.apache.org/jira/browse/YARN-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287550#comment-15287550 ] Junping Du commented on YARN-4832: -- Per discussion in YARN-2877, I put an addendum patch to update field sequence of NodeHeartbeatResponseProto given another new field added earlier will not released in 2.8. > NM side resource value should get updated if change applied in RM side > -- > > Key: YARN-4832 > URL: https://issues.apache.org/jira/browse/YARN-4832 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Attachments: YARN-4832-addendum.patch, YARN-4832-demo.patch, > YARN-4832-v2.patch, YARN-4832-v3.patch, YARN-4832.patch > > > Now, if we execute CLI to update node (single or multiple) resource in RM > side, NM will not receive any notification. It doesn't affect resource > scheduling but will make resource usage metrics reported by NM a bit weird. > We should sync up new resource between RM and NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4832) NM side resource value should get updated if change applied in RM side
[ https://issues.apache.org/jira/browse/YARN-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-4832: - Attachment: YARN-4832-addendum.patch > NM side resource value should get updated if change applied in RM side > -- > > Key: YARN-4832 > URL: https://issues.apache.org/jira/browse/YARN-4832 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Attachments: YARN-4832-addendum.patch, YARN-4832-demo.patch, > YARN-4832-v2.patch, YARN-4832-v3.patch, YARN-4832.patch > > > Now, if we execute CLI to update node (single or multiple) resource in RM > side, NM will not receive any notification. It doesn't affect resource > scheduling but will make resource usage metrics reported by NM a bit weird. > We should sync up new resource between RM and NM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5018) Online aggregation logic should not run immediately after collectors got started
[ https://issues.apache.org/jira/browse/YARN-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287548#comment-15287548 ] Sangjin Lee commented on YARN-5018: --- Should this also be marked as the merge blocker? And I see that the individual containers are publishing system metrics (CPU and memory) but I don't see them aggregated to the application. Is that also caused by this issue, or is it a separate problem? > Online aggregation logic should not run immediately after collectors got > started > > > Key: YARN-5018 > URL: https://issues.apache.org/jira/browse/YARN-5018 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > > In app level collector, we launch the aggregation logic immediately after the > collector got started. However, at this time, important context data has yet > to be published to the container. Also, if the aggregation result is empty, > we do not need to publish them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
