[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[
https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832295#comment-16832295
]
Prabhu Joseph commented on YARN-9524:
-
[~eyang] Thanks for checking this. Have tested YARN-6929 + YARN-9524 fix and
did not observe any issue, Tracking URL from RM UI redirects correctly to
HistoryServer and which fetches the app logs as expected. Attached screenshots
for reference. Redirect URL works for both the jobs running before and after
the fix.
But have faced issues initially when HistoryServer and ResourceManager not
restarted after applying the fix with only NodeManager was restarted. Can we
check if both ResourceManager and HistoryServer restarted after the fix.
> TestAHSWebServices and TestLogsCLI test case failures
> -
>
> Key: YARN-9524
> URL: https://issues.apache.org/jira/browse/YARN-9524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: Regression
> Attachments: YARN-9524-001.patch, YARN-9524-002.patch
>
>
> {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929.
> {code:java}
> [ERROR] Failures:
> [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014
> [ERROR] Errors:
> [ERROR]
> TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543
> » NullPointer
> [INFO]
> [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1
> [ERROR] Failures:
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] Errors:
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[
https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9524:
Attachment: RM-Tracking-URL.png
Redirects_HistoryServer.png
Fetches_Logs.png
> TestAHSWebServices and TestLogsCLI test case failures
> -
>
> Key: YARN-9524
> URL: https://issues.apache.org/jira/browse/YARN-9524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: Regression
> Attachments: Fetches_Logs.png, RM-Tracking-URL.png,
> Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch
>
>
> {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929.
> {code:java}
> [ERROR] Failures:
> [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014
> [ERROR] Errors:
> [ERROR]
> TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543
> » NullPointer
> [INFO]
> [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1
> [ERROR] Failures:
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] Errors:
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[
https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832301#comment-16832301
]
Hadoop QA commented on YARN-9524:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} YARN-9524 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9524 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24044/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> TestAHSWebServices and TestLogsCLI test case failures
> -
>
> Key: YARN-9524
> URL: https://issues.apache.org/jira/browse/YARN-9524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: Regression
> Attachments: Fetches_Logs.png, RM-Tracking-URL.png,
> Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch
>
>
> {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929.
> {code:java}
> [ERROR] Failures:
> [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014
> [ERROR] Errors:
> [ERROR]
> TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543
> » NullPointer
> [INFO]
> [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1
> [ERROR] Failures:
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] Errors:
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9477: --- Attachment: YARN-9477-002.patch > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9519) TFile log aggregation file format is insensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config
[
https://issues.apache.org/jira/browse/YARN-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-9519:
-
Attachment: YARN-9519.002.patch
> TFile log aggregation file format is insensitive to the
> yarn.log-aggregation.TFile.remote-app-log-dir config
>
>
> Key: YARN-9519
> URL: https://issues.apache.org/jira/browse/YARN-9519
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9519.001.patch, YARN-9519.002.patch
>
>
> The TFile log aggregation file format is not sensitive to the
> yarn.log-aggregation.TFile.remote-app-log-dir config.
> In {{LogAggregationTFileController$initInternal}}:
> {code:java}
> this.remoteRootLogDir = new Path(
> conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
> {code}
> So the remoteRootLogDir is only aware of the
> yarn.nodemanager.remote-app-log-dir config, while other file format, like
> IFile defaults to the file format config, so its priority is higher.
> From {{LogAggregationIndexedFileController$initInternal}}:
> {code:java}
> String remoteDirStr = String.format(
> YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT,
> this.fileControllerName);
> String remoteDir = conf.get(remoteDirStr);
> if (remoteDir == null || remoteDir.isEmpty()) {
> remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR);
> }
> {code}
> (Where these configs are: )
> {code:java}
> public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT
> = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir";
> public static final String NM_REMOTE_APP_LOG_DIR =
> NM_PREFIX + "remote-app-log-dir";
> {code}
> I suggest TFile should try to obtain the remote dir config from
> yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not
> specified falls back to the yarn.nodemanager.remote-app-log-dir config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9519) TFile log aggregation file format is insensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config
[
https://issues.apache.org/jira/browse/YARN-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832469#comment-16832469
]
Adam Antal commented on YARN-9519:
--
Uploaded patch v2 with some tests.
Unfortunately I couldn't touch the tests without refactoring it a bit. Could
you please also check on the tests as well [~snemeth]?
> TFile log aggregation file format is insensitive to the
> yarn.log-aggregation.TFile.remote-app-log-dir config
>
>
> Key: YARN-9519
> URL: https://issues.apache.org/jira/browse/YARN-9519
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9519.001.patch, YARN-9519.002.patch
>
>
> The TFile log aggregation file format is not sensitive to the
> yarn.log-aggregation.TFile.remote-app-log-dir config.
> In {{LogAggregationTFileController$initInternal}}:
> {code:java}
> this.remoteRootLogDir = new Path(
> conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
> {code}
> So the remoteRootLogDir is only aware of the
> yarn.nodemanager.remote-app-log-dir config, while other file format, like
> IFile defaults to the file format config, so its priority is higher.
> From {{LogAggregationIndexedFileController$initInternal}}:
> {code:java}
> String remoteDirStr = String.format(
> YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT,
> this.fileControllerName);
> String remoteDir = conf.get(remoteDirStr);
> if (remoteDir == null || remoteDir.isEmpty()) {
> remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR);
> }
> {code}
> (Where these configs are: )
> {code:java}
> public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT
> = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir";
> public static final String NM_REMOTE_APP_LOG_DIR =
> NM_PREFIX + "remote-app-log-dir";
> {code}
> I suggest TFile should try to obtain the remote dir config from
> yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not
> specified falls back to the yarn.nodemanager.remote-app-log-dir config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9477) Implement VE discovery using libudev
[
https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832505#comment-16832505
]
Hadoop QA commented on YARN-9477:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 7s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
11s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | C
[jira] [Commented] (YARN-9519) TFile log aggregation file format is insensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config
[
https://issues.apache.org/jira/browse/YARN-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832500#comment-16832500
]
Hadoop QA commented on YARN-9519:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new +
24 unchanged - 3 fixed = 25 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
49s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9519 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967782/YARN-9519.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 54cf2d0c836d 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f1875b2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24046/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24046/testReport/ |
| Max. process+thread count | 444 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job
[jira] [Comment Edited] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832510#comment-16832510 ] Peter Bacsko edited comment on YARN-9477 at 5/3/19 1:45 PM: Test failure seems to be unrelated to this patch. Instead, it was most likely caused by a recent commit (YARN-6929). [~Prabhu Joseph] could you please investigate this test error? https://builds.apache.org/job/PreCommit-YARN-Build/24045/testReport/org.apache.hadoop.yarn.server.applicationhistoryservice.webapp/TestAHSWebServices/testContainerLogsMetaForFinishedApps_0_/ was (Author: pbacsko): Test failure seems to be unrelated to this patch. Instead, it was most likely caused by a recent commit (YARN-6929). [~Prabhu Joseph] could you please investigate this test failure? https://builds.apache.org/job/PreCommit-YARN-Build/24045/testReport/org.apache.hadoop.yarn.server.applicationhistoryservice.webapp/TestAHSWebServices/testContainerLogsMetaForFinishedApps_0_/ > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832510#comment-16832510 ] Peter Bacsko commented on YARN-9477: Test failure seems to be unrelated to this patch. Instead, it was most likely caused by a recent commit (YARN-6929). [~Prabhu Joseph] could you please investigate this test failure? https://builds.apache.org/job/PreCommit-YARN-Build/24045/testReport/org.apache.hadoop.yarn.server.applicationhistoryservice.webapp/TestAHSWebServices/testContainerLogsMetaForFinishedApps_0_/ > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated YARN-9285: Attachment: YARN-9285-branch-3.0.001.patch > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated YARN-9285: Affects Version/s: 3.0.0 > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated YARN-9285: Fix Version/s: 2.9.3 3.2.1 2.8.6 3.3.0 3.0.4 > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Fix For: 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3 > > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9285) RM UI progress column is of wrong type
[
https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832566#comment-16832566
]
Hadoop QA commented on YARN-9285:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.0 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
28s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} branch-3.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 3 new + 3 unchanged - 1 fixed = 6 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
7s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:e402791 |
| JIRA Issue | YARN-9285 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967786/YARN-9285-branch-3.0.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux de74da246f07 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.0 / c20d61c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24047/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24047/testRepor
[jira] [Commented] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832570#comment-16832570 ] Prabhu Joseph commented on YARN-9477: - [~pbacsko] Yes this is introduced by YARN-6929. Will be fixed by YARN-9524. > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9527) Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file
[ https://issues.apache.org/jira/browse/YARN-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan reassigned YARN-9527: - Assignee: Jim Brennan > Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file > - > > Key: YARN-9527 > URL: https://issues.apache.org/jira/browse/YARN-9527 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.8.5, 3.1.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > > A rogue ContainerLocalizer can get stuck in a loop continuously downloading > the same file while generating an "Invalid event: LOCALIZED at LOCALIZED" > exception on each iteration. Sometimes this continues long enough that it > fills up a disk or depletes available inodes for the filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-9285: - Fix Version/s: 3.1.3 2.10.0 > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[
https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832675#comment-16832675
]
Eric Yang commented on YARN-9524:
-
[~Prabhu Joseph] In my system, the log files generated for end user in
/mr-history/tmp/hbase is permission 770. This prevents the log file to be
readable by mapred user. The stacktrace looks like this
{code}
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
Permission denied: user=mapred, access=READ_EXECUTE,
inode="/mr-history/tmp/hbase":hbase:hdfs:drwxrwx---
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:261)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1874)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1858)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1808)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:79)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3883)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:1184)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:732)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:531)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:944)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:879)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2828)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1495)
at org.apache.hadoop.ipc.Client.call(Client.java:1392)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy12.getListing(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:678)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy13.getListing(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1665)
... 78 more
{code}
Umask on the system is default to 022. It looks like logs are created with
APP_DIR_PERMISSIONS and it prevents mapred user to read the log files when
hbase user is the owner of the logs and /mr-history is created without setting
to mapred user's primary group. This leads to mapred user has no access to log
file.
> TestAHSWebServices and TestLogsCLI test case failures
> -
>
> Key: YARN-9524
> URL: https://issues.apache.org/jira/browse/YARN-9524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: Regression
>
[jira] [Commented] (YARN-9523) Build application catalog docker image as part of hadoop dist build
[
https://issues.apache.org/jira/browse/YARN-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832684#comment-16832684
]
Hadoop QA commented on YARN-9523:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
12s{color} | {color:green} hadoop-yarn-applications-catalog-docker in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9523 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967721/YARN-9523.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux d75bdee06a59 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f1875b2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24048/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24048/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Build application catalog docker image as part of hadoop dist build
> ---
>
> Key: YARN-9523
> URL: https://issues.apache.org/jira/browse/YA
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[
https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832688#comment-16832688
]
Eric Yang commented on YARN-9524:
-
If I manually perform:
{code}
sudo -u hdfs /usr/local/hadoop-3.3.0-SNAPSHOT/bin/hdfs dfs -mkdir /mr-history
sudo -u hdfs /usr/local/hadoop-3.3.0-SNAPSHOT/bin/hdfs dfs -chmod 770
/mr-history
sudo -u hdfs /usr/local/hadoop-3.3.0-SNAPSHOT/bin/hdfs dfs -chown mapred:hadoop
/mr-history
{code}
This will allow logs to be stored in /mr-history. It looks like the directory
permission problem only occurs, if log aggregation for a job happened sooner
than permission of the directory is setup properly. Do we need to make sure
that log aggregation setup directory permission when directory doesn't exist?
> TestAHSWebServices and TestLogsCLI test case failures
> -
>
> Key: YARN-9524
> URL: https://issues.apache.org/jira/browse/YARN-9524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: Regression
> Attachments: Fetches_Logs.png, RM-Tracking-URL.png,
> Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch
>
>
> {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929.
> {code:java}
> [ERROR] Failures:
> [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014
> [ERROR] Errors:
> [ERROR]
> TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543
> » NullPointer
> [INFO]
> [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1
> [ERROR] Failures:
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] Errors:
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[
https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832744#comment-16832744
]
Prabhu Joseph commented on YARN-9524:
-
[~eyang] MR History Intermediate (/mr-history/tmp) and done (/mr-history/done)
directories are handled separately by {{JobHistoryServer}} and not related to
Log Aggregation.
The Intermediate user directory (/mr-history/tmp/) permission can be
configured by mapreduce.jobhistory.intermediate-user-done-dir.permissions
(Mapreduce-7010). But the jhist summary file permissions are hardcoded to 770
(HISTORY_INTERMEDIATE_FILE_PERMISSIONS). So either mapred user has to own the
/mr-history or belongs to the group of the job user. Have reported
MAPREDUCE-7201 to make files permission also configurable.
This issue is an existing one and not related to our fix. Have validated with
reproducing the issue without the fix.
> TestAHSWebServices and TestLogsCLI test case failures
> -
>
> Key: YARN-9524
> URL: https://issues.apache.org/jira/browse/YARN-9524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: Regression
> Attachments: Fetches_Logs.png, RM-Tracking-URL.png,
> Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch
>
>
> {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929.
> {code:java}
> [ERROR] Failures:
> [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014
> [ERROR] Errors:
> [ERROR]
> TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543
> » NullPointer
> [INFO]
> [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1
> [ERROR] Failures:
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] Errors:
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9523) Build application catalog docker image as part of hadoop dist build
[
https://issues.apache.org/jira/browse/YARN-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832746#comment-16832746
]
Eric Badger commented on YARN-9523:
---
bq. Jonathan Eagles no one disputed against using "dist" profile for building
docker image according to the email thread
The email thread wasn't about the use of the dist profile, though. So I don't
think it's fair to say that their silence equals their implicit agreement.
That being said, I think this is a more interesting conversation. I can see the
argument around putting a docker image in the distribution profile. This would
then spawn the conversation around whether or not we want to distribute a
hadoop docker image at all. I'm not familiar on if such conversations have
happened in the past.
However, regardless of whether we want to build them or not, I don't think that
we should break existing workflows that are building hadoop themselves and
distributing the tarballs within their own companies. Fundamentally changing
what the "dist" profile is does exactly that. I think that doing this in a
minor release has the potential to cause a lot of headaches for people for
something that is a new feature. I would rather that the minority of people
adopting this new feature have to add a config option (e.g. -DBuildDocker) than
forcing a larger group to have to change their current setup.
> Build application catalog docker image as part of hadoop dist build
> ---
>
> Key: YARN-9523
> URL: https://issues.apache.org/jira/browse/YARN-9523
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9523.001.patch
>
>
> It would be nice to make Application catalog docker image as part of the
> distribution. The suggestion is to change from:
> {code:java}
> mvn clean package -Pnative,dist,docker{code}
> to
> {code:java}
> mvn clean package -Pnative,dist{code}
> User can still build tarball only using:
> {code:java}
> mvn clean package -DskipDocker -DskipTests -DskipShade -Pnative,dist{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
Young Chen created YARN-9528: Summary: Federation RMs starting up at the same time can give duplicate application IDs Key: YARN-9528 URL: https://issues.apache.org/jira/browse/YARN-9528 Project: Hadoop YARN Issue Type: Bug Reporter: Young Chen Assignee: Young Chen Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9523) Build application catalog docker image as part of hadoop dist build
[
https://issues.apache.org/jira/browse/YARN-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832837#comment-16832837
]
Eric Yang commented on YARN-9523:
-
[~ebadger] {quote}This would then spawn the conversation around whether or not
we want to distribute a hadoop docker image at all. I'm not familiar on if such
conversations have happened in the past.{quote}
Official Hadoop docker image discussion had happen in the past in HADOOP-14898.
Although the current implementation fail short of expectation, the work is
there and there is a image uploaded to dockerhub as hadoop-runner. I am not
sure how to reproduce for consecutive release because the current image is
missing Hadoop binaries in the docker image. I think it is broken, and needs
attention for repair.
If we want to make the release process consistent, it would be good to have
this patch. I can work around blockades in my local environment when community
doesn't reach consensus. The patch is here, if there is ever a need for change.
> Build application catalog docker image as part of hadoop dist build
> ---
>
> Key: YARN-9523
> URL: https://issues.apache.org/jira/browse/YARN-9523
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-9523.001.patch
>
>
> It would be nice to make Application catalog docker image as part of the
> distribution. The suggestion is to change from:
> {code:java}
> mvn clean package -Pnative,dist,docker{code}
> to
> {code:java}
> mvn clean package -Pnative,dist{code}
> User can still build tarball only using:
> {code:java}
> mvn clean package -DskipDocker -DskipTests -DskipShade -Pnative,dist{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-9528: - Attachment: YARN-9528.01.patch > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[
https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832860#comment-16832860
]
Hadoop QA commented on YARN-9528:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967813/YARN-9528.01.patch |
| Optional Tests | dupname asflicense |
| uname | Linux d18b10bf0770 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f194540 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 413 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24049/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Federation RMs starting up at the same time can give duplicate application IDs
> --
>
> Key: YARN-9528
> URL: https://issues.apache.org/jira/browse/YARN-9528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Young Chen
>Assignee: Young Chen
>Priority: Minor
> Attachments: YARN-9528.01.patch
>
>
> Federation RMs starting up at the same time can give duplicate application IDs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832870#comment-16832870 ] Giovanni Matteo Fumarola commented on YARN-9528: Thanks [~youchen]. +1 on [^YARN-9528.01.patch] Committed to trunk. > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9528: --- Fix Version/s: 3.3.0 > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9518) can't use CGroups with YARN in centos7
[
https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832878#comment-16832878
]
Jonathan Hung commented on YARN-9518:
-
[~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it
for us.
> can't use CGroups with YARN in centos7
> ---
>
> Key: YARN-9518
> URL: https://issues.apache.org/jira/browse/YARN-9518
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0, 2.9.2, 2.8.5, 2.7.7, 3.1.2
>Reporter: Shurong Mai
>Priority: Major
> Labels: cgroup, patch
> Attachments: YARN-9518.patch
>
>
> The os version is centos7.
> {code:java}
> cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
> {code}
> When I had set configuration variables for cgroup with yarn, nodemanager
> could be start without any matter. But when I ran a job, the job failed with
> these exceptional nodemanager logs in the end.
> In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> After I analysed, I found the reason. In centos6, the cgroup "cpu" and
> "cpuacct" subsystem are as follows:
> {code:java}
> /sys/fs/cgroup/cpu
> /sys/fs/cgroup/cpuacct
> {code}
> But in centos7, as follows:
> {code:java}
> /sys/fs/cgroup/cpu -> cpu,cpuacct
> /sys/fs/cgroup/cpuacct -> cpu,cpuacct
> /sys/fs/cgroup/cpu,cpuacct{code}
> "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are
> symbol links.
> As I look at source code, nodemamager get the cgroup subsystem info by
> reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also
> "/sys/fs/cgroup/cpu,cpuacct".
> The resource description arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> There is a comma in the cgroup path, but the comma is separator of multi
> resource. Therefore, the cgroup path is truncated by container-executor as
> "/sys/fs/cgroup/cpu" rather than correct cgroup path "
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> Hence I modify the source code and submit a patch. The idea of patch is that
> nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than
> "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description
> arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> Note that there is no comma in the path, and is a valid path because
> "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct".
> After applied the patch, the problem is resolved and the job can run
> successfully.
> The patch is universally applicable to cgroup subsystem paths, such as cgroup
> network subsystem as follows:
> {code:java}
> /sys/fs/cgroup/net_cls -> net_cls,net_prio
> /sys/fs/cgroup/net_prio -> net_cls,net_prio
> /sys/fs/cgroup/net_cls,net_prio{code}
>
>
> ##
> {panel:title=exceptional nodemanager logs:}
> 2019-04-19 20:17:20,095 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1554210318404_0042_01_01 transitioned from LOCALIZED
> to RUNNING
> 2019-04-19 20:17:20,101 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_1554210318404_0042_01_01 is : 27
> 2019-04-19 20:17:20,103 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
> from container-launch with container ID: container_155421031840
> 4_0042_01_01 and exit code: 27
> ExitCodeException exitCode=27:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
> at org.apache.hadoop.util.Shell.run(Shell.java:482)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.Thread
[jira] [Commented] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832877#comment-16832877 ] Hudson commented on YARN-9528: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16502 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16502/]) YARN-9528. Federation RMs starting up at the same time can give (gifuma: rev d331a2a0c2e430030bf82ba44b677b13cc8e8bc5) * (edit) hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreTables.sql * (edit) hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9518) can't use CGroups with YARN in centos7
[
https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832878#comment-16832878
]
Jonathan Hung edited comment on YARN-9518 at 5/3/19 10:23 PM:
--
[~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it
for us. (we're running 2.7.4)
was (Author: jhung):
[~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it
for us.
> can't use CGroups with YARN in centos7
> ---
>
> Key: YARN-9518
> URL: https://issues.apache.org/jira/browse/YARN-9518
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0, 2.9.2, 2.8.5, 2.7.7, 3.1.2
>Reporter: Shurong Mai
>Priority: Major
> Labels: cgroup, patch
> Attachments: YARN-9518.patch
>
>
> The os version is centos7.
> {code:java}
> cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
> {code}
> When I had set configuration variables for cgroup with yarn, nodemanager
> could be start without any matter. But when I ran a job, the job failed with
> these exceptional nodemanager logs in the end.
> In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> After I analysed, I found the reason. In centos6, the cgroup "cpu" and
> "cpuacct" subsystem are as follows:
> {code:java}
> /sys/fs/cgroup/cpu
> /sys/fs/cgroup/cpuacct
> {code}
> But in centos7, as follows:
> {code:java}
> /sys/fs/cgroup/cpu -> cpu,cpuacct
> /sys/fs/cgroup/cpuacct -> cpu,cpuacct
> /sys/fs/cgroup/cpu,cpuacct{code}
> "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are
> symbol links.
> As I look at source code, nodemamager get the cgroup subsystem info by
> reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also
> "/sys/fs/cgroup/cpu,cpuacct".
> The resource description arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> There is a comma in the cgroup path, but the comma is separator of multi
> resource. Therefore, the cgroup path is truncated by container-executor as
> "/sys/fs/cgroup/cpu" rather than correct cgroup path "
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> Hence I modify the source code and submit a patch. The idea of patch is that
> nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than
> "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description
> arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> Note that there is no comma in the path, and is a valid path because
> "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct".
> After applied the patch, the problem is resolved and the job can run
> successfully.
> The patch is universally applicable to cgroup subsystem paths, such as cgroup
> network subsystem as follows:
> {code:java}
> /sys/fs/cgroup/net_cls -> net_cls,net_prio
> /sys/fs/cgroup/net_prio -> net_cls,net_prio
> /sys/fs/cgroup/net_cls,net_prio{code}
>
>
> ##
> {panel:title=exceptional nodemanager logs:}
> 2019-04-19 20:17:20,095 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1554210318404_0042_01_01 transitioned from LOCALIZED
> to RUNNING
> 2019-04-19 20:17:20,101 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_1554210318404_0042_01_01 is : 27
> 2019-04-19 20:17:20,103 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
> from container-launch with container ID: container_155421031840
> 4_0042_01_01 and exit code: 27
> ExitCodeException exitCode=27:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
> at org.apache.hadoop.util.Shell.run(Shell.java:482)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concu
[jira] [Created] (YARN-9529) Log correct cpu controller path on error
Jonathan Hung created YARN-9529: --- Summary: Log correct cpu controller path on error Key: YARN-9529 URL: https://issues.apache.org/jira/browse/YARN-9529 Project: Hadoop YARN Issue Type: Improvement Reporter: Jonathan Hung Assignee: Jonathan Hung Attachments: YARN-9529.001.patch The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9529) Log correct cpu controller path on error
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9529: Attachment: YARN-9529.001.patch > Log correct cpu controller path on error > > > Key: YARN-9529 > URL: https://issues.apache.org/jira/browse/YARN-9529 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9529.001.patch > > > The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[
https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832922#comment-16832922
]
Eric Yang commented on YARN-9524:
-
[~Prabhu Joseph] Sounds fair. I will commit patch 002, if no objections.
> TestAHSWebServices and TestLogsCLI test case failures
> -
>
> Key: YARN-9524
> URL: https://issues.apache.org/jira/browse/YARN-9524
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: Regression
> Attachments: Fetches_Logs.png, RM-Tracking-URL.png,
> Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch
>
>
> {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929.
> {code:java}
> [ERROR] Failures:
> [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014
> [ERROR] Errors:
> [ERROR]
> TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543
> » NullPointer
> [INFO]
> [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1
> [ERROR] Failures:
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624
> [ERROR] Errors:
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...
> [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 »
> WebApplication j...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error
[
https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832937#comment-16832937
]
Hadoop QA commented on YARN-9529:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 19s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9529 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967818/YARN-9529.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7a15c7ab419d 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d331a2a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/24050/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24050/testReport/ |
| Max. process+thread count | 468 (vs. ulimit of 1) |
| modules | C
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error
[
https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832960#comment-16832960
]
Jonathan Hung commented on YARN-9529:
-
Thanks! This was when reconfiguring
{{yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}. The previous
log prints this dir's parent dir which was never writable, so the NM should
never have been able to start up. But it's checking the writability of this
configured dir (not its parent), hence the logging change.
> Log correct cpu controller path on error
>
>
> Key: YARN-9529
> URL: https://issues.apache.org/jira/browse/YARN-9529
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9529.001.patch
>
>
> The base cpu controller path is logged instead of the hadoop cgroup path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832958#comment-16832958 ] Anthony Hsu commented on YARN-9529: --- +1 (non-binding). What were you trying to debug that requires this change? > Log correct cpu controller path on error > > > Key: YARN-9529 > URL: https://issues.apache.org/jira/browse/YARN-9529 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9529.001.patch > > > The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9517) When aggregation is not enabled, can't see the container log
[
https://issues.apache.org/jira/browse/YARN-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832991#comment-16832991
]
Shurong Mai commented on YARN-9517:
---
Hi, [~wangda] ， I just thought the problem was resolved by the patch, so I
closed this Jira issue. I haven't commit the patch to these branches.
> When aggregation is not enabled, can't see the container log
>
>
> Key: YARN-9517
> URL: https://issues.apache.org/jira/browse/YARN-9517
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.3.0, 2.4.1, 2.5.2, 2.6.5, 3.2.0, 2.9.2, 2.8.5,
> 2.7.7, 3.1.2
>Reporter: Shurong Mai
>Priority: Major
> Labels: patch
> Attachments: YARN-9517.patch
>
>
> yarn-site.xml
> {code:java}
>
> yarn.log-aggregation-enable
> false
>
> {code}
>
> When aggregation is not enabled, we click the "container log link"(in web
> page
> "http://xx:19888/jobhistory/attempts/job_1556431770792_0001/m/SUCCESSFUL";)
> after a job is finished successfully.
> It will jump to the webpage displaying "Aggregation is not enabled. Try the
> nodemanager at yy:48038" after we click, and the url is
> "http://xx:19888/jobhistory/logs/yy:48038/container_1556431770792_0001_01_02/attempt_1556431770792_0001_m_00_0/hadoop";
> I also fund this problem in all hadoop version 2.x.y and 3.x.y and I submit
> a patch which is simple and can apply to this hadoop version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
