[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[ https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832295#comment-16832295 ] Prabhu Joseph commented on YARN-9524: - [~eyang] Thanks for checking this. Have tested YARN-6929 + YARN-9524 fix and did not observe any issue, Tracking URL from RM UI redirects correctly to HistoryServer and which fetches the app logs as expected. Attached screenshots for reference. Redirect URL works for both the jobs running before and after the fix. But have faced issues initially when HistoryServer and ResourceManager not restarted after applying the fix with only NodeManager was restarted. Can we check if both ResourceManager and HistoryServer restarted after the fix. > TestAHSWebServices and TestLogsCLI test case failures > - > > Key: YARN-9524 > URL: https://issues.apache.org/jira/browse/YARN-9524 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, test >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: Regression > Attachments: YARN-9524-001.patch, YARN-9524-002.patch > > > {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929. > {code:java} > [ERROR] Failures: > [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014 > [ERROR] Errors: > [ERROR] > TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543 > » NullPointer > [INFO] > [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1 > [ERROR] Failures: > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] Errors: > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j... > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j...{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[ https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9524: Attachment: RM-Tracking-URL.png Redirects_HistoryServer.png Fetches_Logs.png > TestAHSWebServices and TestLogsCLI test case failures > - > > Key: YARN-9524 > URL: https://issues.apache.org/jira/browse/YARN-9524 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, test >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: Regression > Attachments: Fetches_Logs.png, RM-Tracking-URL.png, > Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch > > > {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929. > {code:java} > [ERROR] Failures: > [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014 > [ERROR] Errors: > [ERROR] > TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543 > » NullPointer > [INFO] > [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1 > [ERROR] Failures: > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] Errors: > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j... > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j...{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[ https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832301#comment-16832301 ] Hadoop QA commented on YARN-9524: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color} | {color:red} YARN-9524 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-9524 | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/24044/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > TestAHSWebServices and TestLogsCLI test case failures > - > > Key: YARN-9524 > URL: https://issues.apache.org/jira/browse/YARN-9524 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, test >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: Regression > Attachments: Fetches_Logs.png, RM-Tracking-URL.png, > Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch > > > {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929. > {code:java} > [ERROR] Failures: > [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014 > [ERROR] Errors: > [ERROR] > TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543 > » NullPointer > [INFO] > [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1 > [ERROR] Failures: > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] Errors: > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j... > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j...{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9477: --- Attachment: YARN-9477-002.patch > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9519) TFile log aggregation file format is insensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config
[ https://issues.apache.org/jira/browse/YARN-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated YARN-9519: - Attachment: YARN-9519.002.patch > TFile log aggregation file format is insensitive to the > yarn.log-aggregation.TFile.remote-app-log-dir config > > > Key: YARN-9519 > URL: https://issues.apache.org/jira/browse/YARN-9519 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9519.001.patch, YARN-9519.002.patch > > > The TFile log aggregation file format is not sensitive to the > yarn.log-aggregation.TFile.remote-app-log-dir config. > In {{LogAggregationTFileController$initInternal}}: > {code:java} > this.remoteRootLogDir = new Path( > conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, > YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR)); > {code} > So the remoteRootLogDir is only aware of the > yarn.nodemanager.remote-app-log-dir config, while other file format, like > IFile defaults to the file format config, so its priority is higher. > From {{LogAggregationIndexedFileController$initInternal}}: > {code:java} > String remoteDirStr = String.format( > YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT, > this.fileControllerName); > String remoteDir = conf.get(remoteDirStr); > if (remoteDir == null || remoteDir.isEmpty()) { > remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, > YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR); > } > {code} > (Where these configs are: ) > {code:java} > public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT > = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir"; > public static final String NM_REMOTE_APP_LOG_DIR = > NM_PREFIX + "remote-app-log-dir"; > {code} > I suggest TFile should try to obtain the remote dir config from > yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not > specified falls back to the yarn.nodemanager.remote-app-log-dir config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9519) TFile log aggregation file format is insensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config
[ https://issues.apache.org/jira/browse/YARN-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832469#comment-16832469 ] Adam Antal commented on YARN-9519: -- Uploaded patch v2 with some tests. Unfortunately I couldn't touch the tests without refactoring it a bit. Could you please also check on the tests as well [~snemeth]? > TFile log aggregation file format is insensitive to the > yarn.log-aggregation.TFile.remote-app-log-dir config > > > Key: YARN-9519 > URL: https://issues.apache.org/jira/browse/YARN-9519 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Affects Versions: 3.2.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-9519.001.patch, YARN-9519.002.patch > > > The TFile log aggregation file format is not sensitive to the > yarn.log-aggregation.TFile.remote-app-log-dir config. > In {{LogAggregationTFileController$initInternal}}: > {code:java} > this.remoteRootLogDir = new Path( > conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, > YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR)); > {code} > So the remoteRootLogDir is only aware of the > yarn.nodemanager.remote-app-log-dir config, while other file format, like > IFile defaults to the file format config, so its priority is higher. > From {{LogAggregationIndexedFileController$initInternal}}: > {code:java} > String remoteDirStr = String.format( > YarnConfiguration.LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT, > this.fileControllerName); > String remoteDir = conf.get(remoteDirStr); > if (remoteDir == null || remoteDir.isEmpty()) { > remoteDir = conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, > YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR); > } > {code} > (Where these configs are: ) > {code:java} > public static final String LOG_AGGREGATION_REMOTE_APP_LOG_DIR_FMT > = YARN_PREFIX + "log-aggregation.%s.remote-app-log-dir"; > public static final String NM_REMOTE_APP_LOG_DIR = > NM_PREFIX + "remote-app-log-dir"; > {code} > I suggest TFile should try to obtain the remote dir config from > yarn.log-aggregation.TFile.remote-app-log-dir first, and only if that is not > specified falls back to the yarn.nodemanager.remote-app-log-dir config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832505#comment-16832505 ] Hadoop QA commented on YARN-9477: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 5s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 27s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 54s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 43s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 7s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 11s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 47s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}159m 14s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices | \\ \\ || Subsystem || Report/Notes || | Docker | C
[jira] [Commented] (YARN-9519) TFile log aggregation file format is insensitive to the yarn.log-aggregation.TFile.remote-app-log-dir config
[ https://issues.apache.org/jira/browse/YARN-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832500#comment-16832500 ] Hadoop QA commented on YARN-9519: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 43s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 22s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new + 24 unchanged - 3 fixed = 25 total (was 27) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 34s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 49s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 54m 48s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | YARN-9519 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967782/YARN-9519.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 54cf2d0c836d 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f1875b2 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/24046/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24046/testReport/ | | Max. process+thread count | 444 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common | | Console output | https://builds.apache.org/job
[jira] [Comment Edited] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832510#comment-16832510 ] Peter Bacsko edited comment on YARN-9477 at 5/3/19 1:45 PM: Test failure seems to be unrelated to this patch. Instead, it was most likely caused by a recent commit (YARN-6929). [~Prabhu Joseph] could you please investigate this test error? https://builds.apache.org/job/PreCommit-YARN-Build/24045/testReport/org.apache.hadoop.yarn.server.applicationhistoryservice.webapp/TestAHSWebServices/testContainerLogsMetaForFinishedApps_0_/ was (Author: pbacsko): Test failure seems to be unrelated to this patch. Instead, it was most likely caused by a recent commit (YARN-6929). [~Prabhu Joseph] could you please investigate this test failure? https://builds.apache.org/job/PreCommit-YARN-Build/24045/testReport/org.apache.hadoop.yarn.server.applicationhistoryservice.webapp/TestAHSWebServices/testContainerLogsMetaForFinishedApps_0_/ > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832510#comment-16832510 ] Peter Bacsko commented on YARN-9477: Test failure seems to be unrelated to this patch. Instead, it was most likely caused by a recent commit (YARN-6929). [~Prabhu Joseph] could you please investigate this test failure? https://builds.apache.org/job/PreCommit-YARN-Build/24045/testReport/org.apache.hadoop.yarn.server.applicationhistoryservice.webapp/TestAHSWebServices/testContainerLogsMetaForFinishedApps_0_/ > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated YARN-9285: Attachment: YARN-9285-branch-3.0.001.patch > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated YARN-9285: Affects Version/s: 3.0.0 > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated YARN-9285: Fix Version/s: 2.9.3 3.2.1 2.8.6 3.3.0 3.0.4 > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Fix For: 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3 > > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832566#comment-16832566 ] Hadoop QA commented on YARN-9285: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} branch-3.0 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 28s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 51s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} branch-3.0 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 11s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 3 new + 3 unchanged - 1 fixed = 6 total (was 4) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 45s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 60m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:e402791 | | JIRA Issue | YARN-9285 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967786/YARN-9285-branch-3.0.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux de74da246f07 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | branch-3.0 / c20d61c | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/24047/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24047/testRepor
[jira] [Commented] (YARN-9477) Implement VE discovery using libudev
[ https://issues.apache.org/jira/browse/YARN-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832570#comment-16832570 ] Prabhu Joseph commented on YARN-9477: - [~pbacsko] Yes this is introduced by YARN-6929. Will be fixed by YARN-9524. > Implement VE discovery using libudev > > > Key: YARN-9477 > URL: https://issues.apache.org/jira/browse/YARN-9477 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9477-001.patch, YARN-9477-002.patch, > YARN-9477-POC.patch, YARN-9477-POC2.patch, YARN-9477-POC3.patch > > > Right now we have a Python script which is able to discover VE cards using > pyudev: https://pyudev.readthedocs.io/en/latest/ > Java does not officially support libudev. There are some projects on Github > (example: https://github.com/Zubnix/udev-java-bindings) but they're not > available as Maven artifacts. > However it's not that difficult to create a minimal layer around libudev > using JNA. We don't have to wrap every function, we need to call 4-5 methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9527) Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file
[ https://issues.apache.org/jira/browse/YARN-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan reassigned YARN-9527: - Assignee: Jim Brennan > Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file > - > > Key: YARN-9527 > URL: https://issues.apache.org/jira/browse/YARN-9527 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.8.5, 3.1.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > > A rogue ContainerLocalizer can get stuck in a loop continuously downloading > the same file while generating an "Invalid event: LOCALIZED at LOCALIZED" > exception on each iteration. Sometimes this continues long enough that it > fills up a disk or depletes available inodes for the filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9285) RM UI progress column is of wrong type
[ https://issues.apache.org/jira/browse/YARN-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-9285: - Fix Version/s: 3.1.3 2.10.0 > RM UI progress column is of wrong type > -- > > Key: YARN-9285 > URL: https://issues.apache.org/jira/browse/YARN-9285 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.2, 2.8.6, 2.9.3 >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Labels: bug > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: YARN-9285-branch-2.8.001.patch, > YARN-9285-branch-2.8.002.patch, YARN-9285-branch-2.9.001.patch, > YARN-9285-branch-3.0.001.patch, YARN-9285-branch-3.001.patch, > YARN-9285.001.patch, YARN-9285.002.patch > > > The column type assigned for progress column in the application report is not > correct. > The rank of the progress column should be 16, and 18. In WebPageUtils.java > the "atargets" needs to be incremented by 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[ https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832675#comment-16832675 ] Eric Yang commented on YARN-9524: - [~Prabhu Joseph] In my system, the log files generated for end user in /mr-history/tmp/hbase is permission 770. This prevents the log file to be readable by mapred user. The stacktrace looks like this {code} Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=mapred, access=READ_EXECUTE, inode="/mr-history/tmp/hbase":hbase:hdfs:drwxrwx--- at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:261) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1874) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1858) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1808) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:79) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3883) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:1184) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:732) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:531) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:944) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:879) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2828) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549) at org.apache.hadoop.ipc.Client.call(Client.java:1495) at org.apache.hadoop.ipc.Client.call(Client.java:1392) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy12.getListing(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:678) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy13.getListing(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1665) ... 78 more {code} Umask on the system is default to 022. It looks like logs are created with APP_DIR_PERMISSIONS and it prevents mapred user to read the log files when hbase user is the owner of the logs and /mr-history is created without setting to mapred user's primary group. This leads to mapred user has no access to log file. > TestAHSWebServices and TestLogsCLI test case failures > - > > Key: YARN-9524 > URL: https://issues.apache.org/jira/browse/YARN-9524 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, test >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: Regression >
[jira] [Commented] (YARN-9523) Build application catalog docker image as part of hadoop dist build
[ https://issues.apache.org/jira/browse/YARN-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832684#comment-16832684 ] Hadoop QA commented on YARN-9523: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 29m 0s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s{color} | {color:green} hadoop-yarn-applications-catalog-docker in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 43m 47s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | YARN-9523 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967721/YARN-9523.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux d75bdee06a59 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f1875b2 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24048/testReport/ | | Max. process+thread count | 447 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/24048/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Build application catalog docker image as part of hadoop dist build > --- > > Key: YARN-9523 > URL: https://issues.apache.org/jira/browse/YA
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[ https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832688#comment-16832688 ] Eric Yang commented on YARN-9524: - If I manually perform: {code} sudo -u hdfs /usr/local/hadoop-3.3.0-SNAPSHOT/bin/hdfs dfs -mkdir /mr-history sudo -u hdfs /usr/local/hadoop-3.3.0-SNAPSHOT/bin/hdfs dfs -chmod 770 /mr-history sudo -u hdfs /usr/local/hadoop-3.3.0-SNAPSHOT/bin/hdfs dfs -chown mapred:hadoop /mr-history {code} This will allow logs to be stored in /mr-history. It looks like the directory permission problem only occurs, if log aggregation for a job happened sooner than permission of the directory is setup properly. Do we need to make sure that log aggregation setup directory permission when directory doesn't exist? > TestAHSWebServices and TestLogsCLI test case failures > - > > Key: YARN-9524 > URL: https://issues.apache.org/jira/browse/YARN-9524 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, test >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: Regression > Attachments: Fetches_Logs.png, RM-Tracking-URL.png, > Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch > > > {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929. > {code:java} > [ERROR] Failures: > [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014 > [ERROR] Errors: > [ERROR] > TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543 > » NullPointer > [INFO] > [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1 > [ERROR] Failures: > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] Errors: > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j... > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j...{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[ https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832744#comment-16832744 ] Prabhu Joseph commented on YARN-9524: - [~eyang] MR History Intermediate (/mr-history/tmp) and done (/mr-history/done) directories are handled separately by {{JobHistoryServer}} and not related to Log Aggregation. The Intermediate user directory (/mr-history/tmp/) permission can be configured by mapreduce.jobhistory.intermediate-user-done-dir.permissions (Mapreduce-7010). But the jhist summary file permissions are hardcoded to 770 (HISTORY_INTERMEDIATE_FILE_PERMISSIONS). So either mapred user has to own the /mr-history or belongs to the group of the job user. Have reported MAPREDUCE-7201 to make files permission also configurable. This issue is an existing one and not related to our fix. Have validated with reproducing the issue without the fix. > TestAHSWebServices and TestLogsCLI test case failures > - > > Key: YARN-9524 > URL: https://issues.apache.org/jira/browse/YARN-9524 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, test >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: Regression > Attachments: Fetches_Logs.png, RM-Tracking-URL.png, > Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch > > > {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929. > {code:java} > [ERROR] Failures: > [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014 > [ERROR] Errors: > [ERROR] > TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543 > » NullPointer > [INFO] > [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1 > [ERROR] Failures: > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] Errors: > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j... > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j...{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9523) Build application catalog docker image as part of hadoop dist build
[ https://issues.apache.org/jira/browse/YARN-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832746#comment-16832746 ] Eric Badger commented on YARN-9523: --- bq. Jonathan Eagles no one disputed against using "dist" profile for building docker image according to the email thread The email thread wasn't about the use of the dist profile, though. So I don't think it's fair to say that their silence equals their implicit agreement. That being said, I think this is a more interesting conversation. I can see the argument around putting a docker image in the distribution profile. This would then spawn the conversation around whether or not we want to distribute a hadoop docker image at all. I'm not familiar on if such conversations have happened in the past. However, regardless of whether we want to build them or not, I don't think that we should break existing workflows that are building hadoop themselves and distributing the tarballs within their own companies. Fundamentally changing what the "dist" profile is does exactly that. I think that doing this in a minor release has the potential to cause a lot of headaches for people for something that is a new feature. I would rather that the minority of people adopting this new feature have to add a config option (e.g. -DBuildDocker) than forcing a larger group to have to change their current setup. > Build application catalog docker image as part of hadoop dist build > --- > > Key: YARN-9523 > URL: https://issues.apache.org/jira/browse/YARN-9523 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9523.001.patch > > > It would be nice to make Application catalog docker image as part of the > distribution. The suggestion is to change from: > {code:java} > mvn clean package -Pnative,dist,docker{code} > to > {code:java} > mvn clean package -Pnative,dist{code} > User can still build tarball only using: > {code:java} > mvn clean package -DskipDocker -DskipTests -DskipShade -Pnative,dist{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
Young Chen created YARN-9528: Summary: Federation RMs starting up at the same time can give duplicate application IDs Key: YARN-9528 URL: https://issues.apache.org/jira/browse/YARN-9528 Project: Hadoop YARN Issue Type: Bug Reporter: Young Chen Assignee: Young Chen Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9523) Build application catalog docker image as part of hadoop dist build
[ https://issues.apache.org/jira/browse/YARN-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832837#comment-16832837 ] Eric Yang commented on YARN-9523: - [~ebadger] {quote}This would then spawn the conversation around whether or not we want to distribute a hadoop docker image at all. I'm not familiar on if such conversations have happened in the past.{quote} Official Hadoop docker image discussion had happen in the past in HADOOP-14898. Although the current implementation fail short of expectation, the work is there and there is a image uploaded to dockerhub as hadoop-runner. I am not sure how to reproduce for consecutive release because the current image is missing Hadoop binaries in the docker image. I think it is broken, and needs attention for repair. If we want to make the release process consistent, it would be good to have this patch. I can work around blockades in my local environment when community doesn't reach consensus. The patch is here, if there is ever a need for change. > Build application catalog docker image as part of hadoop dist build > --- > > Key: YARN-9523 > URL: https://issues.apache.org/jira/browse/YARN-9523 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9523.001.patch > > > It would be nice to make Application catalog docker image as part of the > distribution. The suggestion is to change from: > {code:java} > mvn clean package -Pnative,dist,docker{code} > to > {code:java} > mvn clean package -Pnative,dist{code} > User can still build tarball only using: > {code:java} > mvn clean package -DskipDocker -DskipTests -DskipShade -Pnative,dist{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-9528: - Attachment: YARN-9528.01.patch > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832860#comment-16832860 ] Hadoop QA commented on YARN-9528: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 48s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 2s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 2s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | YARN-9528 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967813/YARN-9528.01.patch | | Optional Tests | dupname asflicense | | uname | Linux d18b10bf0770 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f194540 | | maven | version: Apache Maven 3.3.9 | | Max. process+thread count | 413 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn U: hadoop-yarn-project/hadoop-yarn | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/24049/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832870#comment-16832870 ] Giovanni Matteo Fumarola commented on YARN-9528: Thanks [~youchen]. +1 on [^YARN-9528.01.patch] Committed to trunk. > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9528: --- Fix Version/s: 3.3.0 > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9518) can't use CGroups with YARN in centos7
[ https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832878#comment-16832878 ] Jonathan Hung commented on YARN-9518: - [~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it for us. > can't use CGroups with YARN in centos7 > --- > > Key: YARN-9518 > URL: https://issues.apache.org/jira/browse/YARN-9518 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 2.9.2, 2.8.5, 2.7.7, 3.1.2 >Reporter: Shurong Mai >Priority: Major > Labels: cgroup, patch > Attachments: YARN-9518.patch > > > The os version is centos7. > {code:java} > cat /etc/redhat-release > CentOS Linux release 7.3.1611 (Core) > {code} > When I had set configuration variables for cgroup with yarn, nodemanager > could be start without any matter. But when I ran a job, the job failed with > these exceptional nodemanager logs in the end. > In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as > node manager - Is a directory " > After I analysed, I found the reason. In centos6, the cgroup "cpu" and > "cpuacct" subsystem are as follows: > {code:java} > /sys/fs/cgroup/cpu > /sys/fs/cgroup/cpuacct > {code} > But in centos7, as follows: > {code:java} > /sys/fs/cgroup/cpu -> cpu,cpuacct > /sys/fs/cgroup/cpuacct -> cpu,cpuacct > /sys/fs/cgroup/cpu,cpuacct{code} > "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are > symbol links. > As I look at source code, nodemamager get the cgroup subsystem info by > reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also > "/sys/fs/cgroup/cpu,cpuacct". > The resource description arguments of container-executor is such as follows: > {code:java} > cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks > {code} > There is a comma in the cgroup path, but the comma is separator of multi > resource. Therefore, the cgroup path is truncated by container-executor as > "/sys/fs/cgroup/cpu" rather than correct cgroup path " > /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks > " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as > node manager - Is a directory " > Hence I modify the source code and submit a patch. The idea of patch is that > nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than > "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description > arguments of container-executor is such as follows: > {code:java} > cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks > {code} > Note that there is no comma in the path, and is a valid path because > "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct". > After applied the patch, the problem is resolved and the job can run > successfully. > The patch is universally applicable to cgroup subsystem paths, such as cgroup > network subsystem as follows: > {code:java} > /sys/fs/cgroup/net_cls -> net_cls,net_prio > /sys/fs/cgroup/net_prio -> net_cls,net_prio > /sys/fs/cgroup/net_cls,net_prio{code} > > > ## > {panel:title=exceptional nodemanager logs:} > 2019-04-19 20:17:20,095 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: > Container container_1554210318404_0042_01_01 transitioned from LOCALIZED > to RUNNING > 2019-04-19 20:17:20,101 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container container_1554210318404_0042_01_01 is : 27 > 2019-04-19 20:17:20,103 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch with container ID: container_155421031840 > 4_0042_01_01 and exit code: 27 > ExitCodeException exitCode=27: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.Thread
[jira] [Commented] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832877#comment-16832877 ] Hudson commented on YARN-9528: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16502 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16502/]) YARN-9528. Federation RMs starting up at the same time can give (gifuma: rev d331a2a0c2e430030bf82ba44b677b13cc8e8bc5) * (edit) hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreTables.sql * (edit) hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9518) can't use CGroups with YARN in centos7
[ https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832878#comment-16832878 ] Jonathan Hung edited comment on YARN-9518 at 5/3/19 10:23 PM: -- [~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it for us. (we're running 2.7.4) was (Author: jhung): [~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it for us. > can't use CGroups with YARN in centos7 > --- > > Key: YARN-9518 > URL: https://issues.apache.org/jira/browse/YARN-9518 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 2.9.2, 2.8.5, 2.7.7, 3.1.2 >Reporter: Shurong Mai >Priority: Major > Labels: cgroup, patch > Attachments: YARN-9518.patch > > > The os version is centos7. > {code:java} > cat /etc/redhat-release > CentOS Linux release 7.3.1611 (Core) > {code} > When I had set configuration variables for cgroup with yarn, nodemanager > could be start without any matter. But when I ran a job, the job failed with > these exceptional nodemanager logs in the end. > In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as > node manager - Is a directory " > After I analysed, I found the reason. In centos6, the cgroup "cpu" and > "cpuacct" subsystem are as follows: > {code:java} > /sys/fs/cgroup/cpu > /sys/fs/cgroup/cpuacct > {code} > But in centos7, as follows: > {code:java} > /sys/fs/cgroup/cpu -> cpu,cpuacct > /sys/fs/cgroup/cpuacct -> cpu,cpuacct > /sys/fs/cgroup/cpu,cpuacct{code} > "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are > symbol links. > As I look at source code, nodemamager get the cgroup subsystem info by > reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also > "/sys/fs/cgroup/cpu,cpuacct". > The resource description arguments of container-executor is such as follows: > {code:java} > cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks > {code} > There is a comma in the cgroup path, but the comma is separator of multi > resource. Therefore, the cgroup path is truncated by container-executor as > "/sys/fs/cgroup/cpu" rather than correct cgroup path " > /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks > " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as > node manager - Is a directory " > Hence I modify the source code and submit a patch. The idea of patch is that > nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than > "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description > arguments of container-executor is such as follows: > {code:java} > cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks > {code} > Note that there is no comma in the path, and is a valid path because > "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct". > After applied the patch, the problem is resolved and the job can run > successfully. > The patch is universally applicable to cgroup subsystem paths, such as cgroup > network subsystem as follows: > {code:java} > /sys/fs/cgroup/net_cls -> net_cls,net_prio > /sys/fs/cgroup/net_prio -> net_cls,net_prio > /sys/fs/cgroup/net_cls,net_prio{code} > > > ## > {panel:title=exceptional nodemanager logs:} > 2019-04-19 20:17:20,095 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: > Container container_1554210318404_0042_01_01 transitioned from LOCALIZED > to RUNNING > 2019-04-19 20:17:20,101 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container container_1554210318404_0042_01_01 is : 27 > 2019-04-19 20:17:20,103 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch with container ID: container_155421031840 > 4_0042_01_01 and exit code: 27 > ExitCodeException exitCode=27: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concu
[jira] [Created] (YARN-9529) Log correct cpu controller path on error
Jonathan Hung created YARN-9529: --- Summary: Log correct cpu controller path on error Key: YARN-9529 URL: https://issues.apache.org/jira/browse/YARN-9529 Project: Hadoop YARN Issue Type: Improvement Reporter: Jonathan Hung Assignee: Jonathan Hung Attachments: YARN-9529.001.patch The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9529) Log correct cpu controller path on error
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9529: Attachment: YARN-9529.001.patch > Log correct cpu controller path on error > > > Key: YARN-9529 > URL: https://issues.apache.org/jira/browse/YARN-9529 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9529.001.patch > > > The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9524) TestAHSWebServices and TestLogsCLI test case failures
[ https://issues.apache.org/jira/browse/YARN-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832922#comment-16832922 ] Eric Yang commented on YARN-9524: - [~Prabhu Joseph] Sounds fair. I will commit patch 002, if no objections. > TestAHSWebServices and TestLogsCLI test case failures > - > > Key: YARN-9524 > URL: https://issues.apache.org/jira/browse/YARN-9524 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, test >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: Regression > Attachments: Fetches_Logs.png, RM-Tracking-URL.png, > Redirects_HistoryServer.png, YARN-9524-001.patch, YARN-9524-002.patch > > > {{TestAHSWebServices}} and {{TestLogsCLI}} test case failures after YARN-6929. > {code:java} > [ERROR] Failures: > [ERROR] TestLogsCLI.testFetchApplictionLogsAsAnotherUser:1014 > [ERROR] Errors: > [ERROR] > TestLogsCLI.testFetchFinishedApplictionLogs:420->uploadEmptyContainerLogIntoRemoteDir:1543 > » NullPointer > [INFO] > [ERROR] Tests run: 339, Failures: 1, Errors: 1, Skipped: 1 > [ERROR] Failures: > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] TestAHSWebServices.testContainerLogsForFinishedApps:624 > [ERROR] Errors: > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j... > [ERROR] TestAHSWebServices.testContainerLogsMetaForFinishedApps:942 » > WebApplication j...{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832937#comment-16832937 ] Hadoop QA commented on YARN-9529: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 23s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 21s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 19s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 71m 44s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | YARN-9529 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967818/YARN-9529.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7a15c7ab419d 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d331a2a | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/24050/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24050/testReport/ | | Max. process+thread count | 468 (vs. ulimit of 1) | | modules | C
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832960#comment-16832960 ] Jonathan Hung commented on YARN-9529: - Thanks! This was when reconfiguring {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}. The previous log prints this dir's parent dir which was never writable, so the NM should never have been able to start up. But it's checking the writability of this configured dir (not its parent), hence the logging change. > Log correct cpu controller path on error > > > Key: YARN-9529 > URL: https://issues.apache.org/jira/browse/YARN-9529 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9529.001.patch > > > The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832958#comment-16832958 ] Anthony Hsu commented on YARN-9529: --- +1 (non-binding). What were you trying to debug that requires this change? > Log correct cpu controller path on error > > > Key: YARN-9529 > URL: https://issues.apache.org/jira/browse/YARN-9529 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9529.001.patch > > > The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9517) When aggregation is not enabled, can't see the container log
[ https://issues.apache.org/jira/browse/YARN-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832991#comment-16832991 ] Shurong Mai commented on YARN-9517: --- Hi, [~wangda] , I just thought the problem was resolved by the patch, so I closed this Jira issue. I haven't commit the patch to these branches. > When aggregation is not enabled, can't see the container log > > > Key: YARN-9517 > URL: https://issues.apache.org/jira/browse/YARN-9517 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.2.0, 2.3.0, 2.4.1, 2.5.2, 2.6.5, 3.2.0, 2.9.2, 2.8.5, > 2.7.7, 3.1.2 >Reporter: Shurong Mai >Priority: Major > Labels: patch > Attachments: YARN-9517.patch > > > yarn-site.xml > {code:java} > > yarn.log-aggregation-enable > false > > {code} > > When aggregation is not enabled, we click the "container log link"(in web > page > "http://xx:19888/jobhistory/attempts/job_1556431770792_0001/m/SUCCESSFUL";) > after a job is finished successfully. > It will jump to the webpage displaying "Aggregation is not enabled. Try the > nodemanager at yy:48038" after we click, and the url is > "http://xx:19888/jobhistory/logs/yy:48038/container_1556431770792_0001_01_02/attempt_1556431770792_0001_m_00_0/hadoop"; > I also fund this problem in all hadoop version 2.x.y and 3.x.y and I submit > a patch which is simple and can apply to this hadoop version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org