[jira] [Created] (YARN-5369) Improve Yarn logs command to get container logs based on Node Id
Xuan Gong created YARN-5369: --- Summary: Improve Yarn logs command to get container logs based on Node Id Key: YARN-5369 URL: https://issues.apache.org/jira/browse/YARN-5369 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong It is helpful if we could have yarn logs --applicationId appId --nodeAddress ${nodeId} to get all the container logs which ran on the specific nm. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5368) memory leak at timeline server
Wataru Yukawa created YARN-5368: --- Summary: memory leak at timeline server Key: YARN-5368 URL: https://issues.apache.org/jira/browse/YARN-5368 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.7.1 Environment: HDP2.4 CentOS 6.7 jdk1.8.0_72 Reporter: Wataru Yukawa memory usage of timeline server machine increases gradually. https://gyazo.com/952dad96c77ae053bae2e4d8c8ab0572 please check since April. According to my investigation, timeline server used about 25GB. top command result {code} 90577 yarn 20 0 28.4g 25g 12m S 0.0 40.1 5162:53 /usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.4.0.0-169 -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.dir=/var/log/hadoop-yarn/yarn ... {code} ps command result {code} $ ps ww 90577 90577 ?Sl 5162:53 /usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.4.0.0-169 -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.dir=/var/log/hadoop-yarn/yarn -Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA -Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir -Dyarn.policy.file=hadoop-policy.xml -Djava.io.tmpdir=/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn -Dyarn.log.dir=/var/log/hadoop-yarn/yarn -Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.home.dir=/usr/hdp/current/hadoop-yarn-timelineserver -Dhadoop.home.dir=/usr/hdp/2.4.0.0-169/hadoop -Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA -Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir -classpath /usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/*:/usr/hdp/2.4.0.0-169/hadoop/.//*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//*:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/*:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//*::/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/current/hadoop-yarn-timelineserver/.//*:/usr/hdp/current/hadoop-yarn-timelineserver/lib/*:/usr/hdp/2.4.0.0-169/hadoop/conf/timelineserver-config/log4j.properties org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer {code} Alghough I set -Xmx1024m, actual memory usage is 25GB. After I restart timeline server, memory usage of timeline server machine decreases. https://gyazo.com/130600c17a7d41df8606727a859ae7e3 Now timelineserver uses less than 1GB memory. top command result {code} 6163 yarn 20 0 3959m 783m 46m S 0.3 1.2 3:37.60 /usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.4.0.0-169 ... {code} I suspect memory leak at timeline server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5367) HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials
Xianyin Xin created YARN-5367: - Summary: HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials Key: YARN-5367 URL: https://issues.apache.org/jira/browse/YARN-5367 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Xianyin Xin Assignee: Xianyin Xin App log aggregation may failed because of the below flow: 0) suppose the token.max-lifetime is 7 days and renew interval is 1 day; 1) start a long running job, like sparkJDBC, of which the AM acts as a service. When submitting the job, HDFS token A in ApplicationSubmissionContext will be added to DelegationTokenRenewer, but not added to systemCredentials; 2) after 1 day, submit a spark query. After received the query, AM will request containers and start tasks. When start the containers, a new HDFS token B is used; 3) after 1 day, kill the job, when doing log aggregation, exception occurs which show token B is not in the HDFS token cache so the connecting to HDFS fails; We should add token A to systemCredentials to make sure token A can be delivered to NMs in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5366) Add support for toggling the removal of completed and failed docker containers
Shane Kumpf created YARN-5366: - Summary: Add support for toggling the removal of completed and failed docker containers Key: YARN-5366 URL: https://issues.apache.org/jira/browse/YARN-5366 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Shane Kumpf Currently, completed and failed docker containers are removed by container-executor. Add a job level environment variable to DockerLinuxContainerRuntime to allow the user to toggle whether they want the container deleted or not and remove the logic from container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5365) Add support for YARN Shared Cache
Ming Ma created YARN-5365: - Summary: Add support for YARN Shared Cache Key: YARN-5365 URL: https://issues.apache.org/jira/browse/YARN-5365 Project: Hadoop YARN Issue Type: Improvement Reporter: Ming Ma YARN provides shared cache in functionality YARN-1492. According to [~ctrezzo] most of the YARN functionality is in hadoop 2.8 and frameworks can start to use it. MR adds the support via MAPREDUCE-5951. Can anyone confirm if Tez supports the upload of application DAG jar and dependent lib jars from client machine to HDFS as part of Tez app submission? From my test, that doesn't seem to happen. Tez expects applications to upload the jars to HDFS beforehand and then set the tez.aux.uris to the HDFS locations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4366) Fix Lint Warnings in YARN Common
[ https://issues.apache.org/jira/browse/YARN-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter resolved YARN-4366. - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Thanks [~templedf]. Committed to trunk and branch-2! > Fix Lint Warnings in YARN Common > > > Key: YARN-4366 > URL: https://issues.apache.org/jira/browse/YARN-4366 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 2.9.0 > > Attachments: YARN-4366.001.patch > > > {noformat} > [WARNING] > /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java:[100,45] > non-varargs call of varargs method with inexact argument type for last > parameter; > cast to java.lang.Class for a varargs call > cast to java.lang.Class[] for a non-varargs call and to suppress this > warning > [WARNING] > /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[62,46] > non-varargs call of varargs method with inexact argument type for last > parameter; > cast to java.lang.Class for a varargs call > cast to java.lang.Class[] for a non-varargs call and to suppress this > warning > [WARNING] > /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[64,34] > non-varargs call of varargs method with inexact argument type for last > parameter; > cast to java.lang.Object for a varargs call > cast to java.lang.Object[] for a non-varargs call and to suppress this > warning > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
Sangjin Lee created YARN-5364: - Summary: timelineservice modules have indirect dependencies on mapreduce artifacts Key: YARN-5364 URL: https://issues.apache.org/jira/browse/YARN-5364 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 3.0.0-alpha1 Reporter: Sangjin Lee Assignee: Sangjin Lee Priority: Minor The new timelineservice and timelineservice-hbase-tests modules have indirect dependencies to mapreduce artifacts through HBase and phoenix. Although it's not causing builds to fail, it's not good hygiene. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5363) For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default
Vinod Kumar Vavilapalli created YARN-5363: - Summary: For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default Key: YARN-5363 URL: https://issues.apache.org/jira/browse/YARN-5363 Project: Hadoop YARN Issue Type: Sub-task Components: log-aggregation Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli For e.g, for a running application, the following happens: {code} # yarn logs -applicationId application_1467838922593_0001 16/07/06 22:07:05 INFO impl.TimelineClientImpl: Timeline service address: http://:8188/ws/v1/timeline/ 16/07/06 22:07:06 INFO client.RMProxy: Connecting to ResourceManager at /:8050 16/07/06 22:07:07 INFO impl.TimelineClientImpl: Timeline service address: http://l:8188/ws/v1/timeline/ 16/07/06 22:07:07 INFO client.RMProxy: Connecting to ResourceManager at /:8050 Can not find any log file matching the pattern: [syslog] for the container: container_e03_1467838922593_0001_01_01 within the application: application_1467838922593_0001 Can not find any log file matching the pattern: [syslog] for the container: container_e03_1467838922593_0001_01_02 within the application: application_1467838922593_0001 Can not find any log file matching the pattern: [syslog] for the container: container_e03_1467838922593_0001_01_03 within the application: application_1467838922593_0001 Can not find any log file matching the pattern: [syslog] for the container: container_e03_1467838922593_0001_01_04 within the application: application_1467838922593_0001 Can not find any log file matching the pattern: [syslog] for the container: container_e03_1467838922593_0001_01_05 within the application: application_1467838922593_0001 Can not find any log file matching the pattern: [syslog] for the container: container_e03_1467838922593_0001_01_06 within the application: application_1467838922593_0001 Can not find any log file matching the pattern: [syslog] for the container: container_e03_1467838922593_0001_01_07 within the application: application_1467838922593_0001 Can not find the logs for the application: application_1467838922593_0001 with the appOwner: {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
Jason Lowe created YARN-5362: Summary: TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail Key: YARN-5362 URL: https://issues.apache.org/jira/browse/YARN-5362 Project: Hadoop YARN Issue Type: Bug Reporter: Jason Lowe Saw the following in a precommit build that only changed an unrelated unit test: {noformat} Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart) Time elapsed: 0.411 sec <<< FAILURE! java.lang.AssertionError: expected null, but was: at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotNull(Assert.java:664) at org.junit.Assert.assertNull(Assert.java:646) at org.junit.Assert.assertNull(Assert.java:656) at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5361) Obtaining logs for completed container says 'file belongs to a running container ' at the end
Sumana Sathish created YARN-5361: Summary: Obtaining logs for completed container says 'file belongs to a running container ' at the end Key: YARN-5361 URL: https://issues.apache.org/jira/browse/YARN-5361 Project: Hadoop YARN Issue Type: Improvement Reporter: Sumana Sathish Assignee: Xuan Gong Priority: Critical Obtaining logs via yarn CLI for completed container but running application says "This log file belongs to a running container (container_e32_1468319707096_0001_01_04) and so may not be complete" which is not correct. {code} LogType:stdout Log Upload Time:Tue Jul 12 10:38:14 + 2016 Log Contents: End of LogType:stdout. This log file belongs to a running container (container_e32_1468319707096_0001_01_04) and so may not be complete. {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/ [Jul 11, 2016 9:55:33 PM] (jitendra) HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS [Jul 12, 2016 12:21:56 AM] (aajisaka) HADOOP-13297. Add missing dependency in setting [Jul 12, 2016 1:06:28 AM] (wang) HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by [Jul 12, 2016 5:36:20 AM] (jianhe) YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.cli.TestLogsCLI hadoop.yarn.client.api.impl.TestYarnClient cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-javadoc-javadoc-root.txt [2.3M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org