[jira] [Created] (YARN-6068) Log aggregation get failed when NM restart even with recovery
Junping Du created YARN-6068:
Summary: Log aggregation get failed when NM restart even with
recovery
Key: YARN-6068
URL: https://issues.apache.org/jira/browse/YARN-6068
Project: Hadoop YARN
Issue Type: Bug
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical
The exception log is as following:
{noformat}
2017-01-05 19:16:36,352 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:abortLogAggregation(527)) - Aborting log aggregation
for application_1483640789847_0001
2017-01-05 19:16:36,352 WARN logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:run(399)) - Aggregation did not complete for
application application_1483640789847_0001
2017-01-05 19:16:36,353 WARN application.ApplicationImpl
(ApplicationImpl.java:handle(461)) - Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
APPLICATION_LOG_HANDLING_FAILED at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:459)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:64)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1084)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1076)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:745)
2017-01-05 19:16:36,355 INFO application.ApplicationImpl
(ApplicationImpl.java:handle(464)) - Application application_1483640789847_0001
transitioned from RUNNING to null
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6067) API Service HA
Gour Saha created YARN-6067: --- Summary: API Service HA Key: YARN-6067 URL: https://issues.apache.org/jira/browse/YARN-6067 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha We need to start thinking about HA for API Service. How do we achieve it? Should API Service become part of the RM process to get a lot of things for free? Should there be some other strategy. We need to start the discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-126) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/YARN-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rémy SAISSY resolved YARN-126. -- Resolution: Won't Fix Target Version/s: (was: ) No longer relevant. > yarn rmadmin help message contains reference to hadoop cli and JT > - > > Key: YARN-126 > URL: https://issues.apache.org/jira/browse/YARN-126 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.0.3-alpha >Reporter: Thomas Graves >Assignee: Rémy SAISSY > Labels: usability > Attachments: YARN-126.002.patch, YARN-126.patch > > > has option to specify a job tracker and the last line for general command > line syntax had "bin/hadoop command [genericOptions] [commandOptions]" > ran "yarn rmadmin" to get usage: > RMAdmin > Usage: java RMAdmin >[-refreshQueues] >[-refreshNodes] >[-refreshUserToGroupsMappings] >[-refreshSuperUserGroupsConfiguration] >[-refreshAdminAcls] >[-refreshServiceAcl] >[-help [cmd]] > Generic options supported are > -conf specify an application configuration file > -D
[jira] [Created] (YARN-6066) Opportunistic containers: Minor fixes: API annotations and config parameter changes
Arun Suresh created YARN-6066: - Summary: Opportunistic containers: Minor fixes: API annotations and config parameter changes Key: YARN-6066 URL: https://issues.apache.org/jira/browse/YARN-6066 Project: Hadoop YARN Issue Type: Bug Reporter: Arun Suresh Priority: Minor Creating this to capture changes suggested by [~leftnoteasy] and [~kasha] in YARN-6041 in its own JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/ [Jan 5, 2017 6:31:05 PM] (wangda) YARN-5959. RM changes to support change of container ExecutionType. [Jan 5, 2017 9:40:43 PM] (wang) HADOOP-13896. Invoke assembly plugin after building jars. [Jan 5, 2017 11:04:47 PM] (cnauroth) HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme [Jan 6, 2017 12:04:58 AM] (xiao) HADOOP-13780. LICENSE/NOTICE are out of date for source artifacts. [Jan 6, 2017 1:21:57 AM] (xiao) HADOOP-13597. Switch KMS from Tomcat to Jetty. Contributed by John [Jan 6, 2017 1:56:08 AM] (templedf) MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu [Jan 6, 2017 3:34:32 AM] (arp) HDFS-11282. Document the missing metrics of DataNode Volume IO -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.mover.TestStorageMover hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-compile-root.txt [124K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-compile-root.txt [124K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-compile-root.txt [124K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [200K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [68K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/209/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K]
[jira] [Created] (YARN-6065) Update javadocs of new added APIs / classes of scheduler/AppSchedulingInfo
Wangda Tan created YARN-6065: Summary: Update javadocs of new added APIs / classes of scheduler/AppSchedulingInfo Key: YARN-6065 URL: https://issues.apache.org/jira/browse/YARN-6065 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6064) Support fromId for flowRuns and flow/flowRun apps.
Rohith Sharma K S created YARN-6064: --- Summary: Support fromId for flowRuns and flow/flowRun apps. Key: YARN-6064 URL: https://issues.apache.org/jira/browse/YARN-6064 Project: Hadoop YARN Issue Type: Sub-task Components: timelinereader Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S Splitting out JIRA YARN-6027 for pagination support for flowRuns, flow apps and flow run apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6063) Inline java doc with Hadoop formats.
Rohith Sharma K S created YARN-6063: --- Summary: Inline java doc with Hadoop formats. Key: YARN-6063 URL: https://issues.apache.org/jira/browse/YARN-6063 Project: Hadoop YARN Issue Type: Sub-task Components: timelinereader Reporter: Rohith Sharma K S Priority: Trivial I see that most of the classes in TimelineReader java doc does not meet Hadoop formatters. This causes every time patch preparation need an extra attention on top of real fix patches. It is better to be source code is inline with Hadoop formatters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/ [Jan 5, 2017 6:31:05 PM] (wangda) YARN-5959. RM changes to support change of container ExecutionType. [Jan 5, 2017 9:40:43 PM] (wang) HADOOP-13896. Invoke assembly plugin after building jars. [Jan 5, 2017 11:04:47 PM] (cnauroth) HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme [Jan 6, 2017 12:04:58 AM] (xiao) HADOOP-13780. LICENSE/NOTICE are out of date for source artifacts. [Jan 6, 2017 1:21:57 AM] (xiao) HADOOP-13597. Switch KMS from Tomcat to Jetty. Contributed by John [Jan 6, 2017 1:56:08 AM] (templedf) MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu [Jan 6, 2017 3:34:32 AM] (arp) HDFS-11282. Document the missing metrics of DataNode Volume IO -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.net.TestDNS hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.mapreduce.v2.TestUberAM Timed out junit tests : org.apache.hadoop.mapreduce.v2.TestMRJobs cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [124K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [212K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [148K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/278/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6062) nodemanager memory leak
gehaijiang created YARN-6062:
Summary: nodemanager memory leak
Key: YARN-6062
URL: https://issues.apache.org/jira/browse/YARN-6062
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.7.1
Reporter: gehaijiang
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
8986 data 20 0 21.3g 19g 7376 S 5.5 20.7 2458:09 java
38432 data 20 0 9.8g 7.9g 6300 S 95.5 8.4 35273:23 java
6653 data 20 0 4558m 3.4g 10m S 9.2 3.6 6640:37 java
$ jps
6653 NodeManager
Nodemanager memory has been up，Reach 10G。
nodemanager yarn-env.sh configure (2G)
YARN_NODEMANAGER_OPTS=" -Xms2048m -Xmn768m
-Xloggc:${YARN_LOG_DIR}/nodemanager.gc.log -XX:+PrintGCDateStamps
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
