[jira] [Created] (YARN-6887) WebServices methods should be moved to an interface
Giovanni Matteo Fumarola created YARN-6887: -- Summary: WebServices methods should be moved to an interface Key: YARN-6887 URL: https://issues.apache.org/jira/browse/YARN-6887 Project: Hadoop YARN Issue Type: Improvement Reporter: Giovanni Matteo Fumarola {{WebServices}} implements 3 methods that are being called from {{RMWebServices}}. These 3 methods should be defined in an interface. It will make the code clean. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6886) AllocationFileLoaderService.loadQueue() should validate that setting do not conflict with parent
Daniel Templeton created YARN-6886: -- Summary: AllocationFileLoaderService.loadQueue() should validate that setting do not conflict with parent Key: YARN-6886 URL: https://issues.apache.org/jira/browse/YARN-6886 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Minor Some settings, like policy, are limited by the queue's parent queue's configuration. We should check those settings when we load the file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
Daniel Templeton created YARN-6885: -- Summary: AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/... Key: YARN-6885 URL: https://issues.apache.org/jira/browse/YARN-6885 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Minor {code} if ("minResources".equals(field.getTagName())) { String text = ((Text)field.getFirstChild()).getData().trim(); Resource val = FairSchedulerConfiguration.parseResourceConfigValue(text); minQueueResources.put(queueName, val); } else if ("maxResources".equals(field.getTagName())) { ...{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6884) AllocationFileLoaderService.loadQueue() has an if without braces
Daniel Templeton created YARN-6884: -- Summary: AllocationFileLoaderService.loadQueue() has an if without braces Key: YARN-6884 URL: https://issues.apache.org/jira/browse/YARN-6884 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Trivial {code} if (!(fieldNode instanceof Element)) continue;{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6883) AllocationFileLoaderService.reloadAllocations() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
Daniel Templeton created YARN-6883: -- Summary: AllocationFileLoaderService.reloadAllocations() should use a switch statement in the main tag parsing loop instead of the if/else-if/... Key: YARN-6883 URL: https://issues.apache.org/jira/browse/YARN-6883 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Minor {code}if ("queue".equals(element.getTagName()) || "pool".equals(element.getTagName())) { queueElements.add(element); } else if ("user".equals(element.getTagName())) { ...{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6882) AllocationFileLoaderService.reloadAllocations() should use the diamond operator
Daniel Templeton created YARN-6882: -- Summary: AllocationFileLoaderService.reloadAllocations() should use the diamond operator Key: YARN-6882 URL: https://issues.apache.org/jira/browse/YARN-6882 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Trivial Here:{code}for (FSQueueType queueType : FSQueueType.values()) { configuredQueues.put(queueType, new HashSet()); }{code} and here:{code}List queueElements = new ArrayList();{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6881) LOG is unused in AllocationConfiguration
Daniel Templeton created YARN-6881: -- Summary: LOG is unused in AllocationConfiguration Key: YARN-6881 URL: https://issues.apache.org/jira/browse/YARN-6881 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton The variable can be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6880) FSQueue.reservedResource can be final
Daniel Templeton created YARN-6880: -- Summary: FSQueue.reservedResource can be final Key: YARN-6880 URL: https://issues.apache.org/jira/browse/YARN-6880 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6879) TestLeafQueue.testDRFUserLimits() has commented out code
Daniel Templeton created YARN-6879: -- Summary: TestLeafQueue.testDRFUserLimits() has commented out code Key: YARN-6879 URL: https://issues.apache.org/jira/browse/YARN-6879 Project: Hadoop YARN Issue Type: Improvement Components: capacity scheduler, test Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Trivial The commented-out code should be deleted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6878) TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the args to assertEqual() in the wrong order
Daniel Templeton created YARN-6878: -- Summary: TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the args to assertEqual() in the wrong order Key: YARN-6878 URL: https://issues.apache.org/jira/browse/YARN-6878 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler, test Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton Priority: Trivial The expected value should come before the actual value. It would be nice to add some assert messages as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6877) Create an abstract log reader for extendability
Xuan Gong created YARN-6877: --- Summary: Create an abstract log reader for extendability Key: YARN-6877 URL: https://issues.apache.org/jira/browse/YARN-6877 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently, TFile log reader is used to read aggregated log in YARN. We need to add an abstract layer, and pick up the correct log reader based on the configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6876) Create an abstract log writer for extendability
Xuan Gong created YARN-6876: --- Summary: Create an abstract log writer for extendability Key: YARN-6876 URL: https://issues.apache.org/jira/browse/YARN-6876 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently, TFile log writer is used to aggregated log in YARN. We need to add an abstract layer, and pick up the correct log writer based on the configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6875) New aggregated log file format for YARN log aggregation.
Xuan Gong created YARN-6875: --- Summary: New aggregated log file format for YARN log aggregation. Key: YARN-6875 URL: https://issues.apache.org/jira/browse/YARN-6875 Project: Hadoop YARN Issue Type: New Feature Reporter: Xuan Gong Assignee: Xuan Gong T-file is the underlying log format for the aggregated logs in YARN. We have seen several performance issues, especially for very large log files. We will introduce a new log format which have better performance for large log files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [DISCUSS] Looking to a 2.9.0 release
Thanks Vrushali for being entirely open as to the current status of ATSv2. I appreciate that we want to ensure things are tested at scale, and as you said we are working on that right now on our clusters. We have tested the feature to demonstrate it works at what we consider moderate scale. I think the criteria for including this feature in the 2.9 release should be if it can be safely turned off and not cause impact to anybody not using the new feature. The confidence for this is high for timeline service v2. Therefore, I think timeline service v2 should definitely be part of 2.9. That is the big draw for us to work on stabilizing a 2.9 release rather than just going to 2.8 and back-porting things ourselves. Thanks, Joep On Tue, Jul 25, 2017 at 11:39 PM, Vrushali Channapattan < vrushalic2...@gmail.com> wrote: > Thanks Subru for initiating this discussion. > > Wanted to share some thoughts in the context of Timeline Service v2. The > current status of this module is that we are ramping up for a second merge > to trunk. We still have a few merge blocker jiras outstanding, which we > think we will finish soon. > > While we have done some testing, we are yet to test at scale. Given all > this, we were thinking of initially targeting a beta release vehicle rather > than a stable release. > > As such, timeline service v2 has branch-2 branch called as > YARN-5355-branch-2 in case anyone wants to try it out. Timeline service v2 > can be turned off and should not affect the cluster. > > thanks > Vrushali > > > > > > On Mon, Jul 24, 2017 at 1:26 PM, Subru Krishnanwrote: > > > Folks, > > > > With the release for 2.8, we would like to look ahead to 2.9 release as > > there are many features/improvements in branch-2 (about 1062 commits), > that > > are in need of a release vechile. > > > > Here's our first cut of the proposal from the YARN side: > > > >1. Scheduler improvements (decoupling allocation from node heartbeat, > >allocation ID, concurrency fixes, LightResource etc). > >2. Timeline Service v2 > >3. Opportunistic containers > >4. Federation > > > > We would like to hear a formal list from HDFS & Hadoop (& MapReduce if > any) > > and will update the Roadmap wiki accordingly. > > > > Considering our familiarity with the above mentioned YARN features, we > > would like to volunteer as the co-RMs for 2.9.0. > > > > We want to keep the timeline at 8-12 weeks to keep the release pragmatic. > > > > Feedback? > > > > -Subru/Arun > > >
[jira] [Created] (YARN-6874) TestHBaseStorageFlowRun.testWriteFlowRunMinMax fails intermittently
Varun Saxena created YARN-6874: -- Summary: TestHBaseStorageFlowRun.testWriteFlowRunMinMax fails intermittently Key: YARN-6874 URL: https://issues.apache.org/jira/browse/YARN-6874 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Saxena {noformat} testWriteFlowRunMinMax(org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun) Time elapsed: 0.088 sec <<< FAILURE! java.lang.AssertionError: expected:<142502690> but was:<1425026901000> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun.testWriteFlowRunMinMax(TestHBaseStorageFlowRun.java:237) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6873) Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice
Yeliang Cang created YARN-6873: -- Summary: Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice Key: YARN-6873 URL: https://issues.apache.org/jira/browse/YARN-6873 Project: Hadoop YARN Issue Type: Sub-task Reporter: Yeliang Cang Assignee: Yeliang Cang -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/475/ [Jul 24, 2017 7:24:23 AM] (aajisaka) HDFS-12176. dfsadmin shows DFS Used%: NaN% if the cluster has zero [Jul 24, 2017 3:19:20 PM] (jlowe) YARN-6768. Improve performance of yarn api record toString and [Jul 24, 2017 7:03:02 PM] (templedf) YARN-6844. AMRMClientImpl.checkNodeLabelExpression() has wrong error [Jul 24, 2017 7:13:50 PM] (templedf) YARN-6779. [Jul 24, 2017 8:44:00 PM] (templedf) YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its [Jul 24, 2017 11:01:45 PM] (raviprak) HADOOP-14597. Native compilation broken with OpenSSL-1.1.0. Contributed [Jul 25, 2017 2:06:15 AM] (raviprak) HDFS-12193. Fix style issues in HttpFS tests. Contributed by Zoran [Jul 25, 2017 6:01:01 AM] (jitendra) HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by [Jul 25, 2017 6:12:08 AM] (aajisaka) YARN-6150. TestContainerManagerSecurity tests for Yarn Server are [Jul 25, 2017 6:24:56 AM] (aajisaka) HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. [Jul 25, 2017 3:28:57 PM] (kihwal) HDFS-12143. Improve performance of getting and removing inode features. [Jul 25, 2017 3:51:53 PM] (brahma) HADOOP-14455. ViewFileSystem#rename should support be supported within [Jul 25, 2017 4:03:49 PM] (kihwal) HDFS-12171. Reduce IIP object allocations for inode lookup. Contributed [Jul 25, 2017 4:57:14 PM] (jianhe) YARN-6804. Allow custom hostname for docker containers in native [Jul 25, 2017 8:00:31 PM] (templedf) YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu [Jul 25, 2017 11:26:48 PM] (jitendra) HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-hdfs-project/hadoop-hdfs-client Possible exposure of partially initialized object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:[line 2888] org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) makes inefficient use of keySet iterator instead of entrySet iterator At SlowDiskReports.java:keySet iterator instead of entrySet iterator At SlowDiskReports.java:[line 105] FindBugs : module:hadoop-hdfs-project/hadoop-hdfs Possible null pointer dereference in org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:[line 302] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String) unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId At HdfsServerConstants.java:[line 193] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int) unconditionally sets the field force At HdfsServerConstants.java:force At HdfsServerConstants.java:[line 217] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean) unconditionally sets the field isForceFormat At HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean) unconditionally sets the field isInteractiveFormat At HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 237] Possible null pointer dereference in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:[line 1339] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:[line 258] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, BasicFileAttributes) due to
[jira] [Created] (YARN-6872) Ensure apps could run given NodeLabels are disabled post RM switchover/restart
Sunil G created YARN-6872: - Summary: Ensure apps could run given NodeLabels are disabled post RM switchover/restart Key: YARN-6872 URL: https://issues.apache.org/jira/browse/YARN-6872 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Post YARN-6031, few apps could be failed during recovery provided they had some label requirements for AM and labels were disable post RM restart/switchover. As discussed in YARN-6031, its better to run such apps as it may be long running apps as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [DISCUSS] Looking to a 2.9.0 release
Thanks Subru for initiating this discussion. Wanted to share some thoughts in the context of Timeline Service v2. The current status of this module is that we are ramping up for a second merge to trunk. We still have a few merge blocker jiras outstanding, which we think we will finish soon. While we have done some testing, we are yet to test at scale. Given all this, we were thinking of initially targeting a beta release vehicle rather than a stable release. As such, timeline service v2 has branch-2 branch called as YARN-5355-branch-2 in case anyone wants to try it out. Timeline service v2 can be turned off and should not affect the cluster. thanks Vrushali On Mon, Jul 24, 2017 at 1:26 PM, Subru Krishnanwrote: > Folks, > > With the release for 2.8, we would like to look ahead to 2.9 release as > there are many features/improvements in branch-2 (about 1062 commits), that > are in need of a release vechile. > > Here's our first cut of the proposal from the YARN side: > >1. Scheduler improvements (decoupling allocation from node heartbeat, >allocation ID, concurrency fixes, LightResource etc). >2. Timeline Service v2 >3. Opportunistic containers >4. Federation > > We would like to hear a formal list from HDFS & Hadoop (& MapReduce if any) > and will update the Roadmap wiki accordingly. > > Considering our familiarity with the above mentioned YARN features, we > would like to volunteer as the co-RMs for 2.9.0. > > We want to keep the timeline at 8-12 weeks to keep the release pragmatic. > > Feedback? > > -Subru/Arun >