Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk

2017-08-17 Thread Wangda Tan 
+hdfs/common/mr

On Thu, Aug 17, 2017 at 1:28 PM, Wangda Tan  wrote:

> Hi all,
>
> I want to hear your thoughts of merging YARN resource profile branch into
> trunk in the next few weeks. The goal is to get it in for Hadoop 3.0 beta1.
>
> *Regarding to testing:*
> We did extensive tests for the feature in the last several months.
> Comparing to latest trunk.
> - For SLS benchmark: We didn't see observable performance gap from
> simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
> containers allocated per second.
> - For microbenchmark: We use performance test cases added by YARN-6775, it
> shows around 5% performance regression comparing to trunk.
>
> *Regarding to API stability: *
> Most new added @Public APIs are @Unstable (We're going to convert some new
> added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we want
> to get this included by beta1 so we get some feedbacks before declaring
> stable API.
>
> There're few pending cleanups under YARN-3926 umbrella JIRA. Besides these
> cleanups, this feature works from end-to-end, we will do another iteration
> of end-to-end tests after cleanup patches got committed.
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Varun
> Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> Karthik Kambatla, Jason Lowe, Arun Suresh.
>
> Thanks,
> Wangda Tan
>

[jira] [Created] (HDFS-12317) HDFS metrics render error in the page of git respository

2017-08-17 Thread Yiqun Lin (JIRA) 
Yiqun Lin created HDFS-12317:


 Summary: HDFS metrics render error in the page of git respository
 Key: HDFS-12317
 URL: https://issues.apache.org/jira/browse/HDFS-12317
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: documentation, metrics
Affects Versions: 2.9.0, 3.0.0-beta1
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor


Some HDFS metrics render error in the page of git respository. 
The page link: 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12039) Ozone: Implement update volume owner in ozone shell

2017-08-17 Thread Weiwei Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HDFS-12039.

   Resolution: Fixed
Fix Version/s: HDFS-7240

> Ozone: Implement update volume owner in ozone shell
> ---
>
> Key: HDFS-12039
> URL: https://issues.apache.org/jira/browse/HDFS-12039
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Weiwei Yang
>Assignee: Lokesh Jain
> Fix For: HDFS-7240
>
>
> Ozone shell command {{updateVolume}} should support to update the owner of a 
> volume, using following syntax
> {code}
> hdfs oz -updateVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -owner 
> xyz -root
> {code}
> this could work from rest api, following command could change the volume 
> owner to {{www}}
> {code}
> curl -X PUT -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" 
> -H "x-ozone-user:www" -H "Authorization:OZONE root" 
> http://ozone1.fyre.ibm.com:9864/volume-wwei-0
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12201) INode#getSnapshotINode() should get INodeAttributes from INodeAttributesProvider for the current INode

2017-08-17 Thread Manoj Govindassamy (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy resolved HDFS-12201.
---
   Resolution: Not A Bug
Fix Version/s: 3.0.0-beta1

[~daryn],
  Thanks for the comments. Agreed, edit log and the fsimage could end up 
persisted with external attributes. FSDirectory seems to be right level to 
aggregate or transform the results with external attributes so as to keep the 
abstraction clean. Closing the bug.

> INode#getSnapshotINode() should get INodeAttributes from 
> INodeAttributesProvider for the current INode
> --
>
> Key: HDFS-12201
> URL: https://issues.apache.org/jira/browse/HDFS-12201
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.8.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12201.test.01.patch
>
>
> Problem: When an external INodeAttributesProvider is enabled, SnapshotDiff is 
> not detecting changes in files when the external ACL/XAttr attributes change. 
> {{FileWithSnapshotFeature#changedBetweenSnapshots()}} when trying to detect 
> changes in snapshots for the given file, does meta data comparison which 
> takes in the attributes retrieved from {{INode#getSnapshotINode()}}
> {{INodeFile}}
> {noformat}
>   @Override
>   public INodeFileAttributes getSnapshotINode(final int snapshotId) {
> FileWithSnapshotFeature sf = this.getFileWithSnapshotFeature();
> if (sf != null) {
>   return sf.getDiffs().getSnapshotINode(snapshotId, this);
> } else {
>   return this;
> }
>   }
> {noformat}
> {{AbstractINodeDiffList#getSnapshotINode}}
> {noformat}
>   public A getSnapshotINode(final int snapshotId, final A currentINode) {
> final D diff = getDiffById(snapshotId);
> final A inode = diff == null? null: diff.getSnapshotINode();
> return inode == null? currentINode: inode;
>   }
> {noformat}
> But, INodeFile, INodeDirectory #getSnapshotINode() returns the current 
> INode's local INodeAttributes if there is anything available for the given 
> snapshot id. When there is an INodeAttributesProvider configured, attributes 
> provided by the external provider could be different from the local. But, 
> getSnapshotINode() always returns the local attributes without retrieving 
> them from attributes provider. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-08-17 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/

[Aug 16, 2017 6:04:36 PM] (jlowe) YARN-7020. 
TestAMRMProxy#testAMRMProxyTokenRenewal is flakey.
[Aug 16, 2017 6:43:24 PM] (subu) YARN-6900. ZooKeeper based implementation of 
the FederationStateStore.
[Aug 16, 2017 9:34:06 PM] (jlowe) MAPREDUCE-6940. Copy-paste error in the
[Aug 16, 2017 11:14:04 PM] (rkanter) MAPREDUCE-6936. Remove unnecessary 
dependency of
[Aug 17, 2017 4:20:27 AM] (aajisaka) HDFS-12269. Better to return a Map rather 
than HashMap in
[Aug 17, 2017 5:17:06 AM] (wang) HDFS-11082. Provide replicated EC policy to 
replicate files. Contributed




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.ipc.TestRPC 
   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 
   hadoop.hdfs.TestSafeModeWithStripedFile 
   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM 
   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics 
   hadoop.hdfs.security.TestDelegationTokenForProxyUser 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.server.router.webapp.TestRouterWebServicesREST 
   hadoop.hdfs.TestNNBench 
   hadoop.mapred.TestMRTimelineEventHandling 
   hadoop.tools.TestDistCpSystem 
   hadoop.fs.azure.TestClientThrottlingAnalyzer 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 
   hadoop.yarn.sls.appmaster.TestAMSimulator 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/diff-compile-javac-root.txt
  [300K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/495/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

[jira] [Created] (HDFS-12316) Verify HDFS snapshot deletion doesn't crash the ongoing file writes

2017-08-17 Thread Manoj Govindassamy (JIRA) 
Manoj Govindassamy created HDFS-12316:
-

 Summary: Verify HDFS snapshot deletion doesn't crash the ongoing 
file writes
 Key: HDFS-12316
 URL: https://issues.apache.org/jira/browse/HDFS-12316
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy


Recently we encountered a case where deletion of HDFS snapshots crashed the 
client that is currently writing to a file under the same snap root. This open 
file was previously captured in the snapshot using the immutable open file in 
snapshot feature "dfs.namenode.snapshot.capture.openfiles".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11813) TestDFSStripedOutputStreamWithFailure070 failed randomly

2017-08-17 Thread Andrew Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-11813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HDFS-11813.

Resolution: Duplicate

This looks the same as HDFS-11882 where we've got a patch that seems close, 
let's dupe to that one.

> TestDFSStripedOutputStreamWithFailure070 failed randomly
> 
>
> Key: HDFS-11813
> URL: https://issues.apache.org/jira/browse/HDFS-11813
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: SammiChen
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
>
> TestDFSStripedOutputStreamWithFailure070 failed randomly. Here is the stack 
> trace,
> java.lang.AssertionError: failed, dn=0, 
> length=1638400java.lang.IllegalStateException
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
>   at 
> org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780)
>   at 
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664)
>   at 
> org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842)
>   at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>   at 
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
>   at 
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472)
>   at 
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:360)
>   at 
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.run(TestDFSStripedOutputStreamWithFailure.java:574)
>   at 
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.test7(TestDFSStripedOutputStreamWithFailure.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:365)
>   at 
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.run(TestDFSStripedOutputStreamWithFailure.java:574)
>   at 
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.test7(TestDFSStripedOutputStreamWithFailure.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org