[jira] [Resolved] (HADOOP-14189) add distcp-site.xml for distcp on branch-2

2017-03-16 Thread Fei Hui (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui resolved HADOOP-14189.
--
Resolution: Duplicate

> add distcp-site.xml for distcp on branch-2
> --
>
> Key: HADOOP-14189
> URL: https://issues.apache.org/jira/browse/HADOOP-14189
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools/distcp
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HADOOP-14189-branch-2.001.patch
>
>
> On hadoop 2.x , we could not config hadoop parameters for distcp. It only 
> uses distcp-default.xml.
> We should add distcp-site.xml to overrides hadoop paramers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14194) Alyun OSS should not use empty endpoint as default

2017-03-16 Thread Mingliang Liu (JIRA) 
Mingliang Liu created HADOOP-14194:
--

 Summary: Alyun OSS should not use empty endpoint as default
 Key: HADOOP-14194
 URL: https://issues.apache.org/jira/browse/HADOOP-14194
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/oss
Reporter: Mingliang Liu
Assignee: Xiaobing Zhou


In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and 
using empty string as a default value.
{code}
String endPoint = conf.getTrimmed(ENDPOINT_KEY, "");
{code}

The plain value without validation is passed to OSSClient. If the endPoint is 
not provided (empty string) or the endPoint is not valid, users will get 
exception from Aliyun OSS sdk with raw exception message like:
{code}
java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected 
authority at index 8: https://

at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359)
at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313)
at com.aliyun.oss.OSSClient.(OSSClient.java:297)
at 
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134)
at 
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272)
at 
org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63)
at 
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47)
at junit.framework.TestCase.runBare(TestCase.java:139)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.net.URISyntaxException: Expected authority at index 8: https://
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.failExpecting(URI.java:2854)
at java.net.URI$Parser.parseHierarchical(URI.java:3102)
at java.net.URI$Parser.parse(URI.java:3053)
at java.net.URI.(URI.java:588)
at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357)
{code}

Let's check endPoint is not null or empty, catch the IllegalArgumentException 
and log it, wrapping the exception with clearer message stating the 
misconfiguration in endpoint or credentials.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14193) Cache the resolved hosts to prevent calls to InetAddress.getByName and normalizeHost

2017-03-16 Thread Jose Miguel Arreola (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Miguel Arreola resolved HADOOP-14193.
--
Resolution: Invalid

> Cache the resolved hosts to prevent calls to InetAddress.getByName and 
> normalizeHost
> 
>
> Key: HADOOP-14193
> URL: https://issues.apache.org/jira/browse/HADOOP-14193
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: common
>Reporter: Jose Miguel Arreola
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When running performance tests, we noticed that a lot of time is taken in 
> resolving the host address.
> In our specific scenario, we saw the function 
> org.apache.hadoop.security.SecurityUtil.getInetAddressByName taking a lot of 
> time to resolve the hosts, and the same function is called a lot of times.
> I saw that org.apache.hadoop.yarn.server.resourcemanager.NodesListManager has 
> a cached resolver already because of the same reason.
> So, the proposal is, to make this cache generic and use it to save some time 
> in this functions that we already know, but have it available so the cache 
> can be used anywhere else.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14193) Cache the resolved hosts prevent calls to InetAddress.getByName and normalizeHost

2017-03-16 Thread Jose Miguel Arreola (JIRA) 
Jose Miguel Arreola created HADOOP-14193:


 Summary: Cache the resolved hosts prevent calls to 
InetAddress.getByName and normalizeHost
 Key: HADOOP-14193
 URL: https://issues.apache.org/jira/browse/HADOOP-14193
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Reporter: Jose Miguel Arreola


When running performance tests, we noticed that a lot of time is taken in 
resolving the host address.
In our specific scenario, we saw the function 
org.apache.hadoop.security.SecurityUtil.getInetAddressByName taking a lot of 
time to resolve the hosts, and the same function is called a lot of times.
I saw that org.apache.hadoop.yarn.server.resourcemanager.NodesListManager has a 
cached resolver already because of the same reason.
So, the proposal is, to make this cache generic and use it to save some time in 
this functions that we already know, but have it available so the cache can be 
used anywhere else.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14192) Aliyun OSS FileSystem contract test should implement getTestBaseDir()

2017-03-16 Thread Mingliang Liu (JIRA) 
Mingliang Liu created HADOOP-14192:
--

 Summary: Aliyun OSS FileSystem contract test should implement 
getTestBaseDir()
 Key: HADOOP-14192
 URL: https://issues.apache.org/jira/browse/HADOOP-14192
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/oss
Reporter: Mingliang Liu
Assignee: Mingliang Liu


[HADOOP-14170] is the recent effort of improving the file system contract tests 
{{FileSystemContractBaseTest}}, which make {{path()}} method final and add a 
new method {{getTestBaseDir()}} for subclasses to implement. Aliyun OSS should 
override that as it uses unique directory (naming with fork id) for supporting 
parallel tests. Plus, the current {{testWorkingDirectory}} needs not override 
per changes in {{FileSystemContractBaseTest}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-7880) The Single Node and Cluster Setup docs don't cover HDFS

2017-03-16 Thread Arpit Agarwal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HADOOP-7880.
---
Resolution: Not A Problem

This is covered by our docs now, resolving.
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html

> The Single Node and Cluster Setup docs don't cover HDFS
> ---
>
> Key: HADOOP-7880
> URL: https://issues.apache.org/jira/browse/HADOOP-7880
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>
> The main docs page (http://hadoop.apache.org/common/docs/r0.23.0) only has 
> HDFS docs for federation. Only MR2 is covered in the single node and cluster 
> setup documentation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2017-03-16 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/

[Mar 15, 2017 6:48:17 PM] (junping_du) YARN-6294. ATS client should better 
handle Socket closed case.
[Mar 15, 2017 7:28:22 PM] (arp) HDFS-11419. DFSTopologyNodeImpl#chooseRandom 
optimizations. Contributed
[Mar 16, 2017 1:01:45 AM] (arp) HDFS-11511. Support Timeout when checking 
single disk. Contributed by
[Mar 16, 2017 4:59:55 AM] (jianhe) YARN-6332. Make RegistrySecurity use short 
user names for ZK ACLs.




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.tools.TestDFSAdminWithHA 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-compile-root.txt
  [132K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-compile-root.txt
  [132K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-compile-root.txt
  [132K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [240K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/259/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
  [16K]

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-03-16 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/

[Mar 15, 2017 9:18:05 AM] (sunilg) YARN-6328. Fix a spelling mistake in 
CapacityScheduler. Contributed by
[Mar 15, 2017 10:05:03 AM] (yqlin) HDFS-11420. Edit file should not be 
processed by the same type processor
[Mar 15, 2017 10:24:09 AM] (rohithsharmaks) YARN-6336. Jenkins report YARN new 
UI build failure. Contributed by
[Mar 15, 2017 6:48:17 PM] (junping_du) YARN-6294. ATS client should better 
handle Socket closed case.
[Mar 15, 2017 7:28:22 PM] (arp) HDFS-11419. DFSTopologyNodeImpl#chooseRandom 
optimizations. Contributed
[Mar 16, 2017 1:01:45 AM] (arp) HDFS-11511. Support Timeout when checking 
single disk. Contributed by
[Mar 16, 2017 4:59:55 AM] (jianhe) YARN-6332. Make RegistrySecurity use short 
user names for ZK ACLs.




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.mapreduce.v2.app.job.impl.TestJobImpl 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/diff-compile-javac-root.txt
  [180K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [272K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
  [20K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/347/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-16 Thread Junping Du 
Thanks Steve. That's Awesome! I will kick off a new RC soon.
Shall we reopen HDFS-6200 given issues here? Making it in release note of 2.8.0 
could confuse people as it doesn't work in HA deployment.

Thanks,

Junping

From: Steve Loughran
Sent: Thursday, March 16, 2017 7:27 AM
To: Junping Du
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

> On 16 Mar 2017, at 00:25, Junping Du  wrote:
>
> bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client to 
> pull in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. 
> So I think this affects users of the hadoop-client artifact, which has 
> existed for a long time.
>
> I could miss that. Thanks for reminding! From my quick check: 
> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
> sounds like 669 artifacts from other projects were depending on it.
>
>
> I think we should withdraw the current RC bits. Please stop the verification 
> & vote.
>
> I will kick off another RC immediately when HDFS-11431 get fixed.

is done. hadoop-hdfs without any server-side dependencies is now a 
hadoop-client dependency.

Release notes:

The hadoop-client POM now includes a leaner hdfs-client, stripping out all the 
transitive dependencies on JARs only needed for the Hadoop HDFS daemon itself. 
The specific jars now excluded are: leveldbjni-all, jetty-util, commons-daemon, 
xercesImpl, netty and servlet-api.

This should make downstream projects dependent JARs smaller, and avoid version 
conflict problems with the specific JARs now excluded.

Applications may encounter build problems if they did depend on these JARs, and 
which didn't explicitly include them. There are two fixes for this

* explicitly include the JARs, stating which version of them you want.
* add a dependency on hadoop-hdfs. For Hadoop 2.8+, this will add the missing 
dependencies. For builds against older versions of Hadoop, this will be 
harmless, as hadoop-hdfs and all its dependencies are already pulled in by the 
hadoop-client POM.




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-16 Thread Steve Loughran 

> On 16 Mar 2017, at 00:25, Junping Du  wrote:
> 
> bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client to 
> pull in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. 
> So I think this affects users of the hadoop-client artifact, which has 
> existed for a long time.
> 
> I could miss that. Thanks for reminding! From my quick check: 
> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
> sounds like 669 artifacts from other projects were depending on it.
> 
> 
> I think we should withdraw the current RC bits. Please stop the verification 
> & vote.
> 
> I will kick off another RC immediately when HDFS-11431 get fixed.

is done. hadoop-hdfs without any server-side dependencies is now a 
hadoop-client dependency.

Release notes:

The hadoop-client POM now includes a leaner hdfs-client, stripping out all the 
transitive dependencies on JARs only needed for the Hadoop HDFS daemon itself. 
The specific jars now excluded are: leveldbjni-all, jetty-util, commons-daemon, 
xercesImpl, netty and servlet-api.

This should make downstream projects dependent JARs smaller, and avoid version 
conflict problems with the specific JARs now excluded.

Applications may encounter build problems if they did depend on these JARs, and 
which didn't explicitly include them. There are two fixes for this

* explicitly include the JARs, stating which version of them you want.
* add a dependency on hadoop-hdfs. For Hadoop 2.8+, this will add the missing 
dependencies. For builds against older versions of Hadoop, this will be 
harmless, as hadoop-hdfs and all its dependencies are already pulled in by the 
hadoop-client POM.




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14191) Duplicate hadoop-minikdc dependency in hadoop-common module

2017-03-16 Thread Akira Ajisaka (JIRA) 
Akira Ajisaka created HADOOP-14191:
--

 Summary: Duplicate hadoop-minikdc dependency in hadoop-common 
module
 Key: HADOOP-14191
 URL: https://issues.apache.org/jira/browse/HADOOP-14191
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Akira Ajisaka
Priority: Minor


The following warning appears:
{noformat}
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must 
be unique: org.apache.hadoop:hadoop-minikdc:jar -> duplicate declaration of 
version (?) @ line 234, column 17
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-16 Thread Kuhu Shukla 
+1 (non-binding)-Downloaded source.-Verified signatures.- Compiled the 
source.-Ran sample jobs like MR sleep on pseudo distributed cluster. (Mac OS)
Thanks Junping and others!Regards,Kuhu
On Wednesday, March 15, 2017, 7:25:46 PM CDT, Junping Du  
wrote:bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client 
to pull in HDFS-related code. It doesn't have its own dependency on 
hadoop-hdfs. So I think this affects users of the hadoop-client artifact, which 
has existed for a long time.

I could miss that. Thanks for reminding! From my quick check: 
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
sounds like 669 artifacts from other projects were depending on it.


I think we should withdraw the current RC bits. Please stop the verification & 
vote.

I will kick off another RC immediately when HDFS-11431 get fixed.


Thanks,


Junping



From: Andrew Wang 
Sent: Wednesday, March 15, 2017 2:04 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

Hi Junping, inline,


>From my understanding, this issue is related to our previous improvements with 
>separating client and server jars in HDFS-6200. If we use the new "client" jar 
>in NN HA deployment, then we will hit the issue reported.

>From my read of the poms, hadoop-client depends on hadoop-hdfs-client to pull 
>in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. So I 
>think this affects users of the hadoop-client artifact, which has existed for 
>a long time.

Essentially all of our customer deployments run with NN HA, so this would 
affect a lot of users.

I can see two options here:

- Without any change in 2.8.0, if user hit the issue when they deploy HA 
cluster by using new client jar, adding back hdfs jar just like how things work 
previously

- Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
to client jar or adding dependency between client jar and server jar. There 
must be some arguments there on which way to fix is better especially 
ConfiguredFailoverProxyProvider still has some sever side dependencies.


I would prefer the first option, given:

- The issue fixing time is unpredictable as there are still discussion on how 
to fix this issue. Our 2.8.0 release shouldn't be an endless journey which has 
been deferred several times for more serious issue.

Looks like we have a patch being actively revved and reviewed to fix this by 
making hadoop-hdfs-client depend on hadoop-hdfs. Thanks to Steven and Steve for 
working on this.

Steve proposed doing a proper split in a later JIRA.

- We have workaround for this improvement, no regression happens due to this 
issue. People can still use hdfs jar in old way. The worst case is improvement 
for HDFS doesn't work in some cases - that shouldn't block the whole release.

Based on the above, I think there is a regression for users of the 
hadoop-client artifact.

If it actually only affects users of hadoop-hdfs-client, then I agree we can 
document it as a Known Issue and fix it later.

Best,
Andrew

[jira] [Created] (HADOOP-14190) add more on s3 regions to the s3a documentation

2017-03-16 Thread Steve Loughran (JIRA) 
Steve Loughran created HADOOP-14190:
---

 Summary: add more on s3 regions to the s3a documentation
 Key: HADOOP-14190
 URL: https://issues.apache.org/jira/browse/HADOOP-14190
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: documentation, fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor


As part of some other documentation, I have the complete list of current S3 
regions. Add these to src/test/resources/core-site.xml, and list in docs

There's also one extra option: declare the XML file as a resource in 
src/main/resources, one which people could then load into a config file. 
Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-16 Thread Steve Loughran 

> On 15 Mar 2017, at 23:04, Josh Elser  wrote:
> 
> A tag is immutable, but you (or someone else) could remove the tag you pushed 
> and re-push a new one. That's why the commit id is important -- it ensures 
> that everyone else knows the exact commit being voted on.
> 

There's tag signing too, "git tag --sign". We can/should use that for 
authenticating tags, saying that "the release is tag 2.8.x signed by me"

> Junping Du wrote:
>> The latest commit on RC2 is: e51312e8e106efb2ebd4844eecacb51026fac8b7.
>> btw, I think tags are immutable. Isn't it?
>> 
>> Thanks,
>> 
>> Junping
>> 
>> From: Steve Loughran
>> Sent: Wednesday, March 15, 2017 12:30 PM
>> To: Junping Du
>> Cc: common-dev@hadoop.apache.org
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)
>> 
>>> On 14 Mar 2017, at 08:41, Junping Du  wrote:
>>> 
>>> Hi all,
>>> With several important fixes get merged last week, I've created a new 
>>> release candidate (RC2) for Apache Hadoop 2.8.0.
>>> 
>>> This is the next minor release to follow up 2.7.0 which has been 
>>> released for more than 1 year. It comprises 2,919 fixes, improvements, and 
>>> new features. Most of these commits are released for the first time in 
>>> branch-2.
>>> 
>>>  More information about the 2.8.0 release plan can be found here: 
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>> 
>>>  Please note that RC0 and RC1 are not voted public because significant 
>>> issues are found just after RC tag getting published.
>>> 
>>>  The RC is available at: 
>>> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
>>> 
>>>  The RC tag in git is: release-2.8.0-RC2
>> 
>> given tags are so easy to move, we need to be relying on one or more of:
>> -the commit ID,
>> -the tag being signed
>> 
>> Junping: what is the commit Id for the release?
>> 
>>>  The maven artifacts are available via repository.apache.org at: 
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1056
>>> 
>> 
>> thanks, I'll play with these downstream, as well as checking out and trying 
>> to build on windows
>> 
>>>  Please try the release and vote; the vote will run for the usual 5 
>>> days, ending on 03/20/2017 PDT time.
>>> 
>>> Thanks,
>>> 
>>> Junping
>> 
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: test failures - newbie help needed

2017-03-16 Thread Steve Loughran 
some of this looks network related. Ubuntu often makes a mess of the loopback 
address of localhost: find the localhost entry in /etc/hosts and change it to 
127.0.0.1 and see if that helps

otherwise, the hdfs-dev list is the one to really ping about HDFS issues, which 
all of these appear to be. Mostly related to XAttrs, interestingly enough


Pick one isolated test, like testWebHDFSXAttr#FSXAttrBaseTest.testListXAttr ; 
run it with the log4j.properties in hdfs src/test/resources set to log 
org.apache.hadoop at debug and see what is going on underneath. That's 
generally how I start debugging test failures, before switching to the IDE and 
stepping through.

Good luck

Steve

PS if it makes you feel any better, staring a test run failures is a core skill 
of all Hadoop developers


> On 16 Mar 2017, at 10:33, Simon Scott  wrote:
> 
> Hi
> 
> Please excuse me. I am new to building Hadoop. I am attempting to build and 
> run tests for the trunk and encountering a series of test failures which for 
> sure indicates that I am doing something wrong. I have scoured the wiki and 
> can't see what I'm missing. Can anybody offer any advice?
> 
> I am running the tests in Ubuntu 14.04 hosted in VirtualBox on Windows 8.1 
> host.
> 
> Using this command: "mvn clean test"
> 
> List of failed tests below.
> 
> Many thanks
> Simon
> 
> Failed tests:
>  
> TestHDFSFileSystemContract>FileSystemContractBaseTest.testRenameDirectoryAsExistingDirectory:536
>  Renamed nested file1 exists
>  TestTrashWithEncryptionZones.testDeleteEZWithMultipleUsers:164 Non-admin 
> could delete an encryption zone with multiple users : /zones/zone1 
> expected:<1> but was:<0>
>  TestDataNodeVolumeFailure.testFailedVolumeBeingRemovedFromDataNode:247 
> expected:<1> but was:<2>
>  
> TestFileContextXAttr>FSXAttrBaseTest.testUnreadableBySuperuserXAttr:1142->FSXAttrBaseTest.access$000:67->FSXAttrBaseTest.doTestUnreadableBySuperuserXAttr:1158->FSXAttrBaseTest.doTUBSXAInt:1213->FSXAttrBaseTest.verifyFileAccess:1241
>  open failed but expected it to succeed
>  TestFileContextXAttr>FSXAttrBaseTest.testRawXAttrs:1024 setXAttr should have 
> thrown
>  TestFileContextXAttr>FSXAttrBaseTest.testListXAttrs:820 expected IOException
>  TestFileContextXAttr>FSXAttrBaseTest.testGetXAttrs:458 expected IOException
>  
> TestNameNodeXAttr>FSXAttrBaseTest.testUnreadableBySuperuserXAttr:1142->FSXAttrBaseTest.access$000:67->FSXAttrBaseTest.doTestUnreadableBySuperuserXAttr:1158->FSXAttrBaseTest.doTUBSXAInt:1213->FSXAttrBaseTest.verifyFileAccess:1241
>  open failed but expected it to succeed
>  TestNameNodeXAttr>FSXAttrBaseTest.testRawXAttrs:1024 setXAttr should have 
> thrown
>  TestNameNodeXAttr>FSXAttrBaseTest.testListXAttrs:820 expected IOException
>  TestNameNodeXAttr>FSXAttrBaseTest.testGetXAttrs:458 expected IOException
>  
> TestNamenodeCapacityReport.testXceiverCount:198->testXceiverCountInternal:344 
> Test resulted in an unexpected exit
>  TestEncryptionZonesWithKMS>TestEncryptionZones.testGetEZAsNonSuperUser:581 
> expected AccessControlException
>  TestEncryptionZonesWithKMS>TestEncryptionZones.testBasicOperations:441 
> createEncryptionZone is superuser-only operation
>  TestReservedRawPaths.testAdminAccessOnly:262 access to /.reserved/raw is 
> superuser-only operation
>  TestDFSShell.testSetXAttrPermission:2992 Returned should be 1 expected:<1> 
> but was:<0>
>  
> TestWebHDFSXAttr>FSXAttrBaseTest.testUnreadableBySuperuserXAttr:1142->FSXAttrBaseTest.access$000:67->FSXAttrBaseTest.doTestUnreadableBySuperuserXAttr:1158->FSXAttrBaseTest.doTUBSXAInt:1213->FSXAttrBaseTest.verifyFileAccess:1241
>  open failed but expected it to succeed
>  TestWebHDFSXAttr>FSXAttrBaseTest.testRawXAttrs:1024 setXAttr should have 
> thrown
>  TestWebHDFSXAttr>FSXAttrBaseTest.testListXAttrs:820 expected IOException
>  TestWebHDFSXAttr>FSXAttrBaseTest.testGetXAttrs:458 expected IOException
>  
> TestWebHdfsFileSystemContract>FileSystemContractBaseTest.testRenameDirectoryAsExistingDirectory:536
>  Renamed nested file1 exists
>  TestEncryptionZones.testGetEZAsNonSuperUser:581 expected 
> AccessControlException
>  TestEncryptionZones.testBasicOperations:441 createEncryptionZone is 
> superuser-only operation
> 
> Tests in error:
>  TestRBWBlockInvalidation.testBlockInvalidationWhenRBWReplicaMissedInDN:122 > 
>  ...
>  TestDataNodeVolumeFailure.tearDown:142->Object.wait:-2 >  test timed out 
> after...
>  TestDataNodeVolumeFailureReporting.tearDown:109->Object.wait:-2 >  test 
> timed ...
>  TestDataNodeVolumeFailureReporting.tearDown:109 >  test timed out after 
> 12...
>  TestDataNodeVolumeFailureReporting.tearDown:109 >  test timed out after 
> 12...
>  TestDataNodeVolumeFailureReporting.tearDown:109->Object.wait:-2 >  test 
> timed ...
>  TestFileContextXAttr>FSXAttrBaseTest.testRemoveXAttrPermissions:656 > Remote 
> N...
>  
> TestNameNodeMetadataConsistency.testGenerationStampInFuture:109->waitForNumBytes:161
>  > Timeout
>

test failures - newbie help needed

2017-03-16 Thread Simon Scott 
Hi

Please excuse me. I am new to building Hadoop. I am attempting to build and run 
tests for the trunk and encountering a series of test failures which for sure 
indicates that I am doing something wrong. I have scoured the wiki and can't 
see what I'm missing. Can anybody offer any advice?

I am running the tests in Ubuntu 14.04 hosted in VirtualBox on Windows 8.1 host.

Using this command: "mvn clean test"

List of failed tests below.

Many thanks
Simon

Failed tests:
  
TestHDFSFileSystemContract>FileSystemContractBaseTest.testRenameDirectoryAsExistingDirectory:536
 Renamed nested file1 exists
  TestTrashWithEncryptionZones.testDeleteEZWithMultipleUsers:164 Non-admin 
could delete an encryption zone with multiple users : /zones/zone1 expected:<1> 
but was:<0>
  TestDataNodeVolumeFailure.testFailedVolumeBeingRemovedFromDataNode:247 
expected:<1> but was:<2>
  
TestFileContextXAttr>FSXAttrBaseTest.testUnreadableBySuperuserXAttr:1142->FSXAttrBaseTest.access$000:67->FSXAttrBaseTest.doTestUnreadableBySuperuserXAttr:1158->FSXAttrBaseTest.doTUBSXAInt:1213->FSXAttrBaseTest.verifyFileAccess:1241
 open failed but expected it to succeed
  TestFileContextXAttr>FSXAttrBaseTest.testRawXAttrs:1024 setXAttr should have 
thrown
  TestFileContextXAttr>FSXAttrBaseTest.testListXAttrs:820 expected IOException
  TestFileContextXAttr>FSXAttrBaseTest.testGetXAttrs:458 expected IOException
  
TestNameNodeXAttr>FSXAttrBaseTest.testUnreadableBySuperuserXAttr:1142->FSXAttrBaseTest.access$000:67->FSXAttrBaseTest.doTestUnreadableBySuperuserXAttr:1158->FSXAttrBaseTest.doTUBSXAInt:1213->FSXAttrBaseTest.verifyFileAccess:1241
 open failed but expected it to succeed
  TestNameNodeXAttr>FSXAttrBaseTest.testRawXAttrs:1024 setXAttr should have 
thrown
  TestNameNodeXAttr>FSXAttrBaseTest.testListXAttrs:820 expected IOException
  TestNameNodeXAttr>FSXAttrBaseTest.testGetXAttrs:458 expected IOException
  TestNamenodeCapacityReport.testXceiverCount:198->testXceiverCountInternal:344 
Test resulted in an unexpected exit
  TestEncryptionZonesWithKMS>TestEncryptionZones.testGetEZAsNonSuperUser:581 
expected AccessControlException
  TestEncryptionZonesWithKMS>TestEncryptionZones.testBasicOperations:441 
createEncryptionZone is superuser-only operation
  TestReservedRawPaths.testAdminAccessOnly:262 access to /.reserved/raw is 
superuser-only operation
  TestDFSShell.testSetXAttrPermission:2992 Returned should be 1 expected:<1> 
but was:<0>
  
TestWebHDFSXAttr>FSXAttrBaseTest.testUnreadableBySuperuserXAttr:1142->FSXAttrBaseTest.access$000:67->FSXAttrBaseTest.doTestUnreadableBySuperuserXAttr:1158->FSXAttrBaseTest.doTUBSXAInt:1213->FSXAttrBaseTest.verifyFileAccess:1241
 open failed but expected it to succeed
  TestWebHDFSXAttr>FSXAttrBaseTest.testRawXAttrs:1024 setXAttr should have 
thrown
  TestWebHDFSXAttr>FSXAttrBaseTest.testListXAttrs:820 expected IOException
  TestWebHDFSXAttr>FSXAttrBaseTest.testGetXAttrs:458 expected IOException
  
TestWebHdfsFileSystemContract>FileSystemContractBaseTest.testRenameDirectoryAsExistingDirectory:536
 Renamed nested file1 exists
  TestEncryptionZones.testGetEZAsNonSuperUser:581 expected 
AccessControlException
  TestEncryptionZones.testBasicOperations:441 createEncryptionZone is 
superuser-only operation

Tests in error:
  TestRBWBlockInvalidation.testBlockInvalidationWhenRBWReplicaMissedInDN:122 >  
...
  TestDataNodeVolumeFailure.tearDown:142->Object.wait:-2 >  test timed out 
after...
  TestDataNodeVolumeFailureReporting.tearDown:109->Object.wait:-2 >  test timed 
...
  TestDataNodeVolumeFailureReporting.tearDown:109 >  test timed out after 
12...
  TestDataNodeVolumeFailureReporting.tearDown:109 >  test timed out after 
12...
  TestDataNodeVolumeFailureReporting.tearDown:109->Object.wait:-2 >  test timed 
...
  TestFileContextXAttr>FSXAttrBaseTest.testRemoveXAttrPermissions:656 > Remote 
N...
  
TestNameNodeMetadataConsistency.testGenerationStampInFuture:109->waitForNumBytes:161
 > Timeout
  TestNameNodeXAttr>FSXAttrBaseTest.testRemoveXAttrPermissions:656 > Remote No 
m...
  TestFSImage.testCompression:96->setCompressCodec:102->testPersistHelper:108 > 
IO
  TestDiskBalancerCommand.testRunMultipleCommandsUnderOneSetup > Remote File 
/sy...
  TestDiskBalancer.testBalanceDataBetweenMultiplePairsOfVolumes:187 > 
IllegalArgument
  TestDiskBalancerRPC.testMoveBlockAcrossVolume > Remote File /tmp.txt could 
onl...
  TestDFSRSDefault10x4StripedOutputStream>TestDFSStripedOutputStream.setup:91 > 
DiskError
  
TestDFSRSDefault10x4StripedOutputStream>TestDFSStripedOutputStream.testFileSmallerThanOneStripe2:132->TestDFSStripedOutputStream.testOneFile:187
 > Timeout
  
TestDFSRSDefault10x4StripedOutputStream>TestDFSStripedOutputStream.testFileMoreThanABlockGroup3:176->TestDFSStripedOutputStream.testOneFile:186
 > IO
  TestParallelRead.setupCluster:37->TestParallelReadUtil.setupCluster:71 > IO 
Ca...
  TestParallelRead.teardownCluster:42->TestParallelReadUtil.teardownCluster:394 
> NullPointer

[jira] [Created] (HADOOP-14189) add distcp-site.xml for distcp on branch-2

2017-03-16 Thread Fei Hui (JIRA) 
Fei Hui created HADOOP-14189:


 Summary: add distcp-site.xml for distcp on branch-2
 Key: HADOOP-14189
 URL: https://issues.apache.org/jira/browse/HADOOP-14189
 Project: Hadoop Common
  Issue Type: Task
  Components: tools/distcp
Reporter: Fei Hui


On hadoop 2.x , we could not config hadoop parameters for distcp. It only uses 
distcp-default.xml.
We should add distcp-site.xml to overrides hadoop paramers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org