Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-01-24 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/

[Jan 23, 2018 5:09:11 PM] (stevel) HADOOP-15185. Update adls connector to use 
the current version of ADLS
[Jan 23, 2018 6:53:27 PM] (jianhe) YARN-7766. Introduce a new config property 
for YARN Service dependency
[Jan 23, 2018 10:03:53 PM] (jianhe) YARN-7782. Enable user re-mapping for 
Docker containers in
[Jan 24, 2018 1:54:39 AM] (billie) YARN-7540 and YARN-7605. Convert yarn app 
cli to call yarn api services
[Jan 24, 2018 2:43:36 AM] (yqlin) HDFS-12963. Error log level in 
ShortCircuitRegistry#removeShm.
[Jan 24, 2018 3:15:44 AM] (inigoiri) HDFS-12772. RBF: Federation Router State 
State Store internal API.
[Jan 24, 2018 5:07:05 AM] (szegedim) YARN-7796. Container-executor fails with 
segfault on certain OS
[Jan 24, 2018 9:34:15 AM] (rohithsharmaks) Revert "YARN-7537 [Atsv2] load hbase 
configuration from filesystem
[Jan 24, 2018 10:26:59 AM] (rohithsharmaks) YARN-7749. [UI2] GPU information 
tab in left hand side disappears when
[Jan 24, 2018 11:26:13 AM] (sunilg) YARN-7806. Distributed Shell should use 
timeline async api's.




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.hdfs.TestLeaseRecovery2 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped 
   hadoop.yarn.server.nodemanager.TestNMAuditLogger 
   
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
 
   hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesApps 
   hadoop.yarn.server.nodemanager.TestNodeManagerShutdown 
   hadoop.yarn.server.nodemanager.webapp.TestNMWebServices 
   hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage 
   hadoop.yarn.server.nodemanager.TestNodeManagerReboot 
   hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesContainers 
   
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapreduce.lib.output.TestJobOutputCommitter 
   hadoop.mapreduce.v2.TestMROldApiJobs 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapred.TestMRTimelineEventHandling 
   hadoop.mapred.TestJobCleanup 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/diff-compile-javac-root.txt
  [280K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/whitespace-tabs.txt
  [292K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [252K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [304K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/667/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [16K]

[jira] [Created] (HADOOP-15191) Add Private/Unstable BulkDelete operations to FS/S3A for DistCP

2018-01-24 Thread Steve Loughran (JIRA) 
Steve Loughran created HADOOP-15191:
---

 Summary: Add Private/Unstable BulkDelete operations to FS/S3A for 
DistCP
 Key: HADOOP-15191
 URL: https://issues.apache.org/jira/browse/HADOOP-15191
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, tools/distcp
Affects Versions: 2.9.0
Reporter: Steve Loughran


Large scale DistCP with the -delete option doesn't finish in a viable time 
because of the final CopyCommitter doing a 1 by 1 delete of all missing files. 
This isn't randomized (the list is sorted), and it's throttled by AWS.

If bulk deletion of files was exposed as an API, distCP would do 1/1000 of the 
REST calls, so not get throttled.

Proposed: add an initially private/unstable interface for stores, 
{{BulkDelete}} which declares a page size and offers a 
{{bulkDelete(List)}} operation for the bulk deletion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Review for HADOOP-14918 heeded

2018-01-24 Thread Aaron Fabbri 
Yep, I'll review it now.

On Wed, Jan 24, 2018 at 9:19 AM, Steve Loughran 
wrote:

> Can I get a review of HADOOP-14918
>
> All it does is rm the local dynamo test for S3Guard because the relevant
> JARs don't get updated by AWS any more, and it is out of sync with the
> latest AWS SDKs. If you (experimentally) upgrade the AWS SDK then you get
> missing method errors in places
>
> Doesn't add code, doesn't add maintenance, just takes away a local test
> option which simplifies the text matrics. thanks
>
> -steve
>

[jira] [Created] (HADOOP-15190) Use Jacoco to generate Unit Test coverage reports

2018-01-24 Thread Duo Xu (JIRA) 
Duo Xu created HADOOP-15190:
---

 Summary: Use Jacoco to generate Unit Test coverage reports
 Key: HADOOP-15190
 URL: https://issues.apache.org/jira/browse/HADOOP-15190
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Reporter: Duo Xu
Assignee: Duo Xu


Currently Hadoop is using maven-clover2-plugin for code coverage, which is 
outdated. And Atlassian open-sourced clover last year so license cannot be 
purchased although we can switch to use the license-free version called 
"openclover".

This Jira is to replace clover with Jacoco, which is actively maintained by the 
community.

  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Review for HADOOP-14918 heeded

2018-01-24 Thread Steve Loughran 
Can I get a review of HADOOP-14918

All it does is rm the local dynamo test for S3Guard because the relevant JARs 
don't get updated by AWS any more, and it is out of sync with the latest AWS 
SDKs. If you (experimentally) upgrade the AWS SDK then you get missing method 
errors in places

Doesn't add code, doesn't add maintenance, just takes away a local test option 
which simplifies the text matrics. thanks

-steve

[jira] [Created] (HADOOP-15189) backport HADOOP-15039 to branch-2 and branch-3

2018-01-24 Thread Genmao Yu (JIRA) 
Genmao Yu created HADOOP-15189:
--

 Summary: backport HADOOP-15039 to branch-2 and branch-3
 Key: HADOOP-15189
 URL: https://issues.apache.org/jira/browse/HADOOP-15189
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Genmao Yu
Assignee: Genmao Yu






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2018-01-24 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/114/

No changes




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-common:1 
   hadoop-hdfs:42 
   bkjournal:1 
   hadoop-yarn-server-resourcemanager:1 
   hadoop-yarn-client:4 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-jobclient:2 
   hadoop-distcp:3 
   hadoop-archives:1 
   hadoop-extras:1 

Failed junit tests :

   hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing 
   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.server.namenode.TestQuotaByStorageType 
   hadoop.hdfs.server.namenode.ha.TestHAMetrics 
   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog 
   hadoop.hdfs.server.namenode.TestFileLimit 
   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits 
   hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots 
   hadoop.hdfs.server.namenode.TestFSImageWithAcl 
   hadoop.hdfs.server.federation.router.TestRouterRpc 
   hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd 
   hadoop.hdfs.server.namenode.TestListOpenFiles 
   hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover 
   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup 
   hadoop.hdfs.server.namenode.TestEditLogAutoroll 
   hadoop.hdfs.server.namenode.TestStreamFile 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot 
   hadoop.hdfs.server.namenode.TestDecommissioningStatus 
   hadoop.hdfs.server.namenode.TestAuditLogger 
   hadoop.hdfs.server.namenode.TestGenericJournalConf 
   hadoop.hdfs.server.namenode.TestTransferFsImage 
   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.namenode.TestAclConfigFlag 
   hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA 
   hadoop.hdfs.server.namenode.TestSecondaryWebUi 
   hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap 
   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA 
   hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot 
   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.federation.router.TestNamenodeHeartbeat 
   hadoop.hdfs.server.namenode.TestCacheDirectives 
   hadoop.hdfs.server.namenode.TestProtectedDirectories 
   hadoop.hdfs.server.namenode.TestLargeDirectoryDelete 
   hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot 
   hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade 
   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshot 
   hadoop.hdfs.server.namenode.TestBackupNode 
   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer 
   hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA 
   hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination 
   hadoop.hdfs.server.federation.router.TestRouterMountTable 
   hadoop.hdfs.server.namenode.TestStartup 
   hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot 
   
hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestHAStateTransitions 
   hadoop.hdfs.server.namenode.TestSaveNamespace 
   hadoop.hdfs.server.namenode.TestNameNodeRpcServerMethods 
   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
   hadoop.hdfs.server.federation.store.driver.TestStateStoreFileSystem 
   hadoop.hdfs.server.namenode.TestDeadDatanode 
   hadoop.hdfs.server.namenode.ha.TestEditLogTailer 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics 
   hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
   hadoop.hdfs.server.namenode.TestEditLogJournalFailures 
   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes