Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/63/

[Dec 7, 2017 9:15:21 PM] (mackrorysd) HADOOP-15080.  Aliyun OSS: update oss sdk 
from 2.8.1 to 2.8.3 to remove
[Dec 8, 2017 5:20:33 AM] (xiao) HADOOP-15012. Add readahead, dropbehind, and 
unbuffer to
[Dec 8, 2017 5:20:33 AM] (xiao) HADOOP-15056. Fix 
TestUnbuffer#testUnbufferException failure.
[Dec 8, 2017 5:26:12 AM] (xiao) HADOOP-14872. CryptoInputStream should 
implement unbuffer. Contributed
[Dec 8, 2017 5:53:33 AM] (wwei) YARN-7607. Remove the trailing duplicated 
timestamp in container
[Dec 8, 2017 12:08:16 PM] (sammi.chen) HADOOP-14997. Add hadoop-aliyun as 
dependency of hadoop-cloud-storage.
[Dec 8, 2017 2:02:00 PM] (sammi.chen) HADOOP-15024. AliyunOSS: Support user 
agent configuration and include




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-common:1 
   hadoop-hdfs:21 
   bkjournal:5 
   hadoop-mapreduce-client-jobclient:4 
   hadoop-distcp:2 
   hadoop-extras:1 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-yarn-client:9 
   hadoop-yarn-server-timelineservice:1 

Failed junit tests :

   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.hdfs.TestSetrepIncreasing 
   hadoop.hdfs.TestTrashWithSecureEncryptionZones 
   hadoop.hdfs.TestBlocksScheduledCounter 
   hadoop.hdfs.TestCrcCorruption 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.TestHDFSTrash 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 
   hadoop.yarn.server.TestContainerManagerSecurity 

Timed out junit tests :

   org.apache.hadoop.log.TestLogLevel 
   org.apache.hadoop.hdfs.TestLeaseRecovery2 
   org.apache.hadoop.hdfs.TestMaintenanceState 
   org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade 
   org.apache.hadoop.hdfs.TestHDFSFileSystemContract 
   org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage 
   org.apache.hadoop.hdfs.web.TestWebHdfsTokens 
   org.apache.hadoop.hdfs.TestFileCreationDelete 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter 
   org.apache.hadoop.hdfs.TestBlockStoragePolicy 
   org.apache.hadoop.hdfs.TestDFSOutputStream 
   org.apache.hadoop.hdfs.web.TestWebHDFS 
   org.apache.hadoop.hdfs.TestAppendSnapshotTruncate 
   org.apache.hadoop.hdfs.web.TestWebHDFSXAttr 
   org.apache.hadoop.hdfs.TestRollingUpgradeRollback 
   org.apache.hadoop.hdfs.TestMiniDFSCluster 
   org.apache.hadoop.hdfs.TestDFSShell 
   org.apache.hadoop.hdfs.TestDataTransferProtocol 
   org.apache.hadoop.hdfs.web.TestWebHDFSAcl 
   org.apache.hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperEditLogStreams 
   org.apache.hadoop.contrib.bkjournal.TestCurrentInprogress 
   org.apache.hadoop.mapred.TestClusterMapReduceTestCase 
   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption 
   org.apache.hadoop.mapred.TestMRTimelineEventHandling 
   org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromTarget 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromSource 
   org.apache.hadoop.tools.TestCopyFiles 
   
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy 
   org.apache.hadoop.yarn.client.TestRMFailover 
   org.apache.hadoop.yarn.client.cli.TestYarnCLI 
   org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA 
   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClientWithReservation 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClient 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
   org.apache.hadoop.yarn.client.api.impl.TestNMClient 
   
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/63/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/616/

[Dec 7, 2017 7:15:40 PM] (lei) HDFS-12840. Creating a file with non-default EC 
policy in a EC zone is
[Dec 7, 2017 7:30:58 PM] (mackrorysd) HADOOP-15098. 
TestClusterTopology#testChooseRandom fails intermittently.
[Dec 7, 2017 10:50:30 PM] (weichiu) HDFS-11915. Sync rbw dir on the first 
hsync() to avoid file lost on
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-6471. Support to add min/max resource 
configuration for a queue.
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7254. UI and metrics changes related to 
absolute resource
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7332. Compute effectiveCapacity per each 
resource vector. (Sunil G
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7411. Inter-Queue preemption's 
computeFixpointAllocation need to
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7482. Max applications calculation per 
queue has to be retrospected
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7483. CapacityScheduler test cases 
cleanup post YARN-5881. (Sunil G
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7538. Fix performance regression 
introduced by Capacity Scheduler
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7544. Use 
queue-path.capacity/maximum-capacity to specify absolute
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7564. Cleanup to fix checkstyle issues 
of YARN-5881 branch.
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7575. NPE in scheduler UI when 
max-capacity is not configured.
[Dec 8, 2017 2:56:54 AM] (wangda) YARN-7533. Documentation for absolute 
resource support in Capacity
[Dec 8, 2017 5:05:55 AM] (xiao) HADOOP-15056. Fix 
TestUnbuffer#testUnbufferException failure.
[Dec 8, 2017 3:03:54 PM] (zhengkai.zk) HADOOP-15104. AliyunOSS: change the 
default value of max error retry.
[Dec 8, 2017 4:00:21 PM] (vinodkv) HADOOP-15059. Undoing the switch of 
Credentials to PB format as default




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Possible null pointer dereference of replication in 
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
 Short, Byte) Dereferenced at INodeFile.java:replication in 
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
 Short, Byte) Dereferenced at INodeFile.java:[line 210] 

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.hdfs.web.TestWebHDFSAcl 
   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade 
   hadoop.hdfs.server.namenode.ha.TestHAStateTransitions 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 
   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS 
   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock 
   hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler 
   hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks 
   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR 
   hadoop.hdfs.server.namenode.TestQuotaByStorageType 
   hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA 
   hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover 
   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes 
   hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot 
   hadoop.hdfs.server.namenode.ha.TestHAMetrics 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA 
   hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId 
   hadoop.hdfs.TestDatanodeReport 
   hadoop.cli.TestCacheAdminCLI 
   hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir 
   hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks 
   hadoop.hdfs.server.namenode.TestMetadataVersionOutput 
   hadoop.hdfs.TestHdfsAdmin 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.web.TestWebHDFSXAttr 
   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   hadoop.hdfs.server.namenode.TestSecureNameNode 
   hadoop.hdfs.server.namenode.TestINodeAttributeProvider 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   

[VOTE] Release Apache Hadoop 3.0.0 RC1

[Andrew Wang]

Hi all,

Let me start, as always, by thanking the efforts of all the contributors
who contributed to this release, especially those who jumped on the issues
found in RC0.

I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
fixed JIRAs since the previous 3.0.0-beta1 release.

You can find the artifacts here:

http://home.apache.org/~wang/3.0.0-RC1/

I've done the traditional testing of building from the source tarball and
running a Pi job on a single node cluster. I also verified that the shaded
jars are not empty.

Found one issue that create-release (probably due to the mvn deploy change)
didn't sign the artifacts, but I fixed that by calling mvn one more time.
Available here:

https://repository.apache.org/content/repositories/orgapachehadoop-1075/

This release will run the standard 5 days, closing on Dec 13th at 12:31pm
Pacific. My +1 to start.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

FYI that we got our last blocker in today, so I'm currently rolling RC1.
Stay tuned!

On Thu, Nov 30, 2017 at 8:32 AM, Allen Wittenauer 
wrote:

>
> > On Nov 30, 2017, at 1:07 AM, Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
> >
> >
> > >. If ATSv1 isn’t replaced by ATSv2, then why is it marked deprecated?
> > Ideally it should not be. Can you point out where it is marked as
> deprecated? If it is in historyserver daemon start, that change made very
> long back when timeline server added.
>
>
> Ahh, I see where all the problems lie.  No one is paying attention to the
> deprecation message because it’s kind of oddly worded:
>
> * It really means “don’t use ‘yarn historyserver’ use ‘yarn
> timelineserver’ ”
> * ‘yarn historyserver’ was removed from the documentation in 2.7.0
> * ‘yarn historyserver’ doesn’t appear in the yarn usage output
> * ‘yarn timelineserver’ runs the exact same class
>
> There’s no reason for ‘yarn historyserver’ to exist in 3.x.  Just run
> ‘yarn timelineserver’ instead.
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Resolved] (MAPREDUCE-7019) java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-7019.
---
Resolution: Invalid

Closing this since I believe the error is coming from the program being 
launched by the streaming job rather than an issue with the streaming framework 
code.  If this is incorrect, please provide details showing where the streaming 
framework code is going awry.


> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed 
> with code 2
> -
>
> Key: MAPREDUCE-7019
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7019
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: shrutika sarda
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/62/

[Dec 7, 2017 4:24:45 AM] (aajisaka) HDFS-12889. Addendum patch to add missing 
file.
[Dec 7, 2017 8:43:21 AM] (cdouglas) HDFS-11576. Block recovery will fail 
indefinitely if recovery time >




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-common:1 
   hadoop-hdfs:16 
   bkjournal:5 
   hadoop-yarn-server-resourcemanager:1 
   hadoop-yarn-client:8 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-jobclient:12 
   hadoop-distcp:3 

Failed junit tests :

   hadoop.crypto.key.kms.server.TestKMS 
   
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 
   hadoop.mapreduce.security.ssl.TestEncryptedShuffle 
   hadoop.mapreduce.security.TestMRCredentials 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 

Timed out junit tests :

   org.apache.hadoop.log.TestLogLevel 
   org.apache.hadoop.hdfs.TestLeaseRecovery2 
   org.apache.hadoop.security.TestPermission 
   org.apache.hadoop.hdfs.web.TestWebHdfsTokens 
   org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream 
   org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade 
   org.apache.hadoop.hdfs.TestFileAppendRestart 
   org.apache.hadoop.hdfs.security.TestDelegationToken 
   org.apache.hadoop.hdfs.TestDFSMkdirs 
   org.apache.hadoop.hdfs.TestDFSOutputStream 
   org.apache.hadoop.hdfs.web.TestWebHDFS 
   org.apache.hadoop.hdfs.web.TestWebHDFSXAttr 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes 
   org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs 
   org.apache.hadoop.hdfs.TestDistributedFileSystem 
   org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication 
   org.apache.hadoop.hdfs.TestDFSShell 
   org.apache.hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy 
   org.apache.hadoop.yarn.client.TestRMFailover 
   org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA 
   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClientWithReservation 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClient 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
   org.apache.hadoop.yarn.client.api.impl.TestNMClient 
   
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell 
   org.apache.hadoop.fs.TestFileSystem 
   org.apache.hadoop.mapred.TestMiniMRClasspath 
   org.apache.hadoop.mapred.TestClusterMapReduceTestCase 
   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption 
   org.apache.hadoop.mapred.TestJobSysDirWithDFS 
   org.apache.hadoop.mapreduce.security.TestBinaryTokenFile 
   org.apache.hadoop.mapred.TestMRTimelineEventHandling 
   org.apache.hadoop.mapred.join.TestDatamerge 
   org.apache.hadoop.mapred.TestReduceFetch 
   org.apache.hadoop.conf.TestNoDefaultsJobConf 
   org.apache.hadoop.tools.TestDistCpSync 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromTarget 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromSource 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/62/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/62/artifact/out/diff-compile-javac-root.txt
  [324K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/62/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/62/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/62/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs: