Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

2018-04-17 Thread Apache Jenkins Server 
For more details, see https://builds.apache.org/job/hadoop-trunk-win/440/

[Apr 16, 2018 5:15:37 PM] (xyao) HADOOP-15369. Avoid usage of 
${project.version} in parent poms.
[Apr 16, 2018 6:24:26 PM] (eyang) YARN-8060.  Added default readiness check for 
service components.   
[Apr 16, 2018 9:16:59 PM] (arp) HDFS-13439. Add test case for read block 
operation when it is moved.
[Apr 16, 2018 10:05:04 PM] (haibochen) YARN-8162. Remove Method 
DirectoryCollection#verifyDirUsingMkdir. (Yufei
[Apr 16, 2018 10:55:00 PM] (virajith) HDFS-13311. RBF: 
TestRouterAdminCLI#testCreateInvalidEntry fails on
[Apr 16, 2018 11:26:54 PM] (inigoiri) YARN-7773. YARN Federation used Mysql as 
state store throw exception,
[Apr 16, 2018 11:52:17 PM] (inigoiri) YARN-8165. Incorrect queue name logging 
in AbstractContainerAllocator.
[Apr 17, 2018 12:32:19 AM] (inigoiri) YARN-8164. Fix a potential NPE in 
AbstractSchedulerPlanFollower.
[Apr 17, 2018 3:23:22 AM] (yqlin) HDFS-13435. RBF: Improve the error loggings 
for printing the stack
[Apr 17, 2018 9:38:31 AM] (sunilg) YARN-8126. Support auto-spawning of admin 
configured services during




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFileUtil 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestSymlinkLocalFSFileContext 
   hadoop.fs.TestTrash 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestNativeCodeLoader 
   hadoop.util.TestNodeHealthScriptRunner 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestHSync 
   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancer 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport 
   hadoop.hdfs.server.namenode.TestAddBlock 
   hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands

[RESULT] [VOTE] Release Apache Hadoop 2.7.6 (RC0)

2018-04-17 Thread Konstantin Shvachko 
Hi everybody,

With 4 binding and 4 non-binding +1s and no -1s the vote for Apache Release
2.7.6 passes.
Thank you everybody for contributing to the release, testing, and voting.

Binding +1s
Zhe Zhang
Brahma Reddy Battula
Jason Lowe
Konstantin Shvachko

Non-binding +1s
Chen Liang
Erik Krogen
Takanobu Asanuma
Ajay Kumar



On Mon, Apr 9, 2018 at 4:14 PM, Konstantin Shvachko 
wrote:

> Hi everybody,
>
> This is the next dot release of Apache Hadoop 2.7 line. The previous one
> 2.7.5 was released on December 14, 2017.
> Release 2.7.6 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 04/16/2018.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

2018-04-17 Thread Konstantin Shvachko 
My formal +1 for 2.7.6 RC0

Thanks,
--Konstantin

On Mon, Apr 9, 2018 at 4:14 PM, Konstantin Shvachko 
wrote:

> Hi everybody,
>
> This is the next dot release of Apache Hadoop 2.7 line. The previous one
> 2.7.5 was released on December 14, 2017.
> Release 2.7.6 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5
> days ending 04/16/2018.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-04-17 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/

[Apr 16, 2018 5:15:37 PM] (xyao) HADOOP-15369. Avoid usage of 
${project.version} in parent poms.
[Apr 16, 2018 6:24:26 PM] (eyang) YARN-8060.  Added default readiness check for 
service components.   
[Apr 16, 2018 9:16:59 PM] (arp) HDFS-13439. Add test case for read block 
operation when it is moved.
[Apr 16, 2018 10:05:04 PM] (haibochen) YARN-8162. Remove Method 
DirectoryCollection#verifyDirUsingMkdir. (Yufei
[Apr 16, 2018 10:55:00 PM] (virajith) HDFS-13311. RBF: 
TestRouterAdminCLI#testCreateInvalidEntry fails on
[Apr 16, 2018 11:26:54 PM] (inigoiri) YARN-7773. YARN Federation used Mysql as 
state store throw exception,
[Apr 16, 2018 11:52:17 PM] (inigoiri) YARN-8165. Incorrect queue name logging 
in AbstractContainerAllocator.
[Apr 17, 2018 12:32:19 AM] (inigoiri) YARN-8164. Fix a potential NPE in 
AbstractSchedulerPlanFollower.




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.util.TestBasicDiskValidator 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.sls.TestReservationSystemInvariants 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/xml.txt
  [4.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [184K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [296K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [12K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/754/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7080) Default speculator won't sepculate the last several submitted reduced task if the total task num is large

2018-04-17 Thread Jason Lowe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-7080.
---
Resolution: Duplicate

Closing as a duplicate of MAPREDUCE-7081.

> Default speculator won't sepculate the last several submitted reduced task if 
> the total task num is large
> -
>
> Key: MAPREDUCE-7080
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7080
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Affects Versions: 2.7.5
>Reporter: Zhizhen Hou
>Priority: Major
>
> DefaultSpeculator speculates a task one time. 
> By default, the number of speculators is max(max(10, 0.01 * tasks.size), 0.1 
> * running tasks)
> I  set mapreduce.job.reduce.slowstart.completedmaps = 1 to start reduce after 
> all the map tasks are finished.
> The cluster has 1000 vcores, and the Job has 5000 reduce jobs.
> At first, 1000 reduces tasks can run simultaneously, number of speculators 
> can speculator at most is 0.1 * 1000 = 100 tasks. Reduce tasks with less data 
> can over shortly, and speculator will speculator a task per second by 
> default. The task be speculated execution may be because the more data to be 
> processed. It will speculator  100 tasks within 100 seconds.
> When 4900 reduces is over, If a reduce is executed with a lot of  data be 
> processed and is put on a slow machine. The speculate opportunity is running 
> out, it will not be speculated. It can increase the execution time of job 
> significantly.
> In short, it may waste the speculate opportunity at first only because the 
> execution time of  reduce with less data to be processed as average time. At  
> end of job, there is no speculate opportunity available, especially last 
> several running tasks, judged the number of the running tasks .
>  
> In my opinion, the number of tasks be speculated can be judged by square of 
> finished task percent. Take an example, if ninety percent of  the task is 
> finished, only 0.9*0.9 = 0.81 speculate opportunity can be used. It will 
> leave enough opportunity for latter tasks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7081) Default speculator won't sepculate the last several submitted reduced task if the total task num is large

2018-04-17 Thread Zhizhen Hou (JIRA) 
Zhizhen Hou created MAPREDUCE-7081:
--

 Summary: Default speculator won't sepculate the last several 
submitted reduced task if the total task num is large
 Key: MAPREDUCE-7081
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7081
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 2.7.5, 2.9.0
Reporter: Zhizhen Hou


DefaultSpeculator speculates a task one time.  By default, the number of 
speculators is max(max(10, 0.01 * tasks.size), 0.1 * running tasks).

I  set mapreduce.job.reduce.slowstart.completedmaps = 1 to start reduce after 
all the map tasks are finished. The cluster has 1000 vcores, and the Job has 
5000 reduce jobs. At first, 1000 reduces tasks can run simultaneously, number 
of speculators can speculator at most is 0.1 * 1000 = 100 tasks. Reduce tasks 
with less data can over shortly, and speculator will speculator a task per 
second by default. The task be speculated execution may be because the more 
data to be processed. It will speculator  100 tasks within 100 seconds. When 
4900 reduces is over, If a reduce is executed with a lot of  data be processed 
and is put on a slow machine. The speculate opportunity is running out, it will 
not be speculated. It can increase the execution time of job significantly.

In short, it may waste the speculate opportunity at first only because the 
execution time of  reduce with less data to be processed as average time. At  
end of job, there is no speculate opportunity available, especially last 
several running tasks, judged the number of the running tasks .  

In my opinion, the number of running tasks should not determine the number of 
speculate opportunity .The number of tasks be speculated can be judged by 
square of finished task percent. Take an example, if ninety percent of  the 
task is finished, only 0.9*0.9 = 0.81 speculate opportunity can be used. It 
will leave enough opportunity for latter tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org