[jira] [Created] (MAPREDUCE-7081) Default speculator won't sepculate the last several submitted reduced task if the total task num is large
Zhizhen Hou created MAPREDUCE-7081: -- Summary: Default speculator won't sepculate the last several submitted reduced task if the total task num is large Key: MAPREDUCE-7081 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7081 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.7.5, 2.9.0 Reporter: Zhizhen Hou DefaultSpeculator speculates a task one time. By default, the number of speculators is max(max(10, 0.01 * tasks.size), 0.1 * running tasks). I set mapreduce.job.reduce.slowstart.completedmaps = 1 to start reduce after all the map tasks are finished. The cluster has 1000 vcores, and the Job has 5000 reduce jobs. At first, 1000 reduces tasks can run simultaneously, number of speculators can speculator at most is 0.1 * 1000 = 100 tasks. Reduce tasks with less data can over shortly, and speculator will speculator a task per second by default. The task be speculated execution may be because the more data to be processed. It will speculator 100 tasks within 100 seconds. When 4900 reduces is over, If a reduce is executed with a lot of data be processed and is put on a slow machine. The speculate opportunity is running out, it will not be speculated. It can increase the execution time of job significantly. In short, it may waste the speculate opportunity at first only because the execution time of reduce with less data to be processed as average time. At end of job, there is no speculate opportunity available, especially last several running tasks, judged the number of the running tasks . In my opinion, the number of running tasks should not determine the number of speculate opportunity .The number of tasks be speculated can be judged by square of finished task percent. Take an example, if ninety percent of the task is finished, only 0.9*0.9 = 0.81 speculate opportunity can be used. It will leave enough opportunity for latter tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7080) Default speculator won't sepculate the last several submitted reduced task if the total task num is large
Zhizhen Hou created MAPREDUCE-7080: -- Summary: Default speculator won't sepculate the last several submitted reduced task if the total task num is large Key: MAPREDUCE-7080 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7080 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.7.5 Reporter: Zhizhen Hou DefaultSpeculator speculates a task one time. By default, the number of speculators is max(max(10, 0.01 * tasks.size), 0.1 * running tasks) I set mapreduce.job.reduce.slowstart.completedmaps = 1 to start reduce after all the map tasks are finished. The cluster has 1000 vcores, and the Job has 5000 reduce jobs. At first, 1000 reduces tasks can run simultaneously, number of speculators can speculator at most is 0.1 * 1000 = 100 tasks. Reduce tasks with less data can over shortly, and speculator will speculator a task per second by default. The task be speculated execution may be because the more data to be processed. It will speculator 100 tasks within 100 seconds. When 4900 reduces is over, If a reduce is executed with a lot of data be processed and is put on a slow machine. The speculate opportunity is running out, it will not be speculated. It can increase the execution time of job significantly. In short, it may waste the speculate opportunity at first only because the execution time of reduce with less data to be processed as average time. At end of job, there is no speculate opportunity available, especially last several running tasks, judged the number of the running tasks . In my opinion, the number of tasks be speculated can be judged by square of finished task percent. Take an example, if ninety percent of the task is finished, only 0.9*0.9 = 0.81 speculate opportunity can be used. It will leave enough opportunity for latter tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/439/ No changes -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 00m 00s) unit Specific tests: Failed junit tests : hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec hadoop.fs.contract.rawlocal.TestRawlocalContractAppend hadoop.fs.TestFsShellCopy hadoop.fs.TestFsShellList hadoop.fs.TestLocalFileSystem hadoop.fs.TestRawLocalFileSystemContract hadoop.fs.TestTrash hadoop.http.TestHttpServer hadoop.http.TestHttpServerLogs hadoop.io.nativeio.TestNativeIO hadoop.ipc.TestSocketFactory hadoop.metrics2.impl.TestStatsDMetrics hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal hadoop.security.TestSecurityUtil hadoop.security.TestShellBasedUnixGroupsMapping hadoop.security.token.TestDtUtilShell hadoop.util.TestNativeCodeLoader hadoop.util.TestNodeHealthScriptRunner hadoop.fs.TestResolveHdfsSymlink hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy hadoop.hdfs.crypto.TestHdfsCryptoStreams hadoop.hdfs.qjournal.client.TestQuorumJournalManager hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics hadoop.hdfs.server.datanode.TestDataNodeFaultInjector hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.datanode.TestHSync hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.mover.TestStorageMover hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport hadoop.hdfs.server.namenode.TestAddBlock hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands hadoop.hdfs.server.namenode.TestCheckpoint hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate hadoop.hdfs.server.namenode.TestEditLogRace hadoop.hdfs.server.namenode.TestFileTruncate hadoop.hdfs.server.namenode.TestFsck hadoop.hdfs.server.namenode.TestFSImage hadoop.hdfs.server.namenode.TestFSImageWithSnapshot hadoop.hdfs.server.namenode.TestNamenodeCapacityReport hadoop.hdfs.server.namenode.TestNameNodeMXBean hadoop.hdfs.server.namenode.TestNestedEncryptionZones hadoop.hdfs.server.namenode.TestQuotaByStorageType hadoop.hdfs.server.namenode.TestReconstructStripedBlocks hadoop.hdfs.server.namenode.TestReencryptionHandler hadoop.hdfs.server.namenode.TestStartup hadoop.hdfs.TestDatanodeRegistration hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs hadoop.hdfs.TestDecommission hadoop.hdfs.TestDFSOutputStream hadoop.hdfs.TestDFSShell hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
[VOTE] Release Apache Hadoop 3.0.2 (RC1)
Hi, All I've created release candidate RC-1 for Apache Hadoop 3.0.2, to address missing source jars in the maven repository in RC-0. Thanks Ajay Kumar for spotting the error. Please note: this is an amendment for Apache Hadoop 3.0.1 release to fix shaded jars in apache maven repository. The codebase of 3.0.2 release is the same as 3.0.1. New bug fixes will be included in Apache Hadoop 3.0.3 instead. The release page is: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC1/ The git tag is release-3.0.2-RC1, and the latest commit is 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee, which is the same as RC-0. The maven artifacts are available at: https://repository.apache.org/content/repositories/orgapachehadoop-1102/ Please try the release, especially, *verify the maven artifacts*, and vote. The vote will run 5 days, ending 4/21/2018. Here is my +1. Best, - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/ No changes -1 overall The following subsystems voted -1: asflicense unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.util.TestBasicDiskValidator hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.TestReencryptionWithKMS hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend hadoop.yarn.server.TestDiskFailures cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/diff-compile-javac-root.txt [288K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/xml.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [184K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [304K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [84K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/753/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)
Thanks for driving the release, Konstatin! +1 (binding) - Verified signatures and digests - Completed a native build from source - Deployed a single-node cluster - Ran some sample jobs Jason On Mon, Apr 9, 2018 at 6:14 PM, Konstantin Shvachko wrote: > Hi everybody, > > This is the next dot release of Apache Hadoop 2.7 line. The previous one 2.7.5 > was released on December 14, 2017. > Release 2.7.6 includes critical bug fixes and optimizations. See more > details in Release Note: > http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html > > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/ > > Please give it a try and vote on this thread. The vote will run for 5 days > ending 04/16/2018. > > My up to date public key is available from: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > Thanks, > --Konstantin - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
