Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/530/ [Jul 16, 2018 4:24:21 PM] (ericp) YARN-8421: when moving app, activeUsers is increased, even though app [Jul 16, 2018 4:46:21 PM] (inigoiri) HDFS-13475. RBF: Admin cannot enforce Router enter SafeMode. Contributed [Jul 16, 2018 5:51:23 PM] (weichiu) HDFS-13524. Occasional "All datanodes are bad" error in [Jul 16, 2018 5:54:41 PM] (wangda) YARN-8511. When AM releases a container, RM removes allocation tags [Jul 16, 2018 5:57:37 PM] (wangda) YARN-8361. Change App Name Placement Rule to use App Name instead of App [Jul 16, 2018 5:58:00 PM] (wangda) YARN-8524. Single parameter Resource / LightWeightResource constructor [Jul 16, 2018 6:32:45 PM] (weichiu) HADOOP-15598. DataChecksum calculate checksum is contented on hashtable [Jul 16, 2018 8:19:53 PM] (xiao) HDFS-13690. Improve error message when creating encryption zone while [Jul 16, 2018 9:38:49 PM] (eyang) YARN-8538. Fixed memory leaks in container-executor and test cases. [Jul 16, 2018 9:41:23 PM] (eyang) YARN-8299. Added CLI and REST API for query container status. [Jul 16, 2018 10:45:55 PM] (weichiu) HDFS-13485. DataNode WebHDFS endpoint throws NPE. Contributed by Siyao [Jul 17, 2018 1:24:18 AM] (shv) Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam [Jul 17, 2018 2:45:08 AM] (yqlin) HDFS-13733. RBF: Add Web UI configurations and descriptions to RBF -1 overall The following subsystems voted -1: compile mvninstall pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 00m 00s) unit Specific tests: Failed junit tests : hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec hadoop.fs.contract.rawlocal.TestRawlocalContractAppend hadoop.fs.TestFileUtil hadoop.fs.TestFsShellCopy hadoop.fs.TestFsShellList hadoop.http.TestHttpServer hadoop.http.TestHttpServerLogs hadoop.io.nativeio.TestNativeIO hadoop.ipc.TestSocketFactory hadoop.metrics2.impl.TestStatsDMetrics hadoop.security.TestSecurityUtil hadoop.security.TestShellBasedUnixGroupsMapping hadoop.security.token.TestDtUtilShell hadoop.util.TestDiskCheckerWithDiskIo hadoop.util.TestNativeCodeLoader hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestDataNodeFaultInjector hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs hadoop.hdfs.TestDFSShell hadoop.hdfs.TestDFSStripedInputStream hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy hadoop.hdfs.TestDFSUpgradeFromImage hadoop.hdfs.TestFetchImage hadoop.hdfs.TestFileConcurrentReader hadoop.hdfs.TestHDFSFileSystemContract hadoop.hdfs.TestPread hadoop.hdfs.TestSecureEncryptionZoneWithKMS hadoop.hdfs.TestTrashWithSecureEncryptionZones hadoop.hdfs.tools.TestDFSAdmin hadoop.hdfs.web.TestWebHDFS hadoop.hdfs.web.TestWebHdfsUrl hadoop.fs.http.server.TestHttpFSServerWebServer hadoop.yarn.logaggregation.filecontroller.ifile.TestLogAggregationIndexFileController hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch hadoop.yarn.server.nodemanager.containermanager.TestAuxServices hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.nodemanager.r
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/840/ [Jul 17, 2018 2:45:08 AM] (yqlin) HDFS-13733. RBF: Add Web UI configurations and descriptions to RBF -1 overall The following subsystems voted -1: docker Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: Hadoop 3.2 Release Plan proposal
On 16 Jul 2018, at 23:45, Sunil G mailto:sun...@apache.org>> wrote: I would also would like to take this opportunity to come up with a detailed plan. - Feature freeze date : all features should be merged by August 10, 2018. That's three weeks from now. While I appreciate a faster cadence, saying "you now have three weeks to get everything in" is a bit sudden. That's pretty aggressive given that many patches can languish unreviewed for months at a time. Also: it's the summer holidays. - Code freeze date : blockers/critical only, no improvements and non blocker/critical bug-fixes August 24, 2018. If the feature freeze is 3 weeks from now, I reserve the right to add new bits of code to the hadoop-aws and hadoop-azure modules as I see fit, up until this cutoff. - Release date: August 31, 2018 At least one RC will have to be built, put out and voted on. I think it'd be good to have that RC generation process nailed down before Aug 24, but you have to include the vote *and the possibility of it failing* into the schedule. Please let me know if I missed any features targeted to 3.2 per this Well there these big todo lists for S3 & S3Guard. https://issues.apache.org/jira/browse/HADOOP-15226 https://issues.apache.org/jira/browse/HADOOP-15220 There's a bigger bit of work coming on for Azure Datalake Gen 2 https://issues.apache.org/jira/browse/HADOOP-15407 I don't think this is quite ready yet, I've been doing work on it, but if we have a 3 week deadline, I'm going to expect some timely reviews on https://issues.apache.org/jira/browse/HADOOP-15546 I've uprated that to a blocker feature; will review the S3 & S3Guard JIRAs to see which of those are blocking. Then there are some pressing "guave, java 9 prep" timeline. I would like to volunteer myself as release manager of 3.2.0 release. well volunteered! Please let me know if you have any suggestions. I think this raises a good q: what timetable should we have for the 3.2. & 3.3 releases; if we do want a faster cadence, then having the outline time from the 3.2 to the 3.3 release means that there's less concern about things not making the 3.2 dealine -Steve
[jira] [Resolved] (MAPREDUCE-6948) TestJobImpl.testUnusableNodeTransition failed
[
https://issues.apache.org/jira/browse/MAPREDUCE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved MAPREDUCE-6948.
---
Resolution: Cannot Reproduce
I agree as well. I have not seen any recent precommit failures on 3.x releases
for this unit test.
> TestJobImpl.testUnusableNodeTransition failed
> -
>
> Key: MAPREDUCE-6948
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6948
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Haibo Chen
>Assignee: Jim Brennan
>Priority: Major
> Labels: unit-test
>
> *Error Message*
> expected: but was:
> *Stacktrace*
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.assertJobState(TestJobImpl.java:1041)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testUnusableNodeTransition(TestJobImpl.java:615)
> *Standard out*
> {code}
> 2017-08-30 10:12:21,928 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
> 2017-08-30 10:12:21,939 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl$StubbedJob
> 2017-08-30 10:12:21,940 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,940 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.jobhistory.EventType for class
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,940 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,941 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:setup(1534)) - Adding job token for job_123456789_0001 to
> jobTokenSecretManager
> 2017-08-30 10:12:21,941 WARN [Thread-49] impl.JobImpl
> (JobImpl.java:setup(1540)) - Shuffle secret key missing from job credentials.
> Using job token secret as shuffle secret.
> 2017-08-30 10:12:21,944 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:makeUberDecision(1305)) - Not uberizing job_123456789_0001
> because: not enabled;
> 2017-08-30 10:12:21,944 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:createMapTasks(1562)) - Input size for job
> job_123456789_0001 = 0. Number of splits = 2
> 2017-08-30 10:12:21,945 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:createReduceTasks(1579)) - Number of reduces for job
> job_123456789_0001 = 1
> 2017-08-30 10:12:21,945 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_123456789_0001Job Transitioned from NEW
> to INITED
> 2017-08-30 10:12:21,946 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_123456789_0001Job Transitioned from
> INITED to SETUP
> 2017-08-30 10:12:21,954 INFO [CommitterEvent Processor #0]
> commit.CommitterEventHandler (CommitterEventHandler.java:run(231)) -
> Processing the event EventType: JOB_SETUP
> 2017-08-30 10:12:21,978 INFO [AsyncDispatcher event handler] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_123456789_0001Job Transitioned from
> SETUP to RUNNING
> 2017-08-30 10:12:21,983 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl$5
> 2017-08-30 10:12:22,000 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:transition(1953)) - Num completed Tasks: 1
> 2017-08-30 10:12:22,029 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:transition(1953)) - Num completed Tasks: 2
> 2017-08-30 10:12:22,032 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:actOnUnusableNode(1354)) - TaskAttempt killed because it ran on
> unusable node Mock for NodeId, hashCode: 1280187896.
> AttemptId:attempt_123456789_0001_m_00_0
> 2017-08-30 10:12:22,03
Re: Hadoop 3.2 Release Plan proposal
Hi Sunil, Thanks for bringing up this proposal. For feature release, should we put this feature in? YARN-8379 Preempt to balance in already satisfied queues Thanks Zian > On Jul 16, 2018, at 11:45 PM, Sunil G wrote: > > Hi All, > > > To continue a faster cadence of releases to accommodate more features, > > we could plan a Hadoop 3.2 release around August end. > > > To start the process sooner, and to establish a timeline, I propose > > to target Hadoop 3.2.0 release by August end 2018. (About 1.5 months from > now). > > > I would also would like to take this opportunity to come up with a detailed > plan. > > - Feature freeze date : all features should be merged by August 10, 2018. > > - Code freeze date : blockers/critical only, no improvements and non > blocker/critical > > bug-fixes August 24, 2018. > > - Release date: August 31, 2018 > > > I have tried to come up with a list of features on my radar which could be > candidates > > for a 3.2 release: > > - YARN-3409, Node Attributes support. (Owner: Naganarasimha/Sunil) > > - YARN-8135, Hadoop Submarine project for DeepLearning workloads in YARN > (Owner: Wangda Tan) > > - YARN Native Service / Docker feature hardening and stabilization works in > YARN > > > > There are several other HDFS features want to be released with 3.2 as well, > I am quoting few here: > > - HDFS-10285 Storage Policy Satisfier (Owner: Uma/Rakesh) > > - Improvements to HDFS-12615 Router-based HDFS federation > > > > Please let me know if I missed any features targeted to 3.2 per this > > timeline. I would like to volunteer myself as release manager of 3.2.0 > release. > > > Please let me know if you have any suggestions. > > > > Thanks, > > Sunil Govindan - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
