[jira] [Created] (HDFS-13076) Merge work for HDFS-10285
Rakesh R created HDFS-13076: --- Summary: Merge work for HDFS-10285 Key: HDFS-13076 URL: https://issues.apache.org/jira/browse/HDFS-13076 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rakesh R This Jira is to run aggregated HDFS-10285 branch patch against trunk and check for any jenkins issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Hadoop 3.1.0 release discussion
Hi Uma, Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any other blockers under HDFS-10285. I think you guys should be able to start voting thread in time for merging to trunk. - Wangda On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma wrote: > Hi Wangda, > > > > Sorry for the delay. > > > > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers: > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two > JIRAs? > > We have only one blocker now that is HDFS-13050 and we finished the key > implementation from HDFS-12995, by HDFS-13075. > > We are planning to start vote by tomorrow (29th PST time). So, we request > you to give us time for running vote. We will keep SPS off by default. So, > interested users only can enable explicitly. > > > > Regards, > > Uma > > > > *From: *Wangda Tan > *Date: *Friday, January 26, 2018 at 5:21 PM > *To: *Uma Gangumalla > *Cc: *"hdfs-dev@hadoop.apache.org" , " > yarn-...@hadoop.apache.org" , " > common-...@hadoop.apache.org" , " > mapreduce-...@hadoop.apache.org" , Vinod > Kumar Vavilapalli > *Subject: *Re: Hadoop 3.1.0 release discussion > > > > Hi All, > > > > Just a reminder about feature freeze date. > > > > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from > today), If you've any features which live in a branch and targeted to > 3.1.0, please reply this email thread. Ideally, we should finish branch > merging before feature freeze date. > > > > Here's an updated 3.1.0 feature status: > > > > 1. Merged features: > > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler. > > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works > end-to-end. > > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services. > > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename > commits. > > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf > Queues While Doing Queue Mapping. > > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. > > * (Zhankun) YARN-5983: FPGA support. Majority implementations completed > and merged to trunk. Except for UI/documentation. > > > > 2. Features close to finish: > > * (Uma) HDFS-10285: HDFS SPS. There're two remaining > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of > the two JIRAs? > > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and > anti-affinity support. (Voting thread started). > > > > 3. Tentative features: > > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic > containers. Only one pending patch. Plan to finish before Jan 7th. > > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to > be done before Jan 2018. > > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks > challenging to be done before Jan 2018. > > * (Varun V) YARN-5673: container-executor write. Given security > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be > moved to 3.2. > > > > Thanks, > > Wangda > > > > > > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma > wrote: > > Sure, Wangda. > > Regards, > Uma > > > On 1/18/18, 10:19 AM, "Wangda Tan" wrote: > > Thanks Uma, > > Could you update this thread once the merge vote started? > > Best, > Wangda > > On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma < > uma.ganguma...@intel.com> > wrote: > > > HI Wangda, > > > > Thank you for the head-up mail. > > We are in the branch (HDFS-10285) and trying to push the tasks > sooner > > before the deadline. > > > > Regards, > > Uma > > > > On 1/17/18, 11:35 AM, "Wangda Tan" wrote: > > > > Hi All, > > > > Since we're fast approaching previously proposed feature freeze > date > > (Jan > > 30, about 13 days from today). If you've any features which live > in a > > branch and targeted to 3.1.0, please reply this email thread. > Ideally, > > we > > should finish branch merging before feature freeze date. > > > > Here's an updated 3.1.0 feature status: > > > > 1. Merged & Completed features: > > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler. > > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and > works > > end-to-end. > > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native > services. > > * (Steve Loughran): HADOOP-13786: S3Guard committer for > zero-rename > > commits. > > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation > of Leaf > > Queues While Doing Queue Mapping. > > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. > > > > 2. Features close to finish: > > * (Zhankun) YARN-5983: FPGA support. Majority implementations > > completed and > > merged to trunk. Except for UI/documentation. > > * (Uma) HDFS-10285: HDFS SPS. Majority
Re: Hadoop 3.1.0 release discussion
Hi Wangda, Sorry for the delay. >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers: >>HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two JIRAs? We have only one blocker now that is HDFS-13050 and we finished the key implementation from HDFS-12995, by HDFS-13075. We are planning to start vote by tomorrow (29th PST time). So, we request you to give us time for running vote. We will keep SPS off by default. So, interested users only can enable explicitly. Regards, Uma From: Wangda Tan Date: Friday, January 26, 2018 at 5:21 PM To: Uma Gangumalla Cc: "hdfs-dev@hadoop.apache.org" , "yarn-...@hadoop.apache.org" , "common-...@hadoop.apache.org" , "mapreduce-...@hadoop.apache.org" , Vinod Kumar Vavilapalli Subject: Re: Hadoop 3.1.0 release discussion Hi All, Just a reminder about feature freeze date. Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from today), If you've any features which live in a branch and targeted to 3.1.0, please reply this email thread. Ideally, we should finish branch merging before feature freeze date. Here's an updated 3.1.0 feature status: 1. Merged features: * (Sunil) YARN-5881: Support absolute value in CapacityScheduler. * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works end-to-end. * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services. * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits. * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping. * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. * (Zhankun) YARN-5983: FPGA support. Majority implementations completed and merged to trunk. Except for UI/documentation. 2. Features close to finish: * (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two JIRAs? * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and anti-affinity support. (Voting thread started). 3. Tentative features: * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic containers. Only one pending patch. Plan to finish before Jan 7th. * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be done before Jan 2018. * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks challenging to be done before Jan 2018. * (Varun V) YARN-5673: container-executor write. Given security refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2. Thanks, Wangda On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma mailto:uma.ganguma...@intel.com>> wrote: Sure, Wangda. Regards, Uma On 1/18/18, 10:19 AM, "Wangda Tan" mailto:wheele...@gmail.com>> wrote: Thanks Uma, Could you update this thread once the merge vote started? Best, Wangda On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma mailto:uma.ganguma...@intel.com>> wrote: > HI Wangda, > > Thank you for the head-up mail. > We are in the branch (HDFS-10285) and trying to push the tasks sooner > before the deadline. > > Regards, > Uma > > On 1/17/18, 11:35 AM, "Wangda Tan" mailto:wheele...@gmail.com>> wrote: > > Hi All, > > Since we're fast approaching previously proposed feature freeze date > (Jan > 30, about 13 days from today). If you've any features which live in a > branch and targeted to 3.1.0, please reply this email thread. Ideally, > we > should finish branch merging before feature freeze date. > > Here's an updated 3.1.0 feature status: > > 1. Merged & Completed features: > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler. > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works > end-to-end. > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services. > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename > commits. > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf > Queues While Doing Queue Mapping. > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. > > 2. Features close to finish: > * (Zhankun) YARN-5983: FPGA support. Majority implementations > completed and > merged to trunk. Except for UI/documentation. > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some > discussions going on about implementation. > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and > anti-affinity support. Close to finish, on track to be merged before > Jan 30. > > 3. Tentative features: > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic > containers. Only one pending patch. Plan to finish before Jan 7th. > * (Haibo Chen). YARN-1011: Resource overcom
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/671/ [Jan 28, 2018 6:18:51 AM] (jianhe) YARN-7765. Fixed an issue that kerberos tgt not found when NM posting - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/118/ [Jan 26, 2018 7:45:36 PM] (arp) HDFS-13054. Handling PathIsNotEmptyDirectoryException in DFSClient [Jan 26, 2018 9:46:01 PM] (jlowe) MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by -1 overall The following subsystems voted -1: asflicense unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Unreaped Processes : hadoop-hdfs:44 bkjournal:8 hadoop-yarn-server-resourcemanager:1 hadoop-yarn-client:4 hadoop-yarn-applications-distributedshell:1 hadoop-mapreduce-client-jobclient:3 hadoop-distcp:3 hadoop-extras:1 Failed junit tests : hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing hadoop.hdfs.server.namenode.ha.TestPipelinesFailover hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.server.namenode.TestQuotaByStorageType hadoop.hdfs.server.namenode.ha.TestHAMetrics hadoop.hdfs.server.blockmanagement.TestSlowDiskTracker hadoop.hdfs.server.namenode.TestFileLimit hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots hadoop.hdfs.server.namenode.TestFSImageWithAcl hadoop.hdfs.server.federation.router.TestRouterRpc hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd hadoop.hdfs.server.namenode.TestListOpenFiles hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup hadoop.hdfs.server.namenode.TestEditLogAutoroll hadoop.hdfs.server.namenode.TestStreamFile hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot hadoop.hdfs.server.namenode.TestDecommissioningStatus hadoop.hdfs.server.namenode.TestAuditLogger hadoop.hdfs.server.namenode.TestGenericJournalConf hadoop.hdfs.server.namenode.TestTransferFsImage hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.namenode.TestAclConfigFlag hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA hadoop.hdfs.server.namenode.TestSecondaryWebUi hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.datanode.TestDataNodeMetrics hadoop.hdfs.server.namenode.TestFSNamesystemMBean hadoop.hdfs.server.datanode.TestStorageReport hadoop.hdfs.server.namenode.TestCacheDirectives hadoop.hdfs.server.namenode.TestProtectedDirectories hadoop.hdfs.server.namenode.TestLargeDirectoryDelete hadoop.hdfs.server.namenode.TestXAttrConfigFlag hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication hadoop.hdfs.server.namenode.snapshot.TestSnapshot hadoop.hdfs.server.namenode.TestBackupNode hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination hadoop.hdfs.server.federation.router.TestRouterMountTable hadoop.hdfs.server.namenode.TestStartup hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestHAStateTransitions hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.hdfs.server.namenode.TestSaveNamespace hadoop.hdfs.server.namenode.TestNameNodeRpcServerMethods hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot hadoop.hdfs.server.federation.store.driver.TestStateStoreFileSystem hadoop.hdfs.server.namenode.TestDeadDatanode
