Re: Apache Hadoop 3.0.1 Release plan
I have reduced the priority for YARN-5742 as we have work around. Also, I have removed target version since we do not have plans to handle this in coming releases. -Rohith Sharma K S On 8 February 2018 at 22:30, Lei Xu wrote: > Hi, Brahma > > Thanks for reminder. YARN-5742 does not look like a blocker to me. I > will create a RC right after HADOOP-14060. > > On Thu, Feb 8, 2018 at 7:35 AM, Kihwal Lee wrote: > > HADOOP-14060 is a blocker. Daryn will add more detail to the jira or to > > this thread. > > > > On Thu, Feb 8, 2018 at 7:01 AM, Brahma Reddy Battula < > brbapa...@gmail.com> > > wrote: > >> > >> Hi Eddy, > >> > >> HDFS-12990 got committed to 3.0.1,can we have RC for 3.0.1 (only > >> YARN-5742 > >> blocker is open ) ? > >> > >> > >> On Sat, Feb 3, 2018 at 12:40 AM, Chris Douglas > >> wrote: > >> > >> > On Fri, Feb 2, 2018 at 10:22 AM, Arpit Agarwal > >> > > >> > wrote: > >> > > Do you plan to roll an RC with an uncommitted fix? That isn't the > >> > > right > >> > approach. > >> > > >> > The fix will be committed to the release branch. We'll vote on the > >> > release, and if it receives a majority of +1 votes then it becomes > >> > 3.0.1. That's how the PMC decides how to move forward. In this case, > >> > that will also resolve whether or not it can be committed to trunk. > >> > > >> > If this logic is unpersuasive, then we can require a 2/3 majority to > >> > replace the codebase. Either way, the PMC will vote to define the > >> > consensus view when it is not emergent. > >> > > >> > > This issue has good visibility and enough discussion. > >> > > >> > Yes, it has. We always prefer consensus to voting, but when discussion > >> > reveals that complete consensus is impossible, we still need a way > >> > forward. This is rare, and usually reserved for significant changes > >> > (like merging YARN). Frankly, it's embarrassing to resort to it here, > >> > but here we are. > >> > > >> > > If there is a binding veto in effect then the change must be > >> > > abandoned. > >> > Else you should be able to proceed with committing. However, 3.0.0 > must > >> > be > >> > called out as an abandoned release if we commit it. > >> > > >> > This is not accurate. A binding veto from any committer halts > >> > progress, but the PMC sets the direction of the project. That includes > >> > making decisions that are not universally accepted. -C > >> > > >> > > On 2/1/18, 3:01 PM, "Lei Xu" wrote: > >> > > > >> > > Sounds good to me, ATM. > >> > > > >> > > On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers > >> > wrote: > >> > > > Hey Anu, > >> > > > > >> > > > My feeling on HDFS-12990 is that we've discussed it quite a > bit > >> > already and > >> > > > it doesn't seem at this point like either side is going to > >> > > budge. > >> > I'm > >> > > > certainly happy to have a phone call about it, but I don't > >> > > expect > >> > that we'd > >> > > > make much progress. > >> > > > > >> > > > My suggestion is that we simply include the patch posted to > >> > HDFS-12990 in > >> > > > the 3.0.1 RC and call this issue out clearly in the subsequent > >> > VOTE thread > >> > > > for the 3.0.1 release. Eddy, are you up for that? > >> > > > > >> > > > Best, > >> > > > Aaron > >> > > > > >> > > > On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu > wrote: > >> > > >> > >> > > >> +Xiao > >> > > >> > >> > > >> My understanding is that we will have this for 3.0.1. Xiao, > >> > could > >> > > >> you give your inputs here? > >> > > >> > >> > > >> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer < > >> > aengin...@hortonworks.com> > >> > > >> wrote: > >> > > >> > Hi Eddy, > >> > > >> > > >> > > >> > Thanks for driving this release. Just a quick question, do > we > >> > have time > >> > > >> > to close this issue? > >> > > >> > https://issues.apache.org/jira/browse/HDFS-12990 > >> > > >> > > >> > > >> > or are we abandoning it? I believe that this is the last > >> > > window > >> > for us > >> > > >> > to fix this issue. > >> > > >> > > >> > > >> > Should we have a call and get this resolved one way or > >> > > another? > >> > > >> > > >> > > >> > Thanks > >> > > >> > Anu > >> > > >> > > >> > > >> > On 2/1/18, 10:51 AM, "Lei Xu" wrote: > >> > > >> > > >> > > >> > Hi, All > >> > > >> > > >> > > >> > I just cut branch-3.0.1 from branch-3.0. Please make > >> > > sure > >> > all > >> > > >> > patches > >> > > >> > targeted to 3.0.1 being checked in both branch-3.0 and > >> > branch-3.0.1. > >> > > >> > > >> > > >> > Thanks! > >> > > >> > Eddy > >> > > >> > > >> > > >> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu > >> > > > >> > wrote: > >> > > >> > > Hi, All > >> > > >> > > > >> > > >> > > We have released Apache Hadoop 3.0.0 in December [1]. > >> > > To > >> > further > >> > > >> > > impr
[jira] [Created] (HDFS-13129) Add a test for DfsAdmin refreshSuperUserGroupsConfiguration
Mukul Kumar Singh created HDFS-13129: Summary: Add a test for DfsAdmin refreshSuperUserGroupsConfiguration Key: HDFS-13129 URL: https://issues.apache.org/jira/browse/HDFS-13129 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh UserGroup can be refreshed using -refreshSuperUserGroupsConfiguration. This jira will add a test to verify that the user group information is updated correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/ [Feb 8, 2018 4:59:48 PM] (xyao) HDFS-13120. Snapshot diff could be corrupted after concat. Contributed [Feb 8, 2018 6:55:54 PM] (stevel) HADOOP-15214. Make Hadoop compatible with Guava 21.0. Contributed by [Feb 8, 2018 7:35:30 PM] (jianhe) YARN-5428. Allow for specifying the docker client configuration [Feb 8, 2018 8:32:43 PM] (yufei) YARN-7655. Avoid AM preemption caused by RRs for specific nodes or -1 overall The following subsystems voted -1: findbugs mvnsite unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources At Resource.java:by returning Resource.resources At Resource.java:[line 234] Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.hdfs.TestSafeMode hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 hadoop.hdfs.TestMaintenanceState hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 hadoop.hdfs.TestDFSStripedOutputStream hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime hadoop.yarn.client.api.impl.TestAMRMClientPlacementConstraints hadoop.mapreduce.v2.TestMRJobs hadoop.mapreduce.v2.TestUberAM cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/diff-compile-javac-root.txt [280K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/diff-checkstyle-root.txt [17M] mvnsite: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/patch-mvnsite-root.txt [112K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/diff-patch-shelldocs.txt [60K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/whitespace-eol.txt [9.2M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/whitespace-tabs.txt [292K] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/xml.txt [8.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [160K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [404K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [48K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [88K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/686/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt [8.0K] Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org - To unsubscr
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/131/ [Feb 8, 2018 1:03:11 PM] (brahma) HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when -1 overall The following subsystems voted -1: asflicense unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Unreaped Processes : hadoop-common:1 hadoop-hdfs:23 bkjournal:2 hadoop-yarn-server-timelineservice:1 hadoop-yarn-client:8 hadoop-yarn-server-timeline-pluginstorage:1 hadoop-yarn-applications-distributedshell:1 hadoop-mapreduce-client-app:1 hadoop-mapreduce-client-jobclient:16 hadoop-distcp:4 hadoop-extras:1 Failed junit tests : hadoop.hdfs.TestSetTimes hadoop.hdfs.TestSeekBug hadoop.hdfs.TestTrashWithEncryptionZones hadoop.hdfs.TestDFSClientRetries hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime hadoop.mapreduce.TestMRJobClient hadoop.mapred.TestReduceFetch hadoop.mapred.TestReduceFetchFromPartialMem hadoop.mapreduce.security.ssl.TestEncryptedShuffle hadoop.mapreduce.security.TestBinaryTokenFile hadoop.fs.TestFileSystem hadoop.mapred.TestFileInputFormatPathFilter hadoop.mapreduce.security.TestMRCredentials hadoop.conf.TestNoDefaultsJobConf hadoop.fs.TestDFSIO hadoop.mapred.TestJobSysDirWithDFS hadoop.tools.TestIntegration hadoop.tools.TestDistCpViewFs hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService Timed out junit tests : org.apache.hadoop.log.TestLogLevel org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.TestDatanodeRegistration org.apache.hadoop.hdfs.TestBlocksScheduledCounter org.apache.hadoop.hdfs.TestDFSClientFailover org.apache.hadoop.hdfs.web.TestWebHdfsTokens org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream org.apache.hadoop.hdfs.TestFileAppendRestart org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter org.apache.hadoop.hdfs.TestDFSOutputStream org.apache.hadoop.hdfs.TestDatanodeReport org.apache.hadoop.hdfs.web.TestWebHDFS org.apache.hadoop.hdfs.web.TestWebHDFSXAttr org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs org.apache.hadoop.hdfs.TestMiniDFSCluster org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.web.TestWebHDFSForHA org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.web.TestWebHDFSAcl org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir org.apache.hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices org.apache.hadoop.yarn.client.TestRMFailover org.apache.hadoop.yarn.client.cli.TestYarnCLI org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA org.apache.hadoop.yarn.client.api.impl.TestYarnClientWithReservation org.apache.hadoop.yarn.client.api.impl.TestYarnClient org.apache.hadoop.yarn.client.api.impl.TestAMRMClient org.apache.hadoop.yarn.client.api.impl.TestNMClient org.apache.hadoop.yarn.server.timeline.TestLogInfo org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMRCJCFileInputFormat org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.ipc.TestMRCJCSocketFactory org.apache.hadoop.mapred.TestMROpportunisticMaps org.apache.hadoop.mapred.TestMerge org.apache.hadoop.tools.TestDistCpWithAcls org.apache.hadoop.tools.TestDistCpSync org.apache.hadoop.tools.TestDistC
Re: [DISCUSS] Meetup for HDFS tests and build infra
On Thu, Feb 8, 2018 at 5:47 PM, 郑锴(铁杰) wrote: >>>I'm looking at you, TestDFSStripedOutputStreamWithFailure ... > AFAIK and IMO, it's pretty hard to get all the test cases stably running > given the limitation of MiniDFSCluster, and if we'd agree on that, we could > remove these cases as unit tests and cover them in integration tests instead > using a true cluster, like based on k8s infra. Since its inception, the Mini*Clusters have made it much easier to write unit tests (good!), but these tests take more time and resources than fine-grained tests (unfortunate) and they're often unreliable (bad). As we see in issues like HDFS-12711, the resource problem seems to harm reliability across the entire suite of tests (dire). Running integration tests with our CI is helpful, but perhaps we should separate the Mini*Cluster tests and/or press contributors to write lighter unit tests for new functionality. -C > We're lacking basic facility > infra env and tools to get most of the complicated functionalities well > tested and covered, so let's avoid too much complicated tests. Fixing of > such tests should definitely help and be appreciated. > > Regards, > Kai > > -- > 发件人：Chris Douglas > 发送时间：2018年2月8日(星期四) 08:39 > 收件人：Hdfs-dev > 主 题：Re: [DISCUSS] Meetup for HDFS tests and build infra > > Created a poll [1] to inform scheduling. -C > > [1]: https://doodle.com/poll/r22znitzae9apfbf > > On Tue, Feb 6, 2018 at 3:09 PM, Chris Douglas wrote: >> The HDFS build is not healthy. Many of the unit tests aren't actually >> run in Jenkins due to resource exhaustion, haven't been updated since >> build/test/data was the test temp dir, or are chronically unstable >> (I'm looking at you, TestDFSStripedOutputStreamWithFailure). The >> situation has deteriorated slowly, but we can't confidently merge >> patches, let alone significant features, when our CI infra is in this >> state. >> >> How would folks feel about a half to full-day meetup to work through >> patches improving this, specifically? We can improve tests, >> troubleshoot the build, and rev/commit existing patches. It would >> require some preparation, so the simultaneous attention is productive >> and not a coordination bottleneck. I started a wiki page for this [1], >> please add to it. >> >> If enough people can make time for this, say in 2-3 weeks, the project >> would certainly benefit. -C >> >> [1]: https://s.apache.org/ng3C > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
