Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/473/ No changes - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop Ozone 0.4.1-alpha
+1, Binding. Verified the KEYS Built from sources and ran tests: - General Ozone command line tests - Applications like MR and YARN. --Anu On Sat, Oct 12, 2019 at 10:25 AM Xiaoyu Yao wrote: > +1 binding. Verified > * Verify the signature. > * Build from source. > * Deploy docker compose in secure mode and verify ACL, sample MR jobs > > Thanks, > Xiaoyu > > On Fri, Oct 11, 2019 at 5:37 PM Hanisha Koneru > > wrote: > > > Thank you Nanda for putting up the RC. > > > > +1 binding. > > > > Verified the following: > > - Built from source > > - Deployed to 5 node cluster and ran smoke tests. > > - Ran sanity checks > > > > Thanks > > Hanisha > > > > > On Oct 4, 2019, at 10:42 AM, Nanda kumar wrote: > > > > > > Hi Folks, > > > > > > I have put together RC0 for Apache Hadoop Ozone 0.4.1-alpha. > > > > > > The artifacts are at: > > > https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/ > > > > > > The maven artifacts are staged at: > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1238/ > > > > > > The RC tag in git is at: > > > https://github.com/apache/hadoop/tree/ozone-0.4.1-alpha-RC0 > > > > > > And the public key used for signing the artifacts can be found at: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > This release contains 363 fixes/improvements [1]. > > > Thanks to everyone who put in the effort to make this happen. > > > > > > *The vote will run for 7 days, ending on October 11th at 11:59 pm IST.* > > > Note: This release is alpha quality, it’s not recommended to use in > > > production but we believe that it’s stable enough to try out the > feature > > > set and collect feedback. > > > > > > > > > [1] https://s.apache.org/yfudc > > > > > > Thanks, > > > Team Ozone > > > > >
Re: [VOTE] Release Apache Hadoop Ozone 0.4.1-alpha
+1 binding. Verified * Verify the signature. * Build from source. * Deploy docker compose in secure mode and verify ACL, sample MR jobs Thanks, Xiaoyu On Fri, Oct 11, 2019 at 5:37 PM Hanisha Koneru wrote: > Thank you Nanda for putting up the RC. > > +1 binding. > > Verified the following: > - Built from source > - Deployed to 5 node cluster and ran smoke tests. > - Ran sanity checks > > Thanks > Hanisha > > > On Oct 4, 2019, at 10:42 AM, Nanda kumar wrote: > > > > Hi Folks, > > > > I have put together RC0 for Apache Hadoop Ozone 0.4.1-alpha. > > > > The artifacts are at: > > https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/ > > > > The maven artifacts are staged at: > > https://repository.apache.org/content/repositories/orgapachehadoop-1238/ > > > > The RC tag in git is at: > > https://github.com/apache/hadoop/tree/ozone-0.4.1-alpha-RC0 > > > > And the public key used for signing the artifacts can be found at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > This release contains 363 fixes/improvements [1]. > > Thanks to everyone who put in the effort to make this happen. > > > > *The vote will run for 7 days, ending on October 11th at 11:59 pm IST.* > > Note: This release is alpha quality, it’s not recommended to use in > > production but we believe that it’s stable enough to try out the feature > > set and collect feedback. > > > > > > [1] https://s.apache.org/yfudc > > > > Thanks, > > Team Ozone > >
Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree [discussion -> lazy vote]
On Mon, Sep 23, 2019 at 4:02 PM Elek, Marton wrote: > As the overall feedback was positive (in fact many of the answers were > simple +1 votes) I don't think the thread should be repeated under > [VOTE] subject. Therefore I call it for a lazy consensus. > Let's please not do this in the future. These are large enough changes that we'd like to get formal votes both for immediate visibility (VOTE threads are more in your face) as well as for record-keeping in posterity. Thanks +Vinod
Re: VOTE PASSED - Hadoop-3.1.3-RC0 Re: [VOTE] Release Hadoop-3.1.3-RC0
Zhankun et.al, Did we close down the rest of the release process for 3.1.3? I don't yet see this on the site, so asking. If it's not done yet and you are running into issues, please let me know and I can help with the corresponding release management tasks. Thanks +Vinod On Tue, Sep 24, 2019 at 2:16 PM Zhankun Tang wrote: > Hi, > > Thank you all who spent time to verify and vote 3.1.3 release! > I‘m pleased to announce that the vote for 3.1.3 RC0 passed! I'll push the > release bits and send out an announcement for 3.1.3 soon. > > Summary of votes for Hadoop-3.1.3-RC0: > > 4 binding +1s, from Rohith Sharma K S, Sunil Govindan, Weiwei Yang, Eric > Payne > > 3 non-binding +1s, from Xun Liu, Zac Zhou, Zhankun Tang > > 1 non-binding with +0s from Masatake Iwasaki > > and *no -1s*. > > BR, > Zhankun > > On Tue, 24 Sep 2019 at 15:40, zac yuan wrote: > > > Looking forward to 3.1.3 release. Thanks Zhankun for your great effort~ > > > > +1 (non-binding) > > > > Zac Zhou > > > > Xun Liu 于2019年9月23日周一 上午11:49写道： > > > > > +1 > > > > > > Thanks Zhankun for all of your hard work on this release. > > > > > > > > > > On Sep 20, 2019, at 5:34 AM, epa...@apache.org wrote: > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > Thanks Zhankun for all of your hard work on this release. > > > > > > > > I downloaded and built the source and ran it on an insecure > multi-node > > > pseudo cluster. > > > > > > > > I performed various YARN manual tests, including creating custom > > > resources, creating queue submission ACLs, and queue refreshes. > > > > > > > > One concern is that preemption does not seem to be working when only > > the > > > custom resources are over the queue capacity, but I don't think this is > > > something introduced with this release. > > > > > > > > -Eric > > > > > > > > > > > > > > > > On Thursday, September 12, 2019, 3:04:44 AM CDT, Zhankun Tang < > > > zt...@apache.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi folks, > > > > > > > > Thanks to everyone's help on this release. Special thanks to Rohith, > > > > Wei-Chiu, Akira, Sunil, Wangda! > > > > > > > > I have created a release candidate (RC0) for Apache Hadoop 3.1.3. > > > > > > > > The RC release artifacts are available at: > > > > http://home.apache.org/~ztang/hadoop-3.1.3-RC0/ > > > > > > > > The maven artifacts are staged at: > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1228/ > > > > > > > > The RC tag in git is here: > > > > https://github.com/apache/hadoop/tree/release-3.1.3-RC0 > > > > > > > > And my public key is at: > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.* > > > > > > > > For the testing, I have run several Spark and distributed shell jobs > in > > > my > > > > pseudo cluster. > > > > > > > > My +1 (non-binding) to start. > > > > > > > > BR, > > > > Zhankun > > > > > > > > On Wed, 4 Sep 2019 at 15:56, zhankun tang > > wrote: > > > > > > > >> Hi all, > > > >> > > > >> Thanks for everyone helping in resolving all the blockers targeting > > > Hadoop > > > >> 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers > > > issues > > > >> to 3.1.4. > > > >> > > > >> I'll cut the branch today and call a release vote soon. Thanks! > > > >> > > > >> > > > >> [1]. https://s.apache.org/5hj5i > > > >> > > > >> BR, > > > >> Zhankun > > > >> > > > >> > > > >> On Wed, 21 Aug 2019 at 12:38, Zhankun Tang > wrote: > > > >> > > > >>> Hi folks, > > > >>> > > > >>> We have Apache Hadoop 3.1.2 released on Feb 2019. > > > >>> > > > >>> It's been more than 6 months passed and there're > > > >>> > > > >>> 246 fixes[1]. 2 blocker and 4 critical Issues [2] > > > >>> > > > >>> (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker) > > > >>> > > > >>> > > > >>> I propose my plan to do a maintenance release of 3.1.3 in the next > > few > > > >>> (one or two) weeks. > > > >>> > > > >>> Hadoop 3.1.3 release plan: > > > >>> > > > >>> Code Freezing Date: *25th August 2019 PDT* > > > >>> > > > >>> Release Date: *31th August 2019 PDT* > > > >>> > > > >>> > > > >>> Please feel free to share your insights on this. Thanks! > > > >>> > > > >>> > > > >>> [1] https://s.apache.org/zw8l5 > > > >>> > > > >>> [2] https://s.apache.org/fjol5 > > > >>> > > > >>> > > > >>> BR, > > > >>> > > > >>> Zhankun > > > >>> > > > >> > > > > > > > > - > > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > > - > > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > > > > > > > >
[jira] [Created] (YARN-9896) allocDelay in FSPreemptionThread should not be 0
Xianghao Lu created YARN-9896: - Summary: allocDelay in FSPreemptionThread should not be 0 Key: YARN-9896 URL: https://issues.apache.org/jira/browse/YARN-9896 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.9.1 Environment: [allocDelay|[https://github.com/apache/hadoop/blob/e30710aea4e6e55e69372929106cf119af06fd0e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSPreemptionThread.java#L63]] in FSPreemptionThread.java will be 0, because [nmHeartbeatInterval|https://github.com/apache/hadoop/blob/e30710aea4e6e55e69372929106cf119af06fd0e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java#L147] in AbstractYarnScheduler.java has not been initialized. Initialization order [here|https://github.com/apache/hadoop/blob/e30710aea4e6e55e69372929106cf119af06fd0e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java#L1379] make this bug happen. Reporter: Xianghao Lu Assignee: Xianghao Lu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [DISCUSS] ARM/aarch64 support for Hadoop
Hi All, Thanks Vinaya bring the whole thing up and everyone else for discussing, providing thoughts and doing actuall coding. I noticed that protobuf upgrading has been the hottest discussion point about the whole thread and Vinayakumar, Duo Zhang and Akira Ajisaka have done alot about protocbuf and the progress seems well: https://issues.apache.org/jira/browse/HADOOP-13363 During the same time, we have keep trying to build and test hadoop components, we have been debugging for all the problems occurred, and providing possible solutions like: https://issues.apache.org/jira/browse/HADOOP-16614 we would be really apappreciated if some one could check on the probosed solution and provide feedbacks and thoughts. Currently, we mainly focused on YARN module and we have finishing testing and debugging, we can now passing all tests with fixes, we also found root causes and possible sulotions to some already known problems like: https://issues.apache.org/jira/browse/YARN-9511 I think on some level this could prove that we are willing to maintain and improve the stuff in long term. *Depending on this, I would like to propose adding an ARM CI to Hadoop community, we can first start as a periodic job, or maybe for only one component(YARN for example). We will donate machines to the current CI system and we will maintain it and keep solve problems. By adding a CI, it will also allow others that are interested in this to help review and debug problems occoured. We are also willing to provide machines for contibutors that are intrested or want to debuging for ARM related issues.* And another thing I want to add on is, I can see from the thread and previous discussions in ML and Jira, not much people seems interested in things like make Hadoop working on Aarch64 platform. Here I want to provide some ideas that might make this more attractive. When speaking of ARM or Aarch64, devices like raspberry pi may be the first thing that pop up to our head, we might think 'oh, it could be running on raspberry pi, interesting' and treated it as a toy or test and then we go back to our normal life. But actually, we can do much more. As we all know, over 90 percent of the mobile device uses ARM architecture, and there are more and more ARM base datacenter servers are begaining to show up at the market. With the increasing computing capability of mobile devices and increasing speed of wireless connection like 5G, we could imagine what kind of possiblity could we have if we can have Hadoop running on both datacenter and mobile devices at edge. I'm not a Hadoop expert but I can imagine that there is a ton of possiblilities and opportunities for new bussiness models using Hadoop in the future. Probably some of them will change our life. Just my 2 cents. I have also updated in the issue that I've originally submitted, please feel free to leave comments: https://issues.apache.org/jira/browse/HADOOP-16358 BR, On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B wrote: > Thanks @Anu > > I understand the concern. I took it in different manner. > > Anyway, since protobuf upgrade looks huge, and need everyone's eyes on > changes as early as possible, its better to do it trunk itself. > > I was able to come to successfull attempt of upgrading protobuf as per > suggestion of stack in Jira > https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253 > . > > Have created the PR. Please review. Changes looks huge, because all > references of "com.google.protobuf" relocated to > "o.a.h.shaded.com.google.protobuf". > Otherwise changes are reasonable. > > This change is with still keeping the current 2.5.0 dependency, for > downstream builds. So essentially nothing should be changed for downstreams. > > -Vinay > > > On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer > wrote: > >> Yes, I think that is what Sunil and I are trying to suggest; the complex >> dependencies like Protobuf, if you do it in the trunk you have a better >> change of getting it done. Otherwise, at merge point random downstream >> applications which you have never heard of will object, and Hadoop >> compatibility rules are very clear so you cannot fix it. >> >> With that said, even doing this in the trunk is complex; It might be >> good for you to host a meeting and get some feedback. I have openly said it >> is a great idea like "belling the cat", but the effort is in getting the >> community to agree and align. Solve that, most of your technical problems >> will be easier to solve. >> >> If you go into a branch, it might be that the community might forget >> about your work; and when you come in to merge you will see issues which >> you did not think about. >> >> So, Here is what would be great if you can make this happen; for ARM >> work, get a list of dependencies that needed to be upgraded; see if you can >> get the community aligned with this goal; since ARM might not be in
