Re: [VOTE] Moving Ozone to a separated Apache project

[Weiwei Yang]

+1

Weiwei

On Friday, September 25, 2020, Dinesh Chitlangia  wrote:

> +1
>
> Thanks,
> Dinesh
>
> On Fri, Sep 25, 2020 at 2:00 AM Elek, Marton  wrote:
>
> > Hi all,
> >
> > Thank you for all the feedback and requests,
> >
> > As we discussed in the previous thread(s) [1], Ozone is proposed to be a
> > separated Apache Top Level Project (TLP)
> >
> > The proposal with all the details, motivation and history is here:
> >
> >
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Ozone+Hadoop+subproject+to+Apache+TLP+proposal
> >
> > This voting runs for 7 days and will be concluded at 2nd of October, 6AM
> > GMT.
> >
> > Thanks,
> > Marton Elek
> >
> > [1]:
> >
> > https://lists.apache.org/thread.html/rc6c79463330b3e993e24a564c6817
> aca1d290f186a1206c43ff0436a%40%3Chdfs-dev.hadoop.apache.org%3E
> >
> > -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >
> >
>

Re: [ANNOUNCE] New Apache Hadoop Committer - He Xiaoqiao

[Weiwei Yang]

Congratulations Xiaoqiao!

On Thu, Jun 11, 2020 at 11:20 AM Sree Vaddi 
wrote:

> Congratulations, He Xiaoqiao.
>
> Thank you./Sree
>
>
>
> On Thursday, June 11, 2020, 9:54:32 AM PDT, Chao Sun <
> sunc...@apache.org> wrote:
>
>  Congrats Xiaoqiao!
>
> On Thu, Jun 11, 2020 at 9:36 AM Ayush Saxena  wrote:
>
> > Congratulations He Xiaoqiao!!!
> >
> > -Ayush
> >
> > > On 11-Jun-2020, at 9:30 PM, Wei-Chiu Chuang 
> wrote:
> > >
> > > In bcc: general@
> > >
> > > It's my pleasure to announce that He Xiaoqiao has been elected as a
> > > committer on the Apache Hadoop project recognizing his continued
> > > contributions to the
> > > project.
> > >
> > > Please join me in congratulating him.
> > >
> > > Hearty Congratulations & Welcome aboard Xiaoqiao!
> > >
> > > Wei-Chiu Chuang
> > > (On behalf of the Hadoop PMC)
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >

Re: [DISCUSS] making Ozone a separate Apache project

[Weiwei Yang]

+1
Thanks, Marton for starting the discussion.
One question, for the committers who contributed to Ozone before and got
the committer-role in the past (like me), will they carry the
committer-role to the new repo?

On Wed, May 13, 2020 at 9:07 AM Dinesh Chitlangia 
wrote:

> +1
> Thank you Marton for writing up the history and supporting comments.
>
> -Dinesh
>
> On Wed, May 13, 2020 at 3:53 AM Elek, Marton  wrote:
>
> >
> >
> > I would like to start a discussion to make a separate Apache project for
> > Ozone
> >
> >
> >
> > ### HISTORY [1]
> >
> >   * Apache Hadoop Ozone development started on a feature branch of
> > Hadoop repository (HDFS-7240)
> >
> >   * In the October of 2017 a discussion has been started to merge it to
> > the Hadoop main branch
> >
> >   * After a long discussion it's merged to Hadoop trunk at the March of
> > 2018
> >
> >   * During the discussion of the merge, it was suggested multiple times
> > to create a separated project for the Ozone. But at that time:
> >  1). Ozone was tightly integrated with Hadoop/HDFS
> >  2). There was an active plan to use Block layer of Ozone (HDDS or
> > HDSL at that time) as the block level of HDFS
> >  3). The community of Ozone was a subset of the HDFS community
> >
> >   * The first beta release of Ozone was just released. Seems to be a
> > good time before the first GA to make a decision about the future.
> >
> >
> >
> > ### WHAT HAS BEEN CHANGED
> >
> >   During the last years Ozone became more and more independent both at
> > the community and code side. The separation has been suggested again and
> > again (for example by Owen [2] and Vinod [3])
> >
> >
> >
> >   From COMMUNITY point of view:
> >
> >
> >* Fortunately more and more new contributors are helping Ozone.
> > Originally the Ozone community was a subset of HDFS project. But now a
> > bigger and bigger part of the community is related to Ozone only.
> >
> >* It seems to be easier to _build_ the community as a separated
> project.
> >
> >* A new, younger project might have different practices
> > (communication, commiter criteria, development style) compared to old,
> > mature project
> >
> >* It's easier to communicate (and improve) these standards in a
> > separated projects with clean boundaries
> >
> >* Separated project/brand can help to increase the adoption rate and
> > attract more individual contributor (AFAIK it has been seen in Submarine
> > after a similar move)
> >
> >   * Contribution process can be communicated more easily, we can make
> > first time contribution more easy
> >
> >
> >
> >   From CODE point of view Ozone became more and more independent:
> >
> >
> >   * Ozone has different release cycle
> >
> >   * Code is already separated from Hadoop code base
> > (apache/hadoop-ozone.git)
> >
> >   * It has separated CI (github actions)
> >
> >   * Ozone uses different (more strict) coding style (zero toleration of
> > unit test / checkstyle errors)
> >
> >   * The code itself became more and more independent from Hadoop on
> > Maven level. Originally it was compiled together with the in-tree latest
> > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > Configuration...)
> >
> >   * It starts to use multiple version of Hadoop (on client side)
> >
> >   * Volume of resolved issues are already very high on Ozone side (Ozone
> > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > together in the last 2-3 months)
> >
> >
> > Summary: Before the first Ozone GA release, It seems to be a good time
> > to discuss the long-term future of Ozone. Managing it as a separated TLP
> > project seems to have more benefits.
> >
> >
> > Please let me know what your opinion is...
> >
> > Thanks a lot,
> > Marton
> >
> >
> >
> >
> >
> > [1]: For more details, see:
> > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> >
> > [2]:
> >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> >
> > [3]:
> >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] Release Hadoop-3.1.3-RC0

[Weiwei Yang]

+1 (binding)

- Downloaded source, setup a single node cluster
- Verified basic HDFS operations, put/get/cat etc
- Verified basic YARN restful apis, cluster/nodes/scheduler, all seem good
- Run several distributed shell job

Thanks
Weiwei
On Sep 19, 2019, 4:28 PM +0800, Sunil Govindan , wrote:
> +1 (binding)
>
> Thanks Zhankun for putting up the release. Thanks for leading this.
>
> - verified signature
> - ran a local cluster from tar ball
> - ran some MR jobs
> - perform CLI ops, and looks good
> - UI seems fine
>
> Thanks
> Sunil
>
> On Thu, Sep 12, 2019 at 1:34 PM Zhankun Tang  wrote:
>
> > Hi folks,
> >
> > Thanks to everyone's help on this release. Special thanks to Rohith,
> > Wei-Chiu, Akira, Sunil, Wangda!
> >
> > I have created a release candidate (RC0) for Apache Hadoop 3.1.3.
> >
> > The RC release artifacts are available at:
> > http://home.apache.org/~ztang/hadoop-3.1.3-RC0/
> >
> > The maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1228/
> >
> > The RC tag in git is here:
> > https://github.com/apache/hadoop/tree/release-3.1.3-RC0
> >
> > And my public key is at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.*
> >
> > For the testing, I have run several Spark and distributed shell jobs in my
> > pseudo cluster.
> >
> > My +1 (non-binding) to start.
> >
> > BR,
> > Zhankun
> >
> > On Wed, 4 Sep 2019 at 15:56, zhankun tang  wrote:
> >
> > > Hi all,
> > >
> > > Thanks for everyone helping in resolving all the blockers targeting
> > Hadoop
> > > 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers
> > issues
> > > to 3.1.4.
> > >
> > > I'll cut the branch today and call a release vote soon. Thanks!
> > >
> > >
> > > [1]. https://s.apache.org/5hj5i
> > >
> > > BR,
> > > Zhankun
> > >
> > >
> > > On Wed, 21 Aug 2019 at 12:38, Zhankun Tang  wrote:
> > >
> > > > Hi folks,
> > > >
> > > > We have Apache Hadoop 3.1.2 released on Feb 2019.
> > > >
> > > > It's been more than 6 months passed and there're
> > > >
> > > > 246 fixes[1]. 2 blocker and 4 critical Issues [2]
> > > >
> > > > (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker)
> > > >
> > > >
> > > > I propose my plan to do a maintenance release of 3.1.3 in the next few
> > > > (one or two) weeks.
> > > >
> > > > Hadoop 3.1.3 release plan:
> > > >
> > > > Code Freezing Date: *25th August 2019 PDT*
> > > >
> > > > Release Date: *31th August 2019 PDT*
> > > >
> > > >
> > > > Please feel free to share your insights on this. Thanks!
> > > >
> > > >
> > > > [1] https://s.apache.org/zw8l5
> > > >
> > > > [2] https://s.apache.org/fjol5
> > > >
> > > >
> > > > BR,
> > > >
> > > > Zhankun
> > > >
> > >
> >

Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0

[Weiwei Yang]

+1 (binding)

Downloaded tarball, setup a pseudo cluster manually
Verified basic HDFS operations, copy/view files
Verified basic YARN operations, run sample DS jobs
Verified basic YARN restful APIs, e.g cluster/nodes info etc
Set and verified YARN node-attributes, including CLI

Thanks
Weiwei
On Sep 18, 2019, 11:41 AM +0800, zhankun tang , wrote:
> +1 (non-binding).
> Installed and verified it by running several Spark job and DS jobs.
>
> BR,
> Zhankun
>
> On Wed, 18 Sep 2019 at 08:05, Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
> > Verified the source and the binary tar and the sha512 checksums
> > Installed and verified the basic hadoop operations (ran few MR tasks)
> >
> > +1.
> >
> > Thanks,
> > + Naga
> >
> > On Wed, Sep 18, 2019 at 1:32 AM Anil Sadineni 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Tue, Sep 17, 2019 at 9:55 AM Santosh Marella 
> > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > On Wed, Sep 11, 2019 at 12:26 AM Rohith Sharma K S <
> > > > rohithsharm...@apache.org> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I have put together a release candidate (RC0) for Apache Hadoop
> > 3.2.1.
> > > > >
> > > > > The RC is available at:
> > > > > http://home.apache.org/~rohithsharmaks/hadoop-3.2.1-RC0/
> > > > >
> > > > > The RC tag in git is release-3.2.1-RC0:
> > > > > https://github.com/apache/hadoop/tree/release-3.2.1-RC0
> > > > >
> > > > >
> > > > > The maven artifacts are staged at
> > > > >
> > > https://repository.apache.org/content/repositories/orgapachehadoop-1226/
> > > > >
> > > > > You can find my public key at:
> > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > >
> > > > > This vote will run for 7 days(5 weekdays), ending on 18th Sept at
> > 11:59
> > > > pm
> > > > > PST.
> > > > >
> > > > > I have done testing with a pseudo cluster and distributed shell job.
> > My
> > > > +1
> > > > > to start.
> > > > >
> > > > > Thanks & Regards
> > > > > Rohith Sharma K S
> > > > >
> > > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Sadineni
> > > Solutions Architect, Optlin Inc
> > > Ph: 571-438-1974 | www.optlin.com
> > >
> >

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

[Weiwei Yang]

+1 (binding)

Thanks
Weiwei

On Wed, Sep 18, 2019 at 6:35 AM Wangda Tan  wrote:

> +1 (binding).
>
> From my experiences of Submarine project, I think moving to a separate repo
> helps.
>
> - Wangda
>
> On Tue, Sep 17, 2019 at 11:41 AM Subru Krishnan  wrote:
>
> > +1 (binding).
> >
> > IIUC, there will not be an Ozone module in trunk anymore as that was my
> > only concern from the original discussion thread? IMHO, this should be
> the
> > default approach for new modules.
> >
> > On Tue, Sep 17, 2019 at 9:58 AM Salvatore LaMendola (BLOOMBERG/ 731 LEX)
> <
> > slamendo...@bloomberg.net> wrote:
> >
> > > +1
> > >
> > > From: e...@apache.org At: 09/17/19 05:48:32To:
> > hdfs-...@hadoop.apache.org,
> > > mapreduce-...@hadoop.apache.org,  common-dev@hadoop.apache.org,
> > > yarn-...@hadoop.apache.org
> > > Subject: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk
> > > source tree
> > >
> > >
> > > TLDR; I propose to move Ozone related code out from Hadoop trunk and
> > > store it in a separated *Hadoop* git repository apache/hadoop-ozone.git
> > >
> > >
> > > When Ozone was adopted as a new Hadoop subproject it was proposed[1] to
> > > be part of the source tree but with separated release cadence, mainly
> > > because it had the hadoop-trunk/SNAPSHOT as compile time dependency.
> > >
> > > During the last Ozone releases this dependency is removed to provide
> > > more stable releases. Instead of using the latest trunk/SNAPSHOT build
> > > from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
> > >
> > > As we have no more strict dependency between Hadoop trunk SNAPSHOT and
> > > Ozone trunk I propose to separate the two code base from each other
> with
> > > creating a new Hadoop git repository (apache/hadoop-ozone.git):
> > >
> > > With moving Ozone to a separated git repository:
> > >
> > >   * It would be easier to contribute and understand the build (as of
> now
> > > we always need `-f pom.ozone.xml` as a Maven parameter)
> > >   * It would be possible to adjust build process without breaking
> > > Hadoop/Ozone builds.
> > >   * It would be possible to use different Readme/.asf.yaml/github
> > > template for the Hadoop Ozone and core Hadoop. (For example the current
> > > github template [2] has a link to the contribution guideline [3]. Ozone
> > > has an extended version [4] from this guideline with additional
> > > information.)
> > >   * Testing would be more safe as it won't be possible to change core
> > > Hadoop and Hadoop Ozone in the same patch.
> > >   * It would be easier to cut branches for Hadoop releases (based on
> the
> > > original consensus, Ozone should be removed from all the release
> > > branches after creating relase branches from trunk)
> > >
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > Marton
> > >
> > > [1]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8
> > > c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
> > > [2]:
> > >
> > >
> >
> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
> > > [3]:
> > https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
> > > [4]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
> > >
> > > -
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >
> > >
> > >
> >
>

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Weiwei Yang]

+1

Thanks
Weiwei
On Aug 27, 2019, 9:40 AM +0800, Xun Liu , wrote:
> +1 (non-binding)
> This is the best news.
>
> Peter Bacsko  于2019年8月27日周二 上午4:59写道：
>
> > +1 (non-binding)
> >
> > On Sat, Aug 24, 2019 at 4:06 AM Wangda Tan  wrote:
> >
> > > Hi devs,
> > >
> > > This is a voting thread to move Submarine source code, documentation from
> > > Hadoop repo to a separate Apache Git repo. Which is based on discussions
> > of
> > >
> > >
> > https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
> > >
> > > Contributors who have permissions to push to Hadoop Git repository will
> > > have permissions to push to the new Submarine repository.
> > >
> > > This voting thread will run for 7 days and will end at Aug 30th.
> > >
> > > Please let me know if you have any questions.
> > >
> > > Thanks,
> > > Wangda Tan
> > >
> >

Re: Aug Hadoop Community Meetup in China

[Weiwei Yang]

Hi Junping

Thanks. I would like to get a slot to talk about our new open source project: 
YuniKorn.

Thanks
Weiwei
On Jul 23, 2019, 5:08 PM +0800, 俊平堵 , wrote:
> Thanks for these positive feedbacks! The local community has voted the date 
> and location to be 8/10, Beijing. So please book your time ahead if you have 
> interest to join.
> I have gathered a few topics, and some candidate places for hosting this 
> meetup. If you would like to propose more topics, please nominate it here or 
> ping me before this weekend (7/28, CST time).
> Will update here when I have more to share. thx!
>
>
>
>
> <>
>
> <>
>
>
>
> Thanks,
>
> Junping
>
> > 俊平堵  于2019年7月18日周四 下午3:28写道：
> > > Hi, all!
> > > I am glad to let you know that we are organizing Hadoop Contributors 
> > > Meetup in China on Aug.
> > >
> > > This could be the first time hadoop community meetup in China and many 
> > > attendees are expected to come from big data pioneers, such as: Cloudera, 
> > > Tencent, Alibaba, Xiaomi, Didi, JD, Meituan, Toutiao, Sina, etc.
> > >
> > > We're still working out the details, such as dates, contents and 
> > > locations. Here is a quick survey: https://www.surveymonkey.com/r/Y99RT3W 
> > > where you can vote your prefer dates and locations if you would like to 
> > > attend - the survey will end in July. 21. 12PM China Standard Time, and 
> > > result will go public in next day.
> > >
> > > Also, please feel free to reach out to me if you have a topic to propose 
> > > for the meetup.  Will send out an update later with more details when I 
> > > get more to share. Thanks!
> > >
> > > Cheers,
> > >
> > > Junping

Re: Aug Hadoop Community Meetup in China

[Weiwei Yang]

Hi Junping

Thanks for organizing this event !
Just finished the survey, looking forward to meet folks in the coming meet up.

Thanks
Weiwei
On Jul 18, 2019, 3:28 PM +0800, 俊平堵 , wrote:
> Hi, all!
>
> I am glad to let you know that we are organizing
> Hadoop Contributors Meetup in China on Aug.
>
>
> This could be the first time hadoop community meetup in China and many
> attendees are expected to come from big data pioneers, such as: Cloudera,
> Tencent, Alibaba, Xiaomi, Didi, JD, Meituan, Toutiao, Sina, etc.
>
>
> We're still working out the details, such as dates, contents and locations.
> Here is a quick survey: https://www.surveymonkey.com/r/Y99RT3W where you
> can vote your prefer dates and locations if you would like to attend - the
> survey will end in July. 21. 12PM China Standard Time, and result will go
> public in next day.
>
>
> Also, please feel free to reach out to me if you have a topic to propose
> for the meetup. Will send out an update later with more details when I get
> more to share. Thanks!
>
>
> Cheers,
>
>
> Junping

Re: [VOTE] Force "squash and merge" option for PR merge on github UI

[Weiwei Yang]

Thanks Marton, +1 on this.

Weiwei

On Jul 17, 2019, 2:07 PM +0800, Elek, Marton , wrote:
> Hi,
>
> Github UI (ui!) helps to merge Pull Requests to the proposed branch.
> There are three different ways to do it [1]:
>
> 1. Keep all the different commits from the PR branch and create one
> additional merge commit ("Create a merge commit")
>
> 2. Squash all the commits and commit the change as one patch ("Squash
> and merge")
>
> 3. Keep all the different commits from the PR branch but rebase, merge
> commit will be missing ("Rebase and merge")
>
>
>
> As only the option 2 is compatible with the existing development
> practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
> consensus vote: If no objections withing 3 days, I will ask INFRA to
> disable the options 1 and 3 to make the process less error prone.
>
> Please let me know, what do you think,
>
> Thanks a lot
> Marton
>
> ps: Personally I prefer to merge from local as it enables to sign the
> commits and do a final build before push. But this is a different story,
> this proposal is only about removing the options which are obviously
> risky...
>
> ps2: You can always do any kind of merge / commits from CLI, for example
> to merge a feature branch together with keeping the history.
>
> [1]:
> https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

[ANNOUNCE] New Apache Hadoop Committer - Tao Yang

[Weiwei Yang]

Hi Dear Apache Hadoop Community

It's my pleasure to announce that Tao Yang has been elected as an Apache
Hadoop committer, this is to recognize his contributions to Apache Hadoop
YARN project.

Congratulations and welcome on board!

Weiwei
(On behalf of the Apache Hadoop PMC)

Re: Agenda & More Information about Hadoop Community Meetup @ Palo Alto, June 26

[Weiwei Yang]

Thanks Wangda.
Will this event be recorded? It will be extremely helpful for people who are 
unable to join to catch up.

Thanks
Weiwei
On Jun 26, 2019, 4:12 AM +0800, Wangda Tan , wrote:
A friendly reminder,

The meetup will take place tomorrow at 9:00 AM PDT to 4:00 PM PDT.

The address is: 395 Page Mill Rd, Palo Alto, CA 94306
We’ll be in the Bigtop conference room on the 1st floor. Go left after
coming through the main entrance, and it will be on the right.

Zoom: https://cloudera.zoom.us/j/606607666

Please let me know if you have any questions. If you haven't RSVP yet,
please go ahead and RSVP so we can better prepare food, seat, etc.

Thanks,
Wangda

On Wed, Jun 19, 2019 at 4:49 PM Wangda Tan  wrote:

Hi All,

I want to let you know that we have confirmed most of the agenda for
Hadoop Community Meetup. It will be a whole day event.

Agenda & Dial-In info because see below, *please RSVP
at https://www.meetup.com/Hadoop-Contributors/events/262055924/
*

Huge thanks to Daniel Templeton, Wei-Chiu Chuang, Christina Vu for helping
with organizing and logistics.

*Please help to promote meetup information on Twitter, LinkedIn, etc.
Appreciated! *

Best,
Wangda

























































*AM:9:00: Arrival and check-in--9:30 -
10:15:-Talk: Hadoop storage in cloud-native
environmentsAbstract: Hadoop is a mature storage system but designed years
before the cloud-native movement. Kubernetes and other cloud-native tools
are emerging solutions for containerized environments but sometimes they
require different approaches.In this presentation we would like to share
our experiences to run Apache Hadoop Ozone in Kubernetes and the connection
point to other cloud-native ecosystem elements. We will compare the
benefits and drawbacks to use Kubernetes and Hadoop storage together and
show our current achievements and future plans.Speaker: Marton Elek
(Cloudera)10:20 - 11:00:--Talk: Selective Wire Encryption In
HDFSAbstract: Wire data encryption is a key component of the Hadoop
Distributed File System (HDFS). However, such encryption enforcement comes
in as an all-or-nothing feature. In our use case at LinkedIn, we would like
to selectively expose fast unencrypted access to fully managed internal
clients, which can be trusted, while only expose encrypted access to
clients outside of the trusted circle with higher security risks. That way
we minimize performance overhead for trusted internal clients while still
securing data from potential outside threats. Our design extends HDFS
NameNode to run on multiple ports, connecting to different NameNode ports
would end up with different levels of encryption protection. This
protection then gets enforced for both NameNode RPC and the subsequent data
transfers to/from DataNode. This approach comes with minimum operational
and performance overhead.Speaker: Konstantin Shvachko (LinkedIn), Chen
Liang (LinkedIn)11:10 - 11:55:-Talk: YuniKorn: Next Generation
Scheduling for YARN and K8sAbstract: We will talk about our open source
work - YuniKorn scheduler project (Y for YARN, K for K8s, uni- for Unified)
brings long-wanted features such as hierarchical queues, fairness between
users/jobs/queues, preemption to Kubernetes; and it brings service
scheduling enhancements to YARN. Any improvements to this scheduler can
benefit both Kubernetes and YARN community.Speaker: Wangda Tan
(Cloudera)PM:12:00 - 12:55 Lunch Break (Provided by
Cloudera)1:00 -
1:25---Talk: Yarn Efficiency at UberAbstract: We will present the
work done at Uber to improve YARN cluster utilization and job SOA with
elastic resource management, low compute workload on passive datacenter,
preemption, larger container, etc. We will also go through YARN upgrade in
order to adopt new features and talk about the challenges.Speaker: Aihua Xu
(Uber), Prashant Golash (Uber)1:30 - 2:10 One more
talk-2:20 - 4:00---BoF sessions &
Breakout Sessions & Group discussions: Talk about items like JDK 11
support, next releases (2.10.0, 3.3.0, etc.), Hadoop on Cloud, etc.4:00:
Reception provided by
Cloudera.==Join Zoom
Meetinghttps://cloudera.zoom.us/j/116816195
*

[jira] [Reopened] (HADOOP-16344) Make DurationInfo "public unstable"

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang reopened HADOOP-16344:
--

> Make DurationInfo  "public unstable"
> 
>
> Key: HADOOP-16344
> URL: https://issues.apache.org/jira/browse/HADOOP-16344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: util
>Reporter: Kevin Risden
>Assignee: kevin su
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-16344.01.patch
>
>
> HADOOP-16093 moved DurationInfo to hadoop-common org.apache.hadoop.util. It 
> would be useful if DurationInfo was annotated as "public unstable".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15746) Hadoop: branch-2.8 docker image failed to build

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HADOOP-15746.
--
   Resolution: Fixed
Fix Version/s: 2.8.5
   2.9.2

> Hadoop: branch-2.8 docker image failed to build
> ---
>
> Key: HADOOP-15746
> URL: https://issues.apache.org/jira/browse/HADOOP-15746
> Project: Hadoop Common
>  Issue Type: Bug
>    Reporter: Weiwei Yang
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 2.9.2, 2.8.5
>
>
> The jenkins build was failing on hadoop branch-2.8 for quite a while, see an 
> example result in YARN-8664. The error
> {noformat}
>  Step 24/31 : RUN pip install pylint
>  ---> Running in b1f0e6c1d209
>  Downloading/unpacking pylint
>  ...
>  setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old 
> (<12)
>  Complete output from command python setup.py egg_info:
>  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution 
> option: 'python_requires'
> {noformat}
>  detail output can be seen 
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/21808/console].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: YARN SLS improving idea

[Weiwei Yang]

Hi Sichen

Sure, please go ahead creating JIRA to capture your ideas, we can move 
discussion to smaller groups there.
At meanwhile, I am interested how you plan to support a large number of 
“specific” resource requests, look forward to seeing more details.
Looks like 2.b depends on YARN-3409 but it is not yet merged, maybe that can be 
another JIRA.

Thanks!
-- 
Weiwei

On 8/16/18, 12:06 PM, "Sichen Zhao"  wrote:

Hi all,
Thanks for your advise.
1. I will create JIRA once the details are identified.
2. I read the patch YARN-8007. the scheduling request already done, but 
there are some difference between the patch and my ideas:
a) the patch use the SYNTH as schedulingrequest input. But SYNTH can't 
create a large number of specific resource requests. I wanna create a new
schedulingrequest input format for the more authentic input.
b) the patch doesn't add the attribute for the Node. I think this can 
be done.

Best Regards
Sichen Zhao


From: Weiwei Yang 
Sent: Tuesday, August 14, 2018 8:54:55 AM
To: Sunil G; Yufei Gu
Cc: Daniel Templeton; zsc19940...@outlook.com; Hadoop Common; Wangda Tan
Subject: Re: YARN SLS improving idea

Hi Sichen

Thanks for proposing this, this for sure a nice enhancement for SLS.
For SLS improvements, feel free to create JIRA tickets under 
https://issues.apache.org/jira/browse/YARN-5065, that’s the umbrella we track 
all related items.
For scheduling requests support, there is an existing JIRA and an existing 
patch in https://issues.apache.org/jira/browse/YARN-8007, would you like to 
take a look?

Thanks
--
Weiwei

From: Sunil G 
Date: Tuesday, August 14, 2018 at 1:31 AM
To: Yufei Gu 
Cc: Daniel Templeton , YANG WEIWEI 
, "zsc19940...@outlook.com" , 
Hadoop Common , Wangda Tan 
Subject: Re: YARN SLS improving idea

Hi Sichen

1. Add input support for the scheduling request format.
Yes. I suppose this change is in SLS end (client to generate requests)
2. Add support for scheduling request resource format in NMSim.
Makes sense.
3. Adding scheduling request support for the Capacity Scheduler(maybe it is 
already done in current version).
This support is already there.

Adding to this, major challenge is to specify constrains per app level and 
verifying results. I also suggest to cross check the scheduler invariants check 
support added as per YARN-6547 and see how we can incorporate same to predict 
o/p

- Sunil


On Mon, Aug 13, 2018 at 10:39 PM Yufei Gu 
mailto:flyrain...@gmail.com>> wrote:
+YANG WEIWEI<mailto:cheersy...@hotmail.com>

Make sense to me from SLS perspective, but I am not familiar with Placement 
Constraints. Add WeiWei.

Best,

Yufei

`This is not a contribution`


On Sat, Aug 11, 2018 at 8:57 AM Daniel Templeton 
mailto:dan...@cloudera.com>> wrote:
Yufei, Wangda, Sunil, any comments?

Daniel

On 8/11/18 8:48 AM, Sichen Zhao wrote:
> Hi,
> Is there anyone who can reply my ideas?
>
> Best Regards
> Sichen Zhao
>
> 
> From: Sichen Zhao 
mailto:zsc19940...@outlook.com>>
> Sent: Friday, August 10, 2018 11:10
> To: Hadoop Common
> Subject: YARN SLS improving idea
>
> Hi,
> I am a developer from AliBaBa China, i recently used SLS for scheduling 
simulation, SLS currently supports multidimensional resource input(CPU, mem , 
other resources: disk). But SLS can't take scheduling request, which is 
currently widely used in YARN, as input, so Placement Constraints and 
attributes are not supported.
>
> So what i wanna improve the SLS: Add scheduling emulation for scheduling 
request resource format.
>
> The specific work is as follows:
> 1. Add input support for the scheduling request format.
> 2. Add support for scheduling request resource format in NMSim.
> 3. Adding scheduling request support for the Capacity Scheduler(maybe it 
is already done in current version).
>
> What do you think about my ideas?
>
>
> Best Regards
> Sichen Zhao
>
> -
> To unsubscribe, e-mail: 
common-dev-unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: 
common-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org>
>
>
> -
> To unsubscribe, e-mail: 
common-dev-unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: 
common-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org>
>

Re: YARN SLS improving idea

[Weiwei Yang]

Hi Sichen

Thanks for proposing this, this for sure a nice enhancement for SLS.
For SLS improvements, feel free to create JIRA tickets under 
https://issues.apache.org/jira/browse/YARN-5065, that’s the umbrella we track 
all related items.
For scheduling requests support, there is an existing JIRA and an existing 
patch in https://issues.apache.org/jira/browse/YARN-8007, would you like to 
take a look?

Thanks
--
Weiwei

From: Sunil G 
Date: Tuesday, August 14, 2018 at 1:31 AM
To: Yufei Gu 
Cc: Daniel Templeton , YANG WEIWEI 
, "zsc19940...@outlook.com" , 
Hadoop Common , Wangda Tan 
Subject: Re: YARN SLS improving idea

Hi Sichen

1. Add input support for the scheduling request format.
Yes. I suppose this change is in SLS end (client to generate requests)
2. Add support for scheduling request resource format in NMSim.
Makes sense.
3. Adding scheduling request support for the Capacity Scheduler(maybe it is 
already done in current version).
This support is already there.

Adding to this, major challenge is to specify constrains per app level and 
verifying results. I also suggest to cross check the scheduler invariants check 
support added as per YARN-6547 and see how we can incorporate same to predict 
o/p

- Sunil


On Mon, Aug 13, 2018 at 10:39 PM Yufei Gu 
mailto:flyrain...@gmail.com>> wrote:
+YANG WEIWEI

Make sense to me from SLS perspective, but I am not familiar with Placement 
Constraints. Add WeiWei.

Best,

Yufei

`This is not a contribution`


On Sat, Aug 11, 2018 at 8:57 AM Daniel Templeton 
mailto:dan...@cloudera.com>> wrote:
Yufei, Wangda, Sunil, any comments?

Daniel

On 8/11/18 8:48 AM, Sichen Zhao wrote:
> Hi,
> Is there anyone who can reply my ideas?
>
> Best Regards
> Sichen Zhao
>
> 
> From: Sichen Zhao mailto:zsc19940...@outlook.com>>
> Sent: Friday, August 10, 2018 11:10
> To: Hadoop Common
> Subject: YARN SLS improving idea
>
> Hi,
> I am a developer from AliBaBa China, i recently used SLS for scheduling 
> simulation, SLS currently supports multidimensional resource input(CPU, mem , 
> other resources: disk). But SLS can't take scheduling request, which is 
> currently widely used in YARN, as input, so Placement Constraints and 
> attributes are not supported.
>
> So what i wanna improve the SLS: Add scheduling emulation for scheduling 
> request resource format.
>
> The specific work is as follows:
> 1. Add input support for the scheduling request format.
> 2. Add support for scheduling request resource format in NMSim.
> 3. Adding scheduling request support for the Capacity Scheduler(maybe it is 
> already done in current version).
>
> What do you think about my ideas?
>
>
> Best Regards
> Sichen Zhao
>
> -
> To unsubscribe, e-mail: 
> common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> common-dev-h...@hadoop.apache.org
>
>
> -
> To unsubscribe, e-mail: 
> common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> common-dev-h...@hadoop.apache.org
>

Re: Apache Hadoop 3.1.1 release plan

[Weiwei Yang]

Hi Wangda

I would propose to have https://issues.apache.org/jira/browse/YARN-8015 
included in 3.1.1.

Once this is done, we get both intra and inter placement constraint covered so 
users could start to explore this feature. Otherwise the functionality is 
pretty limited. It has been Patch Available for a while, I just promoted it 
targeting to 3.1.1. Hope that makes sense.

Thanks!

--
Weiwei

On 11 May 2018, 9:02 AM +0800, Wangda Tan , wrote:
Hi all,

As we previously proposed RC time (May 1st), we want to release 3.1.1
sooner if possible. As of now, 3.1.1 has 187 fixes [1] on top of 3.1.0, and
there're 10 open blockers/criticals which target to 3.1.1 [2]. I just
posted comments to these open criticals/blockers ticket owners asking about
statuses.

If everybody agrees, I propose start code freeze of branch-3.1 from Sat PDT
time this week, only blockers/criticals can be committed to branch-3.1. To
avoid the burden of committers, I want to delay cutting branch-3.1.1 as
late as possible. If you have any major/minors (For severe issues please
update priorities) tickets want to go to 3.1.1, please reply this email
thread and we can look at them and make a call together.

Please feel free to share your comments and suggestions.

Thanks,
Wangda

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
AND status = Resolved AND fixVersion = 3.1.1
[2] project in (YARN, HADOOP, MAPREDUCE, "Hadoop Development Tools") AND
priority in (Blocker, Critical) AND resolution = Unresolved AND "Target
Version/s" = 3.1.1 ORDER BY priority DESC


On Thu, May 10, 2018 at 5:48 PM, Wangda Tan  wrote:

Thanks Brahma/Sunil,

For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
target version.
For YARN-8236, it is a severe issue and I think it is close to finish.



On Thu, May 10, 2018 at 3:08 AM, Sunil G  wrote:


Thanks Brahma.
Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.

- Sunil


On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
addition to 3.1 line release for improving quality.

Looks only following two are pending which are in review state. Hope you
are monitoring these two.

https://issues.apache.org/jira/browse/YARN-8265
https://issues.apache.org/jira/browse/YARN-8236



Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
branch-3.1


-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com]
Sent: 19 April 2018 17:49
To: Hadoop Common ;
mapreduce-...@hadoop.apache.org; Hdfs-dev ;
yarn-...@hadoop.apache.org
Subject: Apache Hadoop 3.1.1 release plan

Hi, All

We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
quality of the release, we plan to release 3.1.1 at May 06. The focus of
3.1.1 will be fixing blockers / critical bugs and other enhancements. So
far there are 100 JIRAs [1] have fix version marked to 3.1.1.

We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.

Please feel free to share your insights.

Thanks,
Wangda Tan

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
Map/Reduce") AND fixVersion = 3.1.1

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Weiwei Yang]

Hi Wangda

+1 (non-binding)

- Smoke tests with teragen/terasort/DS jobs
- Various of metrics, UI displays validation, compatible tests
- Tested GPU resource-type
- Verified RM fail-over, app-recovery
- Verified 2 threads async-scheduling
- Enabled placement constraints, tested affinity/anti-affinity allocations
- SLS tests

--
Weiwei

On 2 Apr 2018, 1:13 PM +0800, Brahma Reddy Battula , wrote:
Wangda thanks for driving this.

+1(binding)

--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.


On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan  wrote:

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC0)

[Weiwei Yang]

Hi Wangda

While testing the build, we found a bug 
https://issues.apache.org/jira/browse/YARN-8085, it might cause NPE during RM 
fail-over. I think that needs to be included in 3.1.0.

We have run some other tests, and they look good so far. This includes
- Some basic teragen/terasort/DS jobs
- Various of metrics, UI displays, compatible tests
- Creating queues, different queue configurations
- Verified RM fail-over, app-recovery
- Enabled async scheduling
- Enabled placement constraints
- SLS tests

Hope it helps.
Thanks

--
Weiwei

On 29 Mar 2018, 12:34 PM +0800, Ajay Kumar , wrote:
+1 (non-binding)

- verified binary checksum
- built from source and setup 4 node cluster
- run basic hdfs command
- run wordcount, pi & TestDFSIO (read/write)

Ajay

On 3/28/18, 5:45 PM, "Jonathan Hung"  wrote:

Hi Wangda, thanks for handling this release.

+1 (non-binding)

- verified binary checksum
- launched single node RM
- verified refreshQueues functionality
- verified capacity scheduler conf mutation disabled in this case
- verified capacity scheduler conf mutation with leveldb storage
- verified refreshQueues mutation is disabled in this case


Jonathan Hung

On Thu, Mar 22, 2018 at 9:10 AM, Wangda Tan  wrote:

Thanks @Bharat for the quick check, the previously staged repository has
some issues. I re-deployed jars to nexus.

Here's the new repo (1087)

https://repository.apache.org/content/repositories/orgapachehadoop-1087/

Other artifacts remain same, no additional code changes.

On Wed, Mar 21, 2018 at 11:54 PM, Bharat Viswanadham <
bviswanad...@hortonworks.com> wrote:

Hi Wangda,
Maven Artifact repositories is not having all Hadoop jars. (It is missing
many like hadoop-hdfs, hadoop-client etc.,)
https://repository.apache.org/content/repositories/orgapachehadoop-1086/


Thanks,
Bharat


On 3/21/18, 11:44 PM, "Wangda Tan"  wrote:

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC0 for Apache Hadoop 3.1.0. The artifacts are available
here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC0/

The RC tag in git is release-3.1.0-RC0.

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1086/

This vote will run 7 days (5 weekdays), ending on Mar 28 at 11:59 pm
Pacific.

3.1.0 contains 727 [2] fixed JIRA issues since 3.0.0. Notable
additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow
HDFS
block replicas to be provided by an external storage system, etc.

We’d like to use this as a starting release for 3.1.x [1], depending
on how
it goes, get it stabilized and potentially use a 3.1.1 in several
weeks as
the stable release.

We have done testing with a pseudo cluster and distributed shell job.
My +1
to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
(3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER
BY
fixVersion ASC






B�CB��[��X��ܚX�KK[XZ[���Y]�][��X��ܚX�PY���\X�K�ܙ�B��܈Y][ۘ[��[X[��K[XZ[���Y]�Z[Y���\X�K�ܙ�B�

Re: [NOTICE] branch-3.1 created. 3.1.0 code freeze started.

[Weiwei Yang]

Hi Wangda

I am a bit confused about the branches, I am expecting something like following

 branch-3.1.0: 3.1.0
 branch-3.1: 3.1.x
 trunk: 3.2.x

Code freezes in branch-3.1.0 which only allows blocks.
However, did you mean you created branch-3.1 for 3.1.0? In this case, what 
branch should be used for 3.1.x?

--
Weiwei

On 11 Feb 2018, 11:11 AM +0800, Wangda Tan , wrote:
Hi all,

As proposed in [1], code freeze of 3.1.0 is already started. I created
branch-3.1 from trunk and will set target version of trunk to 3.2.0.

Please note that only blockers/criticals can be committed to 3.1.0, and all
such commits should be cherry-picked to branch-3.1.

[1]
https://lists.apache.org/thread.html/0d8050ecef4181f8355b3fed732018ad970b6cd11e9436db322b0dd9@%3Cyarn-dev.hadoop.apache.org%3E

Thanks,
Wangda

Re: [VOTE] Merge YARN-6592 feature branch to trunk

[Weiwei Yang]

+1
We are also looking forward to this : )

--
Weiwei

On 27 Jan 2018, 9:16 AM +0800, Wangda Tan , wrote:
+1

On Sat, Jan 27, 2018 at 2:05 AM, Chris Douglas 
> wrote:
+1 Looking forward to this. -C

On Fri, Jan 26, 2018 at 7:28 AM, Arun Suresh 
> wrote:
> Hello yarn-dev@
>
> Based on the positive feedback from the DISCUSS thread [1], I'd like to
> start a formal vote to merge YARN-6592 [2] to trunk. The vote will run for 5
> days, and will end Jan 31 7:30AM PDT.
>
> This feature adds support for placing containers in YARN using rich
> placement constraints. For example, this can be used by applications to
> co-locate containers on a node or rack (affinity constraint), spread
> containers across nodes or racks (anti-affinity constraint), or even specify
> the maximum number of containers on a node/rack (cardinality constraint).
>
> We have integrated this feature into the Distributed-Shell application for
> feature testing. We have performed end-to-end testing on moderately-sized
> clusters to verify that constraints work fine. Performance tests have been
> done via both SLS tests and Capacity Scheduler performance unit tests, and
> no regression was found. We have opened a JIRA to track Jenkins acceptance
> of the aggregated patch [3]. Documentation is in the process of being
> completed [4]. You can also check our design document for more details [5].
>
> Config flags are needed to enable this feature and it should not have any
> effect on YARN when turned off. Once merged, we plan to work on further
> improvements, which can be tracked in the umbrella YARN-7812 [6].
>
> Kindly do take a look at the branch and raise issues/concerns that need to
> be addressed before the merge.
>
> Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for their
> contributions to this effort, as well as Subru, Chris, Carlo, and Vinod for
> their inputs and discussions.
>
>
> Cheers
> -Arun
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201801.mbox/%3CCAMreUaz%3DGnsjOLZ%3Dem2x%3DQS7qh27euCWNw6Bo_4Cu%2BfXnXhyNA%40mail.gmail.com%3E
> [2] https://issues.apache.org/jira/browse/YARN-6592
> [3] https://issues.apache.org/jira/browse/YARN-7792
> [4] https://issues.apache.org/jira/browse/YARN-7780
> [5]
> https://issues.apache.org/jira/secure/attachment/12867869/YARN-6592-Rich-Placement-Constraints-Design-V1.pdf
> [6] https://issues.apache.org/jira/browse/YARN-7812

-
To unsubscribe, e-mail: 
yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 
yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merge YARN-6592 to trunk

[Weiwei Yang]

+1, thanks for getting to this milestone Arun.
I’ve done some basic validations on a 4 nodes cluster, with some general 
affinity/anti-affinty/cardinality constraints, it worked. I’ve also reviewed 
the doc, it’s in good shape and very illustrative.

Thanks.

--
Weiwei

On 26 Jan 2018, 10:44 AM +0800, Sunil G , wrote:
+1.

Thanks Arun.

I did manual testing for check affinity and anti-affinity features with 
placement allocator. Also checked SLS to see any performance regression, and 
there are not much difference as Arun mentioned.

Thanks all the folks for working on this. Kudos!

- Sunil


On Fri, Jan 26, 2018 at 5:16 AM Arun Suresh 
> wrote:
Hello yarn-dev@

We feel that the YARN-6592 dev branch mostly in shape to be merged into trunk. 
This branch adds support for placing containers in YARN using rich placement 
constraints. For example, this can be used by applications to co-locate 
containers on a node or rack (affinity constraint), spread containers across 
nodes or racks (anti-affinity constraint), or even specify the maximum number 
of containers on a node/rack (cardinality constraint).

We have integrated this feature into the Distributed-Shell application for 
feature testing. We have performed end-to-end testing on moderately-sized 
clusters to verify that constraints work fine. Performance tests have been done 
via both SLS tests and Capacity Scheduler performance unit tests, and no 
regression was found. We have opened a JIRA to track Jenkins acceptance of the 
aggregated patch [2]. Documentation is in the process of being completed [3]. 
You can also check our design document for more details [4].

Config flags are needed to enable this feature and it should not have any 
effect on YARN when turned off. Once merged, we plan to work on further 
improvements, which can be tracked in the umbrella YARN-7812 [5].

Kindly do take a look at the branch and raise issues/concerns that need to be 
addressed before the merge.

Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for their 
contributions to this effort, as well as Subru, Chris, Carlo, and Vinod for 
their inputs and discussions.

Cheers
-Arun


[1] https://issues.apache.org/jira/browse/YARN-6592
[2] https://issues.apache.org/jira/browse/YARN-7792
[3] https://issues.apache.org/jira/browse/YARN-7780
[4] 
https://issues.apache.org/jira/secure/attachment/12867869/YARN-6592-Rich-Placement-Constraints-Design-V1.pdf
[5] https://issues.apache.org/jira/browse/YARN-7812

Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

[Weiwei Yang]

Hi Andrew

Congratulations on the 3.0 GA, another mile stone of Hadoop history. Thanks for 
driving this release, excellent work!
Took a look at the release notes, also seems nice. But one question, who is 
maintaining https://wiki.apache.org/hadoop/PoweredBy? Some of info is quite out 
of date, would like to know how to update.

Thanks

--
Weiwei

On 15 Dec 2017, 2:45 AM +0800, Andrew Wang , wrote:
Hi all,

I'm pleased to announce that Apache Hadoop 3.0.0 is generally available
(GA).

3.0.0 GA consists of 302 bug fixes, improvements, and other enhancements
since 3.0.0-beta1. This release marks a point of quality and stability for
the 3.0.0 release line, and users of earlier 3.0.0-alpha and -beta releases
are encouraged to upgrade.

Looking back, 3.0.0 GA is the culmination of over a year of work on the
3.0.0 line, starting with 3.0.0-alpha1 which was released in September
2016. Altogether, 3.0.0 incorporates 6,242 changes since 2.7.0.

Users are encouraged to read the overview of major changes
 in 3.0.0. The GA release
notes

Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

[Weiwei Yang]

+1 (non-binding)
Thanks for getting this done Sunil.

--
Weiwei

On 5 Dec 2017, 4:06 AM +0800, Eric Payne , 
wrote:
+1. Thanks Sunil for the work on this branch.
Eric

From: Sunil G ; Hdfs-dev 
; Hadoop Common ; 
"mapreduce-...@hadoop.apache.org" 

Re: [DISCUSS] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

[Weiwei Yang]

Hi Sunil

+1 from my side.
Actually we have applied some of these patches to our production cluster since 
Sep this year, on over 2000+ nodes and it works nicely. +1 for the merge. I am 
pretty sure this feature will help a lot of users, especially those on cloud. 
Thanks for getting this done, great job!

--
Weiwei

On 29 Nov 2017, 9:23 PM +0800, Rohith Sharma K S , 
wrote:
+1, thanks Sunil for working on this feature!

-Rohith Sharma K S

On 24 November 2017 at 23:19, Sunil G  wrote:

Hi All,

We would like to bring up the discussion of merging “absolute min/max
resources support in capacity scheduler” branch (YARN-5881) [2] into trunk
in a few weeks. The goal is to get it in for Hadoop 3.1.

*Major work happened in this branch*

- YARN-6471. Support to add min/max resource configuration for a queue
- YARN-7332. Compute effectiveCapacity per each resource vector
- YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to
handle absolute resources.

*Regarding design details*

Please refer [1] for detailed design document.

*Regarding to testing:*

We did extensive tests for the feature in the last couple of months.
Comparing to latest trunk.

- For SLS benchmark: We didn't see observable performance gap from
simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
containers allocated per second.

- For microbenchmark: We use performance test cases added by YARN 6775, it
did not show much performance regression comparing to trunk.

*YARN-5881* 

Re: Apache Hadoop 2.8.3 Release Plan

[Weiwei Yang]

Agree with Konstantin. This two issues has been opened for a while but could 
not reach a consensus on the fix, hope this gets enough attention from the 
community and get them resolved.

Thanks

--
Weiwei

On 22 Nov 2017, 11:18 AM +0800, Konstantin Shvachko , 
wrote:
I would consider these two blockers for 2.8.3 as they crash NN:

https://issues.apache.org/jira/browse/HDFS-12638
https://issues.apache.org/jira/browse/HDFS-12832

Thanks,
--Konstantin

On Tue, Nov 21, 2017 at 11:16 AM, Junping Du  wrote:

Thanks Andrew and Wangda for comments!

To me, an improvement with 17 patches is not a big problem as this is
self-contained and I didn't see a single line of delete/update on existing
code - well, arguably, patches with only adding code can also have big
impact but not likely the case here.

While the dependency discussions on HADOOP-14964 are still going on, I
will leave the decision to JIRA discussion based on which approach we will
choose(shaded?) and impact. If we cannot make consensus in short term,
probably we have to miss this in 2.8.3 release.


Okay. Last call for blocker/critical fixes landing on branch-2.8.3. RC0
will get cut-off shortly.



Thanks,


Junping



From: Wangda Tan > wrote:
The Aliyun OSS code isn't a small improvement. If you look at Sammi's
comment
,
it's
a 17 patch series that is being backported in one shot. What we're talking
about is equivalent to merging a feature branch in a maintenance release. I
see that Kai and Chris are having a discussion about the dependency
changes, which indicates this is not a zero-risk change either. We really
should not be changing dependency versions in a maintenance unless it's
because of a bug.

It's unfortunate from a timing perspective that this missed 2.9.0, but I
still think it should wait for the next minor. Merging a feature into a
maintenance release sets the wrong precedent.

Best,
Andrew

On Tue, Nov 21, 2017 at 1:08 AM, Junping Du > wrote:

Thanks Kai for calling out this feature/improvement for attention and
Andrew for comments.


While I agree that maintenance release should focus on important bug fix
only, I doubt we have strict rules to disallow any features/improvements
to
land on maint release especially when those are small footprint or low
impact on existing code/features. In practice, we indeed has 77 new
features/improvements in latest 2.7.3 and 2.7.4 release.


Back to HADOOP-14964, I did a quick check and it looks like case here
belongs to self-contained improvement that has very low impact on
existing
code base, so I am OK with the improvement get landed on branch-2.8 in
case
it is well reviewed and tested.


However, as RM of branch-2.8, I have two concerns to accept it in our
2.8.3 release:

1. Timing - as I mentioned in beginning, the main purpose of 2.8.3 are
for
several critical bug fixes and we should target to release it very soon -
my current plan is to cut RC out within this week inline with waiting
for 3.0.0 vote closing. Can this improvement be well tested against
branch-2.8.3 within this strictly timeline? It seems a bit rush unless we
have strong commitment on test plan and activities in such a tight time.


2. Upgrading - I haven't heard we settle down the plan of releasing this
feature in 2.9.1 release - though I saw some discussions are going on
at HADOOP-14964. Assume 2.8.3 is released ahead of 2.9.1 and it includes
this improvement, then users consuming this feature/improvement have no
2.9
release to upgrade or forcefully upgrade with 

[jira] [Created] (HADOOP-14400) Fix warnings from spotbugs in hadoop-tools

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-14400:


 Summary: Fix warnings from spotbugs in hadoop-tools
 Key: HADOOP-14400
 URL: https://issues.apache.org/jira/browse/HADOOP-14400
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Weiwei Yang
Assignee: Weiwei Yang


Fix 4 warnings in hadoop-tools project since moved to spotbugs.
# Return value of new 
org.apache.hadoop.tools.rumen.datatypes.DefaultDataType(String) ignored, but 
method has no side effect At MapReduceJobPropertiesParser.java
# org.apache.hadoop.mapred.gridmix.InputStriper$1.compare(Map$Entry, Map$Entry) 
incorrectly handles double value
# Useless object stored in variable keysToUpdateAsFolder of method 
org.apache.hadoop.fs.azure.NativeAzureFileSystem.mkdirs(Path, FsPermission, 
boolean) At NativeAzureFileSystem.java
# org.apache.hadoop.yarn.sls.SLSRunner.simulateInfoMap is a mutable collection 
At SLSRunner.java




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14339) Fix warnings from Spotbugs in hadoop-mapreduce

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-14339:


 Summary: Fix warnings from Spotbugs in hadoop-mapreduce
 Key: HADOOP-14339
 URL: https://issues.apache.org/jira/browse/HADOOP-14339
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Weiwei Yang
Assignee: Weiwei Yang


Fix warnings from Spotbugs in hadoop-mapreduce since switched from findbugs to 
spotbugs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14338) Fix warnings from Spotbugs in hadoop-yarn

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-14338:


 Summary: Fix warnings from Spotbugs in hadoop-yarn
 Key: HADOOP-14338
 URL: https://issues.apache.org/jira/browse/HADOOP-14338
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Weiwei Yang
Assignee: Weiwei Yang


Fix warnings from Spotbugs in hadoop-yarn since switched from findbugs to 
spotbugs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13837) Process check bug in hadoop_stop_daemon

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-13837:


 Summary: Process check bug in hadoop_stop_daemon
 Key: HADOOP-13837
 URL: https://issues.apache.org/jira/browse/HADOOP-13837
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts
Reporter: Weiwei Yang
Assignee: Weiwei Yang


Always get {{ERROR: Unable to kill ...}} after {{Trying to kill with kill -9}}, 
see following output of stop-yarn.sh

{code}
: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying 
to kill with kill -9
: ERROR: Unable to kill 18097
{code}

hadoop_stop_daemon doesn't check process liveness correctly, this bug can be 
reproduced by the script easily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13645) Refine TestRackResolver#testCaching to ensure cache is truly tested

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HADOOP-13645.
--
Resolution: Not A Bug

Though it was not using neat way, it actually tested the cache. It would be 
good to rewrite this test case for better reading, but lower priority. Close as 
Not a bug for now. Any people who is interested to improve this, feel free to 
reopen.

> Refine TestRackResolver#testCaching to ensure cache is truly tested
> ---
>
> Key: HADOOP-13645
> URL: https://issues.apache.org/jira/browse/HADOOP-13645
> Project: Hadoop Common
>  Issue Type: Test
>  Components: util
>Affects Versions: 2.7.3
>    Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HADOOP-13645.01.patch
>
>
> TestRackResolver#testCaching seems not cover the cache testing well, the test 
> case won't fail even the cache wasn't used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13645) Refine TestRackResolver#testCaching to ensure cache is truly tested

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-13645:


 Summary: Refine TestRackResolver#testCaching to ensure cache is 
truly tested
 Key: HADOOP-13645
 URL: https://issues.apache.org/jira/browse/HADOOP-13645
 Project: Hadoop Common
  Issue Type: Test
  Components: util
Reporter: Weiwei Yang
Assignee: Weiwei Yang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13639) Support plain text in ConfServlet http response

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-13639:


 Summary: Support plain text in ConfServlet http response
 Key: HADOOP-13639
 URL: https://issues.apache.org/jira/browse/HADOOP-13639
 Project: Hadoop Common
  Issue Type: Improvement
  Components: conf
Affects Versions: 2.7.3
Reporter: Weiwei Yang


Per discussion in HADOOP-13628, it would be good if ConfServlet also support to 
return plain text http response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13628) Support to retrieve specific property from configuration via REST API

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-13628:


 Summary: Support to retrieve specific property from configuration 
via REST API
 Key: HADOOP-13628
 URL: https://issues.apache.org/jira/browse/HADOOP-13628
 Project: Hadoop Common
  Issue Type: Improvement
  Components: conf
Affects Versions: 2.7.3
Reporter: Weiwei Yang
Assignee: Weiwei Yang


Currently we can use rest API to retrieve all configuration properties per 
daemon, but unable to get a specific property by name. This causes extra parse 
work at client side when dealing with Hadoop configurations, and also it's 
quite over head to send all configuration in a http response over network. 
Propose to support following a {{name}} parameter in the http request,

{code}
curl --header "Accept:application/json" 
http://${RM_HOST}/conf?name=yarn.nodemanager.aux-services

{"property"{"key":"yarn.resourcemanager.hostname","value":"${RM_HOST}","isFinal":false,"resource":"yarn-site.xml"}}
{code}

This change is fully backwards compatible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13588) ConfServlet doesn't set response content type based on accept header

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-13588:


 Summary: ConfServlet doesn't set response content type based on 
accept header 
 Key: HADOOP-13588
 URL: https://issues.apache.org/jira/browse/HADOOP-13588
 Project: Hadoop Common
  Issue Type: Bug
  Components: conf
Reporter: Weiwei Yang
Assignee: Weiwei Yang


ConfServlet provides a general service to retrieve daemon configurations. 
However it doesn't set response content-type according to *Accept* header. For 
example, by issuing following command, 

{code}
curl --header "Accept:application/json" http://${resourcemanager_host}:8088/conf
{code}

I am expecting the response would be in JSON format, however it is still in 
XML. I can only get JSON if I issue

{code}
curl http://${resourcemanager_host}:8088/conf?format="json;
{code}

This is not the common way how clients set content-type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-12943) Add -w -r options in dfs -test command

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-12943:


 Summary: Add -w -r options in dfs -test command
 Key: HADOOP-12943
 URL: https://issues.apache.org/jira/browse/HADOOP-12943
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Weiwei Yang


Currently the dfs -test command only supports 

  -d, -e, -f, -s, -z

options. It would be helpful if we add 

  -w, -r 

to verify permission of r/w before actual read or write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-12371) Add a force option for hdfs dfs -expunge to remove all check points

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HADOOP-12371.
--
Resolution: Later

> Add a force option for hdfs dfs -expunge to remove all check points 
> 
>
> Key: HADOOP-12371
> URL: https://issues.apache.org/jira/browse/HADOOP-12371
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: trash
>Affects Versions: 2.7.1
>    Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: namenode, trash
>
> Hadoop document has that
>  expunge
> Usage: hdfs dfs -expunge
> Empty the Trash. 
> However when I run that command. It returns following message from shell 
> output : The configured checkpoint interval is 0 minutes. Using an interval 
> of 360 minutes that is used for deletion instead. Trash is not emptied.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12374) Description of hdfs expunge command is confusing

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-12374:


 Summary: Description of hdfs expunge command is confusing
 Key: HADOOP-12374
 URL: https://issues.apache.org/jira/browse/HADOOP-12374
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation, trash
Affects Versions: 2.7.1, 2.7.0
Reporter: Weiwei Yang
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-12370) Trash documentation in hdfs design is not up-to-date

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HADOOP-12370.
--
   Resolution: Duplicate
Fix Version/s: 2.8.0

> Trash documentation in hdfs design is not up-to-date
> 
>
> Key: HADOOP-12370
> URL: https://issues.apache.org/jira/browse/HADOOP-12370
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation, trash
>Affects Versions: 2.7.1
>    Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Fix For: 2.8.0
>
>
> HADOOP-2514 removes a global root trash directory, and creates .Trash under 
> user home directories. Document needs to be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12370) Trash documentation in hdfs design is not up-to-date

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-12370:


 Summary: Trash documentation in hdfs design is not up-to-date
 Key: HADOOP-12370
 URL: https://issues.apache.org/jira/browse/HADOOP-12370
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation, trash
Affects Versions: 2.7.1
Reporter: Weiwei Yang
Assignee: Weiwei Yang


HADOOP-2514 removes a global root trash directory, and creates .Trash under 
user home directories. Document needs to be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12371) hadoop fs -expunge is not able to remove trash

[Weiwei Yang (JIRA)]

Weiwei Yang created HADOOP-12371:


 Summary: hadoop fs -expunge is not able to remove trash
 Key: HADOOP-12371
 URL: https://issues.apache.org/jira/browse/HADOOP-12371
 Project: Hadoop Common
  Issue Type: Bug
  Components: trash
Affects Versions: 2.7.1
Reporter: Weiwei Yang
Assignee: Weiwei Yang


After HADOOP-8689, hadoop fs -expunge seems not work anymore. It returns 
following message from shell output : The configured checkpoint interval is 0 
minutes. Using an interval of 360 minutes that is used for deletion instead. 
Never removing the trash files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)