[jira] [Created] (HADOOP-17777) Update clover-maven-plugin version from 3.3.0 to 4.4.1
Wanqiang Ji created HADOOP-1: Summary: Update clover-maven-plugin version from 3.3.0 to 4.4.1 Key: HADOOP-1 URL: https://issues.apache.org/jira/browse/HADOOP-1 Project: Hadoop Common Issue Type: Improvement Reporter: Wanqiang Ji Assignee: Wanqiang Ji Update clover-maven-plugin version to 4.4.1, some important changes are as follows: * the license key is no longer required to run Clover * the Atlassian brand and logos were removed to avoid any trademark violations * the 'org.openclover.*' groupId is used for artifacts The license key is no longer required is the key factor to do this update. So the developer can run and get the code coverage easier. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17776) The java agent args should be passed as the VM arguments
Wanqiang Ji created HADOOP-17776: Summary: The java agent args should be passed as the VM arguments Key: HADOOP-17776 URL: https://issues.apache.org/jira/browse/HADOOP-17776 Project: Hadoop Common Issue Type: Improvement Reporter: Wanqiang Ji Assignee: Wanqiang Ji Some java agents or maven plugins want works well, must pass the agent arguments as the VM arguments, such as JaCoCo etc. PS. Use the JaCoco to generate the code coverage data and put into SonarQube. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Hadoop 3.1.x EOL
+1 (non-binding) Wanqiang Ji On Thu, Jun 3, 2021 at 10:47 PM Sangjin Lee wrote: > +1 > > On Thu, Jun 3, 2021 at 7:35 AM Sean Busbey > wrote: > > > +1 > > > > > On Jun 3, 2021, at 1:14 AM, Akira Ajisaka wrote: > > > > > > Dear Hadoop developers, > > > > > > Given the feedback from the discussion thread [1], I'd like to start > > > an official vote > > > thread for the community to vote and start the 3.1 EOL process. > > > > > > What this entails: > > > > > > (1) an official announcement that no further regular Hadoop 3.1.x > > releases > > > will be made after 3.1.4. > > > (2) resolve JIRAs that specifically target 3.1.5 as won't fix. > > > > > > This vote will run for 7 days and conclude by June 10th, 16:00 JST [2]. > > > > > > Committers are eligible to cast binding votes. Non-committers are > > welcomed > > > to cast non-binding votes. > > > > > > Here is my vote, +1 > > > > > > [1] https://s.apache.org/w9ilb > > > [2] > > > https://www.timeanddate.com/worldclock/fixedtime.html?msg=4=20210610T16=248 > > > > > > Regards, > > > Akira > > > > > > - > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > > > > > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > >
[jira] [Created] (HADOOP-17246) Fix build the hadoop-build Docker image failed
Wanqiang Ji created HADOOP-17246:
Summary: Fix build the hadoop-build Docker image failed
Key: HADOOP-17246
URL: https://issues.apache.org/jira/browse/HADOOP-17246
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Wanqiang Ji
Assignee: Wanqiang Ji
When I build the docker-build image under macOS, it failed caused by:
{code:java}
Command "/usr/bin/python -u -c "import setuptools,
tokenize;__file__='/tmp/pip-build-vKHcWu/isort/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-odL0bY-record/install-record.txt
--single-version-externally-managed --compile" failed with error code 1 in
/tmp/pip-build-vKHcWu/isort/
You are using pip version 8.1.1, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/bash -o pipefail -c pip2 install configparser==4.0.2
pylint==1.9.2' returned a non-zero code: 1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17055) Remove residual code of Ozone
Wanqiang Ji created HADOOP-17055: Summary: Remove residual code of Ozone Key: HADOOP-17055 URL: https://issues.apache.org/jira/browse/HADOOP-17055 Project: Hadoop Common Issue Type: Task Reporter: Wanqiang Ji Assignee: Wanqiang Ji -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] making Ozone a separate Apache project
+1 On Wed, May 13, 2020 at 3:53 PM Elek, Marton wrote: > > > I would like to start a discussion to make a separate Apache project for > Ozone > > > > ### HISTORY [1] > > * Apache Hadoop Ozone development started on a feature branch of > Hadoop repository (HDFS-7240) > > * In the October of 2017 a discussion has been started to merge it to > the Hadoop main branch > > * After a long discussion it's merged to Hadoop trunk at the March of > 2018 > > * During the discussion of the merge, it was suggested multiple times > to create a separated project for the Ozone. But at that time: > 1). Ozone was tightly integrated with Hadoop/HDFS > 2). There was an active plan to use Block layer of Ozone (HDDS or > HDSL at that time) as the block level of HDFS > 3). The community of Ozone was a subset of the HDFS community > > * The first beta release of Ozone was just released. Seems to be a > good time before the first GA to make a decision about the future. > > > > ### WHAT HAS BEEN CHANGED > > During the last years Ozone became more and more independent both at > the community and code side. The separation has been suggested again and > again (for example by Owen [2] and Vinod [3]) > > > > From COMMUNITY point of view: > > >* Fortunately more and more new contributors are helping Ozone. > Originally the Ozone community was a subset of HDFS project. But now a > bigger and bigger part of the community is related to Ozone only. > >* It seems to be easier to _build_ the community as a separated project. > >* A new, younger project might have different practices > (communication, commiter criteria, development style) compared to old, > mature project > >* It's easier to communicate (and improve) these standards in a > separated projects with clean boundaries > >* Separated project/brand can help to increase the adoption rate and > attract more individual contributor (AFAIK it has been seen in Submarine > after a similar move) > > * Contribution process can be communicated more easily, we can make > first time contribution more easy > > > > From CODE point of view Ozone became more and more independent: > > > * Ozone has different release cycle > > * Code is already separated from Hadoop code base > (apache/hadoop-ozone.git) > > * It has separated CI (github actions) > > * Ozone uses different (more strict) coding style (zero toleration of > unit test / checkstyle errors) > > * The code itself became more and more independent from Hadoop on > Maven level. Originally it was compiled together with the in-tree latest > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC, > Configuration...) > > * It starts to use multiple version of Hadoop (on client side) > > * Volume of resolved issues are already very high on Ozone side (Ozone > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all > together in the last 2-3 months) > > > Summary: Before the first Ozone GA release, It seems to be a good time > to discuss the long-term future of Ozone. Managing it as a separated TLP > project seems to have more benefits. > > > Please let me know what your opinion is... > > Thanks a lot, > Marton > > > > > > [1]: For more details, see: > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md > > [2]: > > https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E > > [3]: > > https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
[jira] [Created] (HADOOP-16809) Stripping Submarine site from Hadoop site
Wanqiang Ji created HADOOP-16809: Summary: Stripping Submarine site from Hadoop site Key: HADOOP-16809 URL: https://issues.apache.org/jira/browse/HADOOP-16809 Project: Hadoop Common Issue Type: Task Reporter: Wanqiang Ji Assignee: Wanqiang Ji Now that Submarine is getting out of Hadoop and has its own repo, it's time to stripe the Submarine site from Hadoop site. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16719) Remove the disallowed element config within maven-checkstyle-plugin
[ https://issues.apache.org/jira/browse/HADOOP-16719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji resolved HADOOP-16719. -- Resolution: Fixed > Remove the disallowed element config within maven-checkstyle-plugin > --- > > Key: HADOOP-16719 > URL: https://issues.apache.org/jira/browse/HADOOP-16719 > Project: Hadoop Common > Issue Type: Bug > Reporter: Wanqiang Ji > Assignee: Wanqiang Ji >Priority: Major > Labels: newbie, pull-request-available > > The `format` element is disallowed in configuration within > maven-checkstyle-plugin, detailed info > [http://maven.apache.org/plugins/maven-checkstyle-plugin/check-mojo.html] > Maybe used `format` means `outputFileFormat`, `outputFileFormat`'s default > value is `xml`, so we can remove the `format` element. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16719) Remove the disallowed element config within maven-checkstyle-plugin
Wanqiang Ji created HADOOP-16719: Summary: Remove the disallowed element config within maven-checkstyle-plugin Key: HADOOP-16719 URL: https://issues.apache.org/jira/browse/HADOOP-16719 Project: Hadoop Common Issue Type: Bug Reporter: Wanqiang Ji Assignee: Wanqiang Ji The `format` element is disallowed in configuration within maven-checkstyle-plugin, detailed info [http://maven.apache.org/plugins/maven-checkstyle-plugin/check-mojo.html] Maybe used `format` means `outputFileFormat`, `outputFileFormat`'s default value is `xml`, so we can remove the `format` element. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree
+1 (non-binding) BR, Wanqiang Ji On Tue, Sep 17, 2019 at 5:48 PM Elek, Marton wrote: > > > TLDR; I propose to move Ozone related code out from Hadoop trunk and > store it in a separated *Hadoop* git repository apache/hadoop-ozone.git > > > > > When Ozone was adopted as a new Hadoop subproject it was proposed[1] to > be part of the source tree but with separated release cadence, mainly > because it had the hadoop-trunk/SNAPSHOT as compile time dependency. > > During the last Ozone releases this dependency is removed to provide > more stable releases. Instead of using the latest trunk/SNAPSHOT build > from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now). > > As we have no more strict dependency between Hadoop trunk SNAPSHOT and > Ozone trunk I propose to separate the two code base from each other with > creating a new Hadoop git repository (apache/hadoop-ozone.git): > > With moving Ozone to a separated git repository: > > * It would be easier to contribute and understand the build (as of now > we always need `-f pom.ozone.xml` as a Maven parameter) > * It would be possible to adjust build process without breaking > Hadoop/Ozone builds. > * It would be possible to use different Readme/.asf.yaml/github > template for the Hadoop Ozone and core Hadoop. (For example the current > github template [2] has a link to the contribution guideline [3]. Ozone > has an extended version [4] from this guideline with additional > information.) > * Testing would be more safe as it won't be possible to change core > Hadoop and Hadoop Ozone in the same patch. > * It would be easier to cut branches for Hadoop releases (based on the > original consensus, Ozone should be removed from all the release > branches after creating relase branches from trunk) > > > What do you think? > > Thanks, > Marton > > [1]: > > https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E > [2]: > > https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md > [3]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute > [4]: > > https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Resolved] (HADOOP-16262) Add some optional modules instructions in BUILDING.txt
[
https://issues.apache.org/jira/browse/HADOOP-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wanqiang Ji resolved HADOOP-16262.
--
Resolution: Not A Problem
> Add some optional modules instructions in BUILDING.txt
> --
>
> Key: HADOOP-16262
> URL: https://issues.apache.org/jira/browse/HADOOP-16262
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Wanqiang Ji
> Assignee: Wanqiang Ji
>Priority: Major
> Attachments: HADOOP-16262.001.patch
>
>
> Had three project modules missing in pom.xml. Such as:
> {code:xml}
> hadoop-hdds
> hadoop-ozone
> hadoop-submarine
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Moving Submarine to a separate Apache project proposal
+1 Thanks for Wangda's proposal. It is indeed amazing to see the growth and development of submarine. As the TLP will attract more developers to join. I will put more energy into it and contribute more feature. Look forward to the next change in submarine. Thanks, Wanqiang Ji On Wed, Sep 4, 2019 at 3:09 PM Bibin Chundatt wrote: > +1 > Thank you for the proposal. > I am interested in project. Please include me as well in the project. > > Thanks, > Bibin > > On Tue, Sep 3, 2019 at 6:41 PM Ayush Saxena > wrote: > > > +1 > > Thanx for the proposal. > > > > I would even like to participate in the project. > > Please add me as well. > > > > -Ayush > > > > > > > On 03-Sep-2019, at 6:00 PM, Vinayakumar B > > wrote: > > > > > > +1 > > > > > > Thanks for the proposal. > > > Its very interesting project and looks very promising one by looking at > > the > > > participations from various companies and the speed of development. > > > > > > I would also like to participate in the project. > > > Please add me as well. > > > > > > Thanks, > > > -Vinay > > > > > > On Tue, 3 Sep 2019, 12:38 pm Rakesh Radhakrishnan, > > > > wrote: > > > > > >> +1, Thanks for the proposal. > > >> > > >> I am interested to participate in this project. Please include me as > > well > > >> in the project. > > >> > > >> Thanks, > > >> Rakesh > > >> > > >> On Tue, Sep 3, 2019 at 11:59 AM zhankun tang > > >> wrote: > > >> > > >>> +1 > > >>> > > >>> Thanks for Wangda's proposal. > > >>> > > >>> The submarine project is born within Hadoop, but not limited to > Hadoop. > > >> It > > >>> began with a trainer on YARN but it quickly realized that only a > > trainer > > >> is > > >>> not enough to meet the AI platform requirements. But now there's no > > >>> user-friendly open-source solution covers the whole AI pipeline like > > data > > >>> engineering, training, and serving. And the underlying data > > >> infrastructure > > >>> itself is also evolving, for instance, many people love k8s. Not > > >> mentioning > > >>> there're many AI domain problems in this area to be solved. > > >>> It's almost for sure that building such an ML platform would utilize > > >>> various other open-source components taking ML into consideration > > >>> initially. > > >>> > > >>> I see submarine grows rapidly towards an enterprise-grade ML platform > > >> which > > >>> could potentially enable AI ability for data engineer and scientist. > > This > > >>> is an exciting thing for both the community and the industry. > > >>> > > >>> BR, > > >>> Zhankun > > >>> > > >>> > > >>>> On Tue, 3 Sep 2019 at 13:34, Xun Liu wrote: > > >>>> > > >>>> +1 > > >>>> > > >>>> Hello everyone, I am a member of the submarine development team. > > >>>> I have been contributing to submarine for more than a year. > > >>>> I have seen the progress of submarine development very fast. > > >>>> In more than a year, there are 9 long-term developers of different > > >>>> companies. Contributing, > > >>>> submarine cumulative code has more than 200,000 lines of code, is > > >> growing > > >>>> very fast, > > >>>> and is used in the production environment of multiple companies. > > >>>> > > >>>> In the submarine development group, there are 5 PMCs and 7committer > > >>> members > > >>>> from Hadoop, spark, zeppelin projects. > > >>>> They are very familiar with the development process and > specifications > > >> of > > >>>> the apache community, > > >>>> and can well grasp the project development progress and project > > >> quality. > > >>>> So I recommend submarine to be a TLP project directly. > > >>>> > > >>>> We will continue to contribute to the submarine project. :-) > > >>>> > > >>>> Xun Liu > > >>>>
Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo
+1 (non-binding) Thanks, Wanqiang Ji On Sat, Aug 24, 2019 at 10:06 AM Wangda Tan wrote: > Hi devs, > > This is a voting thread to move Submarine source code, documentation from > Hadoop repo to a separate Apache Git repo. Which is based on discussions of > > https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E > > Contributors who have permissions to push to Hadoop Git repository will > have permissions to push to the new Submarine repository. > > This voting thread will run for 7 days and will end at Aug 30th. > > Please let me know if you have any questions. > > Thanks, > Wangda Tan >
Re: Thoughts about moving submarine to a separate git repo?
+1 (non-binding) We need a separate repository to keep it quickly development. On Tue, Aug 20, 2019 at 3:00 PM zac yuan wrote: > +1 (non-binding) > Looking forward to having a separate repo and making the development more > conveniently and rapidly. > > Thanks > Zac Zhou > > kevin su 于2019年8月20日周二 下午2:45写道： > > > +1, Agree > > using separate repo can better develop and maintain submarine > > > > Xun Liu 於 2019年8月20日 週二 下午2:05寫道： > > > > > Hi Sunil > > > > > > Thank you for your advice and attention. > > > > > > The Hadoop submarine development team needs a separate repo for several > > > reasons: > > > 1. We have developed a separate submarine website to introduce the > > > functions and features of submarine. The common practice is to put it > in > > > the gh-pages branch of the repo repository, automatically generate HTML > > > code, and update the website. > > > > > > 2. We also want to associate github's repo with apache jira. It can > > > automatically synchronize the status of the issue on github, and the > > > comments will be automatically synchronized to Apache's jira. There are > > > some other open source projects managed like this. Very good > experience. > > > Making these settings requires some administrative privileges with > repo. > > > > > > 3. We also need to create different branches for different versions for > > > version management. > > > > > > So, we need a separate repo repository. > > > > > > thank you for your support. :-) > > > > > > > > > Best Regards > > > Xun Liu > > > > > > > > > > On Aug 16, 2019, at 5:25 PM, dashuiguailu...@gmail.com wrote: > > > > > > > > +1 ，Agree that independent development of submarine can better adapt > to > > > the development of machine learning > > > > > > > > > > > > > > > > dashuiguailu...@gmail.com > > > > > > > > From: Xun Liu > > > > Date: 2019-08-16 12:43 > > > > To: common-dev; yarn-dev; hdfs-dev; submarine-dev > > > > Subject: Thoughts about moving submarine to a separate git repo? > > > > Dear Submarine developers, > > > > > > > > My name is Xun Liu, I am a member of the Hadoop submarine development > > > team. > > > > I'm one of the major contributor of Submarine since June 2018. > > > > > > > > I want to hear your thoughts about creating a separate GitHub repo > > under > > > > Apache to do submarine development. This is an independent effort of > > > > Submarine spin-off from the Hadoop project [ > > > > > > > > > > https://lists.apache.org/thread.html/3fab657f905d081b536d9081dc404f7fd20c80eb824c857bc8e16e3b@ > > > ]. > > > > However, once the spin-off is approved, this effort can benefit the > > > > follow-up processes as well. > > > > > > > > Submarine dev community has a total of 8 developers and submits an > > > average > > > > of 4 to 5 PR per day. > > > > But there are a limited number of Hadoop committer actively help > review > > > and > > > > merge patches, which causes development progress delays. > > > > > > > > So we created an external GitHub repo [ > > > > https://github.com/hadoopsubmarine/submarine] and moved all the code > > for > > > > the Hadoop submarine project into the external Github repo. > > > > In this way, everyone can review the code for each other, and now the > > > > development progress of Hadoop submarine is very fast. > > > > > > > > Also, now Submarine has little dependency on Hadoop, we want to have > a > > > > separate CI/CD pipeline to release and test submarine instead of > every > > > time > > > > build whole Hadoop. Putting Submarine under Hadoop will introduce > > > > unnecessary dependencies to Hadoop's top-level pom.xml. > > > > > > > > Our development process still complies with the development rules of > > the > > > > Hadoop community: first, create a ticket in the submarine JIRA, and > > then > > > > develop, in the external GitHub repo repository, the title of each PR > > > will > > > > be accompanied by the JIRA ID number. > > > > > > > > Once the Apache Github repo is created, we going to move all external > > > > commits to the new Apache Github repo. > > > > > > > > Any suggestions are welcome! > > > > > > > > Best Regards > > > > Xun Liu > > > > > > > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > >
Re: YARN+MR APAC Sync Up Meeting Minutes 07/25
Sorry for the previous mail with wrong format, I had fixed it. — Participants: Cloudera, Tencent, Taiwan University, Xiaomi, ByteDance, JD, Didi, Alibaba ByteDance: Version: 2.6.0 1000+ patches Fair scheduler Improvements: like CS global scheduling Scale: 40,000 hosts last year!! Flink + Spark + MR + other computations Not server side federation: client side to route apps to clusters Largest cluster 10,000 nodes Alibaba: Version: 3.1.0 (upgraded 1.5 year before) CS + Global scheduling Overcommitment with opportunistic containers -> increase utilization Scheduler activities (share more details in upcoming meetup) Xiaomi: 2.6.0 -> 3.1.0 (upgrading in progress) Migrate FS to CS Investigating federation TODO: Open umbrella for migration Hadoop Meetup 8/10 discussion: Location TBD, most likely will be a one day event Setting up a review-board, we are glad to have 1 or 2 contract-point from each company in this board, please nominate to junping...@apache.org Call for more talks!!! Wanqiang Ji Wanqiang Ji 于2019年7月25日 周四14:49写道： > July 25, 2019 > > Participants: Cloudera, Tencent, Taiwan University, Xiaomi, ByteDance, JD, > Didi, Alibaba > > ByteDance > >- > >Version: 2.6.0 >- > >1000+ patches >- > >Fair scheduler >- > > Improvements: like CS global scheduling > - > >Scale: 40,000 hosts last year!! >- > >Flink + Spark + MR + other computations >- > >Not server side federation: client side to route apps to clusters >- > >Largest cluster 10,000 nodes > > Alibaba > >- > >Version: 3.1.0 (upgraded 1.5 year before) >- > >CS + Global scheduling >- > >Overcommitment with opportunistic containers -> increase utilization >- > >Scheduler activities (share more details in upcoming meetup) > > Xiaomi > >- > >2.6.0 -> 3.1.0 (upgrading in progress) >- > >Migrate FS to CS >- > >Investigating federation >- > >TODO: Open umbrella for migration > > Hadoop Meetup 8/10 discussion > >- > >Location TBD, most likely will be a one day event > - > >Setting up a review-board, we are glad to have 1 or 2 contract-point >from each company in this board, please nominate to >junping...@apache.org >- > >Call for more talks!!! > > > > FYI > Wanqiang Ji > > 俊平堵 于2019年7月25日 周四11:53写道： > >> Hi Folks, >> >> Kindly remind that we have YARN+MR APAC sync today, and you are >> welcome to join: >> >> >> Time and Date：07/25 1:00 pm (CST Time) >> >> Zoom link：Zoom | https://cloudera.zoom.us/j/880548968 >> >> Summary： >> >> https://docs.google.com/document/d/1GY55sXrekVd-aDyRY7uzaX0hMDPyh3T-AL1kUY2TI5M >> >> >> Thanks, >> >> >> Junping >> >> >> >> Wangda Tan 于2019年6月28日周五 上午2:57写道： >> >> > Hi folks, >> > >> > Here's the Hadoop Community Sync Up proposal/schedule: >> > >> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#heading=h.xh4zfwj8ppmn >> > >> > And here's calendar file: >> > >> > >> > >> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics >> > >> > We gave it a try this week for YARN+MR and Submarine sync, feedbacks >> from >> > participants seems pretty good, lots of new information shared during >> sync >> > up, and companies are using/developing Hadoop can better know each >> other. >> > >> > Next week there're 4 community sync-ups (Two Submarine for different >> > timezones, one YARN+MR, one storage), please join to whichever you're >> > interested: >> > >> > [image: image.png] >> > >> > Zoom info and notes can be found in the Google calendar invitation. >> > >> > Thanks, >> > Wangda >> > >> >
YARN+MR APAC Sync Up Meeting Minutes 07/25
July 25, 2019 Participants: Cloudera, Tencent, Taiwan University, Xiaomi, ByteDance, JD, Didi, Alibaba ByteDance - Version: 2.6.0 - 1000+ patches - Fair scheduler - Improvements: like CS global scheduling - Scale: 40,000 hosts last year!! - Flink + Spark + MR + other computations - Not server side federation: client side to route apps to clusters - Largest cluster 10,000 nodes Alibaba - Version: 3.1.0 (upgraded 1.5 year before) - CS + Global scheduling - Overcommitment with opportunistic containers -> increase utilization - Scheduler activities (share more details in upcoming meetup) Xiaomi - 2.6.0 -> 3.1.0 (upgrading in progress) - Migrate FS to CS - Investigating federation - TODO: Open umbrella for migration Hadoop Meetup 8/10 discussion - Location TBD, most likely will be a one day event - Setting up a review-board, we are glad to have 1 or 2 contract-point from each company in this board, please nominate to junping...@apache.org - Call for more talks!!! FYI Wanqiang Ji 俊平堵 于2019年7月25日 周四11:53写道： > Hi Folks, > > Kindly remind that we have YARN+MR APAC sync today, and you are > welcome to join: > > > Time and Date：07/25 1:00 pm (CST Time) > > Zoom link：Zoom | https://cloudera.zoom.us/j/880548968 > > Summary： > > https://docs.google.com/document/d/1GY55sXrekVd-aDyRY7uzaX0hMDPyh3T-AL1kUY2TI5M > > > Thanks, > > > Junping > > > > Wangda Tan 于2019年6月28日周五 上午2:57写道： > > > Hi folks, > > > > Here's the Hadoop Community Sync Up proposal/schedule: > > > https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#heading=h.xh4zfwj8ppmn > > > > And here's calendar file: > > > > > > > https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics > > > > We gave it a try this week for YARN+MR and Submarine sync, feedbacks from > > participants seems pretty good, lots of new information shared during > sync > > up, and companies are using/developing Hadoop can better know each other. > > > > Next week there're 4 community sync-ups (Two Submarine for different > > timezones, one YARN+MR, one storage), please join to whichever you're > > interested: > > > > [image: image.png] > > > > Zoom info and notes can be found in the Google calendar invitation. > > > > Thanks, > > Wangda > > >
Re: [ANNOUNCE] New Apache Hadoop Committer - Ayush Saxena
Congratulations Ayush ! On Sat, Jul 20, 2019 at 4:48 AM CR Hota wrote: > Congratulations Ayush ! > > On Mon, Jul 15, 2019, 5:56 PM Takanobu Asanuma > wrote: > > > Congrats Ayush! > > > > Regards, > > - Takanobu > > > > > > From: Dinesh Chitlangia > > Sent: Monday, July 15, 2019 23:18 > > To: lqjacklee > > Cc: HarshaKiran Reddy Boreddy; Vinayakumar B; ayushsax...@apache.org; > > Hdfs-dev; Hadoop Common > > Subject: Re: [ANNOUNCE] New Apache Hadoop Committer - Ayush Saxena > > > > Congratulations Ayush! > > > > Cheers, > > Dinesh > > > > > > > > > > On Mon, Jul 15, 2019 at 8:48 AM lqjacklee wrote: > > > > > congratulations. > > > > > > On Mon, Jul 15, 2019 at 6:09 PM HarshaKiran Reddy Boreddy < > > > bharsh...@gmail.com> wrote: > > > > > > > Congratulations Ayush!!! > > > > > > > > > > > > -- Harsha > > > > > > > > On Mon, Jul 15, 2019, 2:15 PM Vinayakumar B > > > > > wrote: > > > > > > > > > In bcc: general@, please bcc: (and not cc:) general@ if you want > to > > > > > include > > > > > > > > > > It's my pleasure to announce that Ayush Saxena has been elected as > > > > > committer > > > > > on the Apache Hadoop project recognising his continued > contributions > > to > > > > the > > > > > project. > > > > > > > > > > Please join me in congratulating him. > > > > > > > > > > Hearty Congratulations & Welcome aboard Ayush! > > > > > > > > > > > > > > > Vinayakumar B > > > > > (On behalf of the Hadoop PMC) > > > > > > > > > > > > > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > >
Re: Aug Hadoop Community Meetup in China
Thanks for organize this event! I always looking forward to meet folks in the coming meet up. On Thu, Jul 18, 2019 at 4:00 PM Weiwei Yang wrote: > Hi Junping > > Thanks for organizing this event ! > Just finished the survey, looking forward to meet folks in the coming meet > up. > > Thanks > Weiwei > On Jul 18, 2019, 3:28 PM +0800, 俊平堵 , wrote: > > Hi, all! > > > > I am glad to let you know that we are organizing > > Hadoop Contributors Meetup in China on Aug. > > > > > > This could be the first time hadoop community meetup in China and many > > attendees are expected to come from big data pioneers, such as: Cloudera, > > Tencent, Alibaba, Xiaomi, Didi, JD, Meituan, Toutiao, Sina, etc. > > > > > > We're still working out the details, such as dates, contents and > locations. > > Here is a quick survey: https://www.surveymonkey.com/r/Y99RT3W where you > > can vote your prefer dates and locations if you would like to attend - > the > > survey will end in July. 21. 12PM China Standard Time, and result will go > > public in next day. > > > > > > Also, please feel free to reach out to me if you have a topic to propose > > for the meetup. Will send out an update later with more details when I > get > > more to share. Thanks! > > > > > > Cheers, > > > > > > Junping >
Re: [ANNOUNCE] New Apache Hadoop Committer - Tao Yang
Congratulations Tao! On Tue, Jul 16, 2019 at 10:14 PM Eric Payne wrote: > Congratulations Tao! Well deserved! > > On Monday, July 15, 2019, 4:54:10 AM CDT, Weiwei Yang > wrote: > > Hi Dear Apache Hadoop Community > > It's my pleasure to announce that Tao Yang has been elected as an Apache > Hadoop committer, this is to recognize his contributions to Apache Hadoop > YARN project. > > Congratulations and welcome on board! > > Weiwei > (On behalf of the Apache Hadoop PMC) >
Re: Any thoughts making Submarine a separate Apache project?
+1 This is a fantastic recommendation. I can see the community grows fast and good collaborative, submarine can be an independent project at now, thanks for all contributors. FYI, Wanqiang Ji On Wed, Jul 10, 2019 at 3:34 PM Xun Liu wrote: > Hi all, > > This is Xun Liu contributing to the Submarine project for deep learning > workloads running with big data workloads together on Hadoop clusters. > > There are a bunch of integrations of Submarine to other projects are > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step > of Submarine is going to integrate with more projects like Apache Arrow, > Redis, MLflow, etc. & be able to handle end-to-end machine learning use > cases like model serving, notebook management, advanced training > optimizations (like auto parameter tuning, memory cache optimizations for > large datasets for training, etc.), and make it run on other platforms like > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project > to Apache so we can put Submarine and TonY together to the same codebase > (Page #30. > > https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30 > ). > > This expands the scope of the original Submarine project in exciting new > ways. Toward that end, would it make sense to create a separate Submarine > project at Apache? This can make faster adoption of Submarine, and allow > Submarine to grow to a full-blown machine learning platform. > > There will be lots of technical details to work out, but any initial > thoughts on this? > > Best Regards, > Xun Liu >
Re: new committer: Gabor Bota
Congratulations, Gabor. On Tue, Jul 2, 2019 at 7:36 AM Sean Mackrory wrote: > The Project Management Committee (PMC) for Apache Hadoop > has invited Gabor Bota to become a committer and we are pleased > to announce that he has accepted. > > Gabor has been working on the S3A file-system, especially on > the robustness and completeness of S3Guard to help deal with > inconsistency in object storage. I'm excited to see his work > with the community continue! > > Being a committer enables easier contribution to the > project since there is no need to go via the patch > submission process. This should enable better productivity. >
Re: [ANNOUNCE] Aaron Fabbri as Hadoop PMC
Congratulations! On Tue, Jun 18, 2019 at 8:29 AM Da Zhou wrote: > Congratulations! > > Regards, > Da > > On Mon, Jun 17, 2019 at 5:14 PM Ajay Kumar .invalid> > wrote: > > > Congrats Aaron!! > > > > On Mon, Jun 17, 2019 at 4:00 PM Daniel Templeton > > wrote: > > > > > I am very pleased to announce that Aaron Fabbri has now been added to > > > the Hadoop PMC. Welcome aboard, Aaron, and Congratulations! > > > > > > Daniel > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > >
[jira] [Created] (HADOOP-16334) Fix yetus-wrapper not working when HADOOP_YETUS_VERSION greater or equal than 0.9.0
Wanqiang Ji created HADOOP-16334: Summary: Fix yetus-wrapper not working when HADOOP_YETUS_VERSION greater or equal than 0.9.0 Key: HADOOP-16334 URL: https://issues.apache.org/jira/browse/HADOOP-16334 Project: Hadoop Common Issue Type: Bug Components: yetus Reporter: Wanqiang Ji Assignee: Wanqiang Ji -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16331) Asflicense test warnings
Wanqiang Ji created HADOOP-16331:
Summary: Asflicense test warnings
Key: HADOOP-16331
URL: https://issues.apache.org/jira/browse/HADOOP-16331
Project: Hadoop Common
Issue Type: Bug
Reporter: Wanqiang Ji
There are some asflicense warnings shown as follows.
{noformat}
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header:
!? /testptch/hadoop/hadoop-assemblies/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml
!? /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/pom.xml
!? /testptch/hadoop/hadoop-tools/hadoop-resourceestimator/pom.xml{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16262) Missing some modules in pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wanqiang Ji reopened HADOOP-16262:
--
> Missing some modules in pom.xml
> ---
>
> Key: HADOOP-16262
> URL: https://issues.apache.org/jira/browse/HADOOP-16262
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Wanqiang Ji
> Assignee: Wanqiang Ji
>Priority: Major
> Attachments: HADOOP-16262.001.patch
>
>
> Had three project modules missing in pom.xml. Such as:
> {code:xml}
> hadoop-hdds
> hadoop-ozone
> hadoop-submarine
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16262) Missing some modules in pom.xml
Wanqiang Ji created HADOOP-16262:
Summary: Missing some modules in pom.xml
Key: HADOOP-16262
URL: https://issues.apache.org/jira/browse/HADOOP-16262
Project: Hadoop Common
Issue Type: Bug
Reporter: Wanqiang Ji
Assignee: Wanqiang Ji
Had three project modules missing in pom.xml. Such as:
{code:xml}
hadoop-hdds
hadoop-ozone
hadoop-submarine
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: New committer: Botong Huang
Congrats !!!
On Thu, Nov 22, 2018 at 9:48 AM Sree V
wrote:
> Congratulations, Botong.We look forward to see many more contributions
> from you.
> Thank you.
> /Sree
>
>
> On Wednesday, November 21, 2018, 5:37:47 PM PST, Xun Liu <
> neliu...@163.com> wrote:
>
> Congrats !!!
>
> > 在 2018年11月22日，上午9:29，Wangda Tan 写道：
> >
> > Congrats!
> >
> > Best,
> > Wangda
> >
> > On Wed, Nov 21, 2018 at 4:23 PM Srinivas Reddy <
> srinivas96all...@gmail.com>
> > wrote:
> >
> >> Congratulations Botong !!!
> >>
> >> -
> >> Srinivas
> >>
> >> - Typed on tiny keys. pls ignore typos.{mobile app}
> >>
> >> On Thu 22 Nov, 2018, 03:27 Chang Qiang Cao wrote:
> >>
> >>> Congrats Botong!
> >>>
> >>> On Wed, Nov 21, 2018 at 2:15 PM Subru Krishnan
> wrote:
> >>>
> The Project Management Committee (PMC) for Apache Hadoophas invited
> Botong Huang to become a committer and we are pleased to announce that
> he has accepted.
> Being a committer enables easier contribution to theproject since
> there is no need to go via the patchsubmission process. This should
> enable better productivity.Being a PMC member enables assistance with
> the managementand to guide the direction of the project.
>
> Congrats and welcome aboard.
>
> -Subru
>
> >>>
> >>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
[jira] [Created] (HADOOP-15907) Add YARN modules to the main modules section in BUILDING.txt
Wanqiang Ji created HADOOP-15907: Summary: Add YARN modules to the main modules section in BUILDING.txt Key: HADOOP-15907 URL: https://issues.apache.org/jira/browse/HADOOP-15907 Project: Hadoop Common Issue Type: Improvement Reporter: Wanqiang Ji In the Maven main modules section, I found the missing main module YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
