Re: [ANNOUNCE] Lisheng Sun is a new Apache Hadoop Committer

[runlin zhang]

Congratulations  Lisheng !

> 在 2020年9月24日，上午2:00，Wei-Chiu Chuang  写道：
> 
> I am pleased to announce that Lisheng Sun has accepted the invitation to
> become a Hadoop committer.
> 
> Lisheng actively contributed to the project since July 2019, and he
> contributed two new features: Dead datanode detector (HDFS-13571
> ) and a new du
> implementation (HDFS-14313
> ) Lots of improvements
> including a number of short circuit read optimization
> HDFS-15161  , speeding up
> NN fsimage loading time: HDFS-13694
>  and HDFS-13693
> . Code wise, he resolved
> 57 Hadoop jiras.
> 
> Let's congratulate Lisheng for this new role!
> 
> Cheers,
> Wei-Chiu Chuang (on behalf of the Apache Hadoop PMC)


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Hui Fei is a new Apache Hadoop Committer

[runlin zhang]

Congratulations ！ 

费总！

> 在 2020年9月24日，上午2:06，Wei-Chiu Chuang  写道：
> 
> I am pleased to announce that Hui Fei has accepted the invitation to become
> a Hadoop committer.
> 
> He started contributing to the project in October 2016. Over the past 4
> years he has contributed a lot in HDFS, especially in Erasure Coding,
> Hadoop 3 upgrade, RBF and Standby Serving reads.
> 
> One of the biggest contributions is Hadoop 2->3 rolling upgrade support.
> This was a major blocker for any existing Hadoop users to adopt Hadoop 3.
> The adoption of Hadoop 3 has gone up after this. In the past the community
> discussed a lot about Hadoop 3 rolling upgrade being a must-have, but no
> one took the initiative to make it happen. I am personally very grateful
> for this.
> 
> The work on EC is impressive as well. He managed to onboard EC in
> production at scale, fixing tricky problems. Again, I am impressed and
> grateful for the contribution in EC.
> 
> In addition to code contributions, he invested a lot in the community:
> 
>> 
>>   - Apache Hadoop Community 2019 Beijing Meetup
>>   https://blogs.apache.org/hadoop/entry/hadoop-community-meetup-beijing-aug 
>> where
>>   he discussed the operational experience of RBF in production
>> 
>> 
>>   - Apache Hadoop Storage Community Sync Online
>>   
>> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit#heading=h.irqxw1iy16zo
>>  where
>>   he discussed the Hadoop 3 rolling upgrade support
>> 
>> 
> Let's congratulate Hui for this new role!
> 
> Cheers,
> Wei-Chiu Chuang (on behalf of the Apache Hadoop PMC)


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0

[runlin zhang]

+1

> 在 2019年9月11日，下午3:26，Rohith Sharma K S  写道：
> 
> Hi folks,
> 
> I have put together a release candidate (RC0) for Apache Hadoop 3.2.1.
> 
> The RC is available at:
> http://home.apache.org/~rohithsharmaks/hadoop-3.2.1-RC0/
> 
> The RC tag in git is release-3.2.1-RC0:
> https://github.com/apache/hadoop/tree/release-3.2.1-RC0
> 
> 
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1226/
> 
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> This vote will run for 7 days(5 weekdays), ending on 18th Sept at 11:59 pm
> PST.
> 
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
> 
> Thanks & Regards
> Rohith Sharma K S


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[runlin zhang]

+1 

> 在 2019年8月24日，上午10:05，Wangda Tan  写道：
> 
> Hi devs,
> 
> This is a voting thread to move Submarine source code, documentation from
> Hadoop repo to a separate Apache Git repo. Which is based on discussions of
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
> 
> Contributors who have permissions to push to Hadoop Git repository will
> have permissions to push to the new Submarine repository.
> 
> This voting thread will run for 7 days and will end at Aug 30th.
> 
> Please let me know if you have any questions.
> 
> Thanks,
> Wangda Tan


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Apache Hadoop Committer - Tao Yang

[runlin zhang]

Congrats Tao！ 

> 在 2019年7月15日，下午5:53，Weiwei Yang  写道：
> 
> Hi Dear Apache Hadoop Community
> 
> It's my pleasure to announce that Tao Yang has been elected as an Apache
> Hadoop committer, this is to recognize his contributions to Apache Hadoop
> YARN project.
> 
> Congratulations and welcome on board!
> 
> Weiwei
> (On behalf of the Apache Hadoop PMC)


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[runlin zhang]

+1 ，That will be great ！

> 在 2019年7月10日，下午3:34，Xun Liu  写道：
> 
> Hi all,
> 
> This is Xun Liu contributing to the Submarine project for deep learning
> workloads running with big data workloads together on Hadoop clusters.
> 
> There are a bunch of integrations of Submarine to other projects are
> finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next step
> of Submarine is going to integrate with more projects like Apache Arrow,
> Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> cases like model serving, notebook management, advanced training
> optimizations (like auto parameter tuning, memory cache optimizations for
> large datasets for training, etc.), and make it run on other platforms like
> Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY project
> to Apache so we can put Submarine and TonY together to the same codebase
> (Page #30.
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> ).
> 
> This expands the scope of the original Submarine project in exciting new
> ways. Toward that end, would it make sense to create a separate Submarine
> project at Apache? This can make faster adoption of Submarine, and allow
> Submarine to grow to a full-blown machine learning platform.
> 
> There will be lots of technical details to work out, but any initial
> thoughts on this?
> 
> Best Regards,
> Xun Liu


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Agenda & More Information about Hadoop Community Meetup @ Palo Alto, June 26

[runlin zhang]

It’s great，I'm really looking forward to this Meetup

> 在 2019年6月20日，上午7:49，Wangda Tan  写道：
> 
> Hi All,
> 
> I want to let you know that we have confirmed most of the agenda for Hadoop
> Community Meetup. It will be a whole day event.
> 
> Agenda & Dial-In info because see below, *please RSVP
> at https://www.meetup.com/Hadoop-Contributors/events/262055924/
> *
> 
> Huge thanks to Daniel Templeton, Wei-Chiu Chuang, Christina Vu for helping
> with organizing and logistics.
> 
> *Please help to promote meetup information on Twitter, LinkedIn, etc.
> Appreciated! *
> 
> Best,
> Wangda
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> *AM:9:00: Arrival and check-in--9:30 -
> 10:15:-Talk: Hadoop storage in cloud-native
> environmentsAbstract: Hadoop is a mature storage system but designed years
> before the cloud-native movement. Kubernetes and other cloud-native tools
> are emerging solutions for containerized environments but sometimes they
> require different approaches.In this presentation we would like to share
> our experiences to run Apache Hadoop Ozone in Kubernetes and the connection
> point to other cloud-native ecosystem elements. We will compare the
> benefits and drawbacks to use Kubernetes and Hadoop storage together and
> show our current achievements and future plans.Speaker: Marton Elek
> (Cloudera)10:20 - 11:00:--Talk: Selective Wire Encryption In
> HDFSAbstract: Wire data encryption is a key component of the Hadoop
> Distributed File System (HDFS). However, such encryption enforcement comes
> in as an all-or-nothing feature. In our use case at LinkedIn, we would like
> to selectively expose fast unencrypted access to fully managed internal
> clients, which can be trusted, while only expose encrypted access to
> clients outside of the trusted circle with higher security risks. That way
> we minimize performance overhead for trusted internal clients while still
> securing data from potential outside threats. Our design extends HDFS
> NameNode to run on multiple ports, connecting to different NameNode ports
> would end up with different levels of encryption protection. This
> protection then gets enforced for both NameNode RPC and the subsequent data
> transfers to/from DataNode. This approach comes with minimum operational
> and performance overhead.Speaker: Konstantin Shvachko (LinkedIn), Chen
> Liang (LinkedIn)11:10 - 11:55:-Talk: YuniKorn: Next Generation
> Scheduling for YARN and K8sAbstract: We will talk about our open source
> work - YuniKorn scheduler project (Y for YARN, K for K8s, uni- for Unified)
> brings long-wanted features such as hierarchical queues, fairness between
> users/jobs/queues, preemption to Kubernetes; and it brings service
> scheduling enhancements to YARN. Any improvements to this scheduler can
> benefit both Kubernetes and YARN community.Speaker: Wangda Tan
> (Cloudera)PM:12:00 - 12:55 Lunch Break (Provided by
> Cloudera)1:00 -
> 1:25---Talk: Yarn Efficiency at UberAbstract: We will present the
> work done at Uber to improve YARN cluster utilization and job SOA with
> elastic resource management, low compute workload on passive datacenter,
> preemption, larger container, etc. We will also go through YARN upgrade in
> order to adopt new features and talk about the challenges.Speaker: Aihua Xu
> (Uber), Prashant Golash (Uber)1:30 - 2:10 One more
> talk-2:20 - 4:00---BoF sessions &
> Breakout Sessions & Group discussions: Talk about items like JDK 11
> support, next releases (2.10.0, 3.3.0, etc.), Hadoop on Cloud, etc.4:00:
> Reception provided by
> Cloudera.==Join Zoom
> Meetinghttps://cloudera.zoom.us/j/116816195
> *


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Making submarine to different release model like Ozone

[runlin zhang]

+1, It is very necessary  to use Submarine on older  Hadoop.  What's more, the 
development of deep learning is too fast,  and Submarine  must  keep faster 
release iterate .

> 在 2019年2月1日，上午2:53，Wangda Tan  写道：
> 
> Hi devs,
> 
> Since we started submarine-related effort last year, we received a lot of
> feedbacks, several companies (such as Netease, China Mobile, etc.)  are
> trying to deploy Submarine to their Hadoop cluster along with big data
> workloads. Linkedin also has big interests to contribute a Submarine TonY (
> https://github.com/linkedin/TonY) runtime to allow users to use the same
> interface.
> 
> From what I can see, there're several issues of putting Submarine under
> yarn-applications directory and have same release cycle with Hadoop:
> 
> 1) We started 3.2.0 release at Sep 2018, but the release is done at Jan
> 2019. Because of non-predictable blockers and security issues, it got
> delayed a lot. We need to iterate submarine fast at this point.
> 
> 2) We also see a lot of requirements to use Submarine on older Hadoop
> releases such as 2.x. Many companies may not upgrade Hadoop to 3.x in a
> short time, but the requirement to run deep learning is urgent to them. We
> should decouple Submarine from Hadoop version.
> 
> And why we wanna to keep it within Hadoop? First, Submarine included some
> innovation parts such as enhancements of user experiences for YARN
> services/containerization support which we can add it back to Hadoop later
> to address common requirements. In addition to that, we have a big overlap
> in the community developing and using it.
> 
> There're several proposals we have went through during Ozone merge to trunk
> discussion:
> https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201803.mbox/%3ccahfhakh6_m3yldf5a2kq8+w-5fbvx5ahfgs-x1vajw8gmnz...@mail.gmail.com%3E
> 
> I propose to adopt Ozone model: which is the same master branch, different
> release cycle, and different release branch. It is a great example to show
> agile release we can do (2 Ozone releases after Oct 2018) with less
> overhead to setup CI, projects, etc.
> 
> *Links:*
> - JIRA: https://issues.apache.org/jira/browse/YARN-8135
> - Design doc
> 
> - User doc
> 
> (3.2.0
> release)
> - Blogposts, {Submarine} : Running deep learning workloads on Apache Hadoop
> ,
> (Chinese Translation: Link )
> - Talks: Strata Data Conf NY
> 
> 
> Thoughts?
> 
> Thanks,
> Wangda Tan


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org