Hi - Very interesting proposal. +1.
Please note that two of your nominated mentors will need to ask to join the IPMC by request at private@i.a.o Regards, Dave > On Dec 11, 2019, at 11:50 AM, Weiwei Yang <w...@apache.org> wrote: > > Greetings folks: > > Please consider the following proposal, which is also on the wiki > <https://cwiki.apache.org/confluence/display/INCUBATOR/YuniKornProposal> > [1]. > I look forward to hearing feedback from you. > > > *YuniKorn - An Unified Resource Scheduler* > > 1. Abstract > YuniKorn is a standalone resource scheduler responsible for scheduling > batch jobs and long-running services on large scale distributed systems > running in on-premises environments as well as different public clouds. > > 2. Proposal > YuniKorn ['ju:nikɔ:n] is a unified resource scheduler aiming to achieve > fine-grained resource sharing for various workloads efficiently on a large > scale, multi-tenant and cloud-native environments. YuniKorn brings a > unified, cross-platform scheduling experience for mixed workloads, with > support for but not limited to, Apache™ Hadoop® YARN and Kubernetes. > > Currently, YuniKorn is an open-source project with Apache 2.0 license. The > source code is hosted as a git-repo under the github.com/cloudera domain. > We would like to share it with the ASF and expand the community to a wider > range of users and contributors. > > 2.1 Background > Enterprise users run their workloads on different platforms such as Apache™ > Hadoop® YARN and Kubernetes. They need to work with different resource > schedulers in order to plan their workloads to run on these platforms > efficiently. The scheduler implementations are fragmented, and not > optimized to balance existing use-cases like batch workloads along with new > needs such as cloud-native architecture, autoscaling, etc. We need a single > resource planning/management framework to manage resources on different > platforms using the same semantics, in order to address all the important > resource management requirements. > > 2.2 Rationale > There is no solution that exists now to address the needs of having a > unified resource scheduling experiences across platforms. That makes it > difficult to manage workloads running on different environments, from > on-premise to Cloud. YuniKorn aims to satisfy these needs. YuniKorn is > designed around the following principles: > > 1) Support different environments > As the compute platforms are evolving quickly, there are more and more > challenges appear in on-prem, cloud or hybrid environments. YuniKorn aims > to bring unified scheduling experiences across multiple environments with > enhanced scheduling capabilities. > > 2) Support extensive types of workloads > To improve the efficiency of the computing platform, a key idea is to run > different types of applications, like long-running services and batch jobs, > on shared resources. YuniKorn is an effort to address all the scheduling > features needed for such mixed workload environments. > > 3) Benefit both big-data and cloud-native communities > A resource scheduler needs to be capable of supporting mixed workloads, > both batch, and long-running services. This is the key to improving cluster > utilization, and to reduce the complexity of dev-ops. By creating a common > scheduler that is decoupled from the container platforms underneath, it can > benefit both Apache™ Hadoop® YARN and the Kubernetes communities. > > 2.3 Initial Goals > Initial goals are: > - Move the existing codebase, documentation to Apache hosted repo > - Set up mailing lists, website, CI/CD pipeline under Apache infrastructure > - Setup JIRA for issue tracking > - Incremental development and releases according to Apache guidelines > - Expand the community and bring more diversified contributors/users to > the community > > 2.4 Current Status > > 2.4.1 Meritocracy > Many of the initial developers of YuniKorn are already Apache committers > and PMC members from other Apache projects, such as Apache Hadoop and > Apache Submarine. Many of us have worked in the Apache Hadoop community for > years and know the Apache way well. We believe strongly in meritocracy in > electing committers and PMC members. We believe that contributions can come > in forms other than just code: for example, one of our initial proposed > committers has contributed solely in the area of project documentation. We > will encourage contributions and participation of all types, and ensure > that contributors are appropriately recognized. > > 2.4.2 Community > YuniKorn is a relatively new open source project, Cloudera is the original > development sponsor for YuniKorn. From the beginning of the project itself, > we had clearly aimed to have this as an open-source project, so we started > to build the community from the very early stages. We received a lot of > feedback and valuable suggestions from other community members while the > project was hosted as an open-source project on GitHub. This feedback has > greatly influenced some of our designs. For e.g, developers from Alibaba > had been involved in the very early stage of development, lots of effort > related to performance/throughput enhancement were contributed by them. > Lots of other organizations further showed their interest to join the > community once we started talking about it in meetups, conferences, etc. > > 2.4.3 Core developers > The project was initiated in Cloudera and so the core developers are > heavily from this organization. Tao Yang from Alibaba joined the > development at a very early stage. The core developers of YuniKorn are > (listed in alphabetical order): > > - Akhil PB (Cloudera) > - Sunil Govindan (Cloudera) > - Tao Yang (Alibaba) > - Vinod Vavilapalli (Cloudera) > - Wangda Tan (Cloudera) > - Weiwei Yang (Cloudera) > - Wilfred Spiegelenburg (Cloudera) > > Given the origin history, the core development team so far has not been > very diverse, but we’ve been attempting to grow that diversity. We have > every hope to continue building a diverse and sustainable community if the > project gets accepted into Apache. > > 2.4.4 Alignment > The motivation of YuniKorn project is to resolve common resource scheduling > problems for various workloads, on large scale distributed systems. Apache > is home to one of these systems in the form of Apache Hadoop YARN. Many of > thee workloads that we expect to leverage YuniKorn are computing engines > like Apache Spark, Apache Flink whether they run on top of YARN or on > Kubernetes. > > 2.5 Known Risks > > 2.5.1 Orphaned products > The core developers of YuniKorn project from different companies plan to > work full time on this project. Currently, the initial team intends to > continue the investments on the YuniKorn project, it will be integrated > into the solutions to the customers. Several other organizations (like > Alibaba) have also started to evaluate the project, and plan to adopt it in > their production environments. We anticipate the adoption will be further > improved once it becomes an Apache project. > > We have also got support from core-platform developers and Apache > committers who are interested in contributing to YuniKorn project from > different companies like Microsoft, Nvidia, Tencent, etc. We’re expecting > to see more contributions from these committers and usage by their internal > platforms. So overall, the risk of YuniKorn being an orphaned project is > low. > > 2.5.2 Inexperience with Open source > Most of the core developers in YuniKorn project are experienced open source > veterans, several developers are Apache committers and PMC members of other > projects, such as Apache™ Hadoop®. And the development style is already > very likely the Apache way > - We have open community meetings to discuss designs, problems and roadmaps > - We publish all patches and issue related discussions on Github > - We enforce the code review and log all comments in GitHub issues > > 2.5.3 Length of Incubation > We started the work 10 months ago, so far the groundwork for YuniKorn is > done and the initial version can work with K8s seamlessly. Based on the > initial contributors’ experience in ASF projects, we don’t expect that > there will be huge gaps before YuniKorn can graduate with regarding to > ASF’s policies on software and releases. The goal is to grow the community > quickly and increase the user base within a few months while making > releases that adhere to the ASF standards. When it reaches a reasonable > size of adoption and a strong community with a good number of > committers/PMC members, we can prompt the graduation. We expect the length > of incubation to be approximately 6 to 12 months. > > 2.5.4 Homogenous Development > The initial proposed list of committers and contributors includes > developers from several institutions and industry participants. The > developers are also from different regions like U.S, Australia, India, and > the development team leverages slack, community mailing list, weekly > community calls to collaborate efficiently. > > 2.5.5 Reliance on Salaried Developers > Clearly, Cloudera has contributed most of the initial development through > salaried developers. But since the very beginning, YuniKorn is built as a > community effort project. We have people from other organizations that are > already collaborating with us on Github. This includes both at the source > code level, as well as participating in designs and providing feedback > through community calls. We expect our reliance on salaried developers to > decrease drastically during the incubation process itself. > > 2.5.6 Relationship to Other Apache Products > > YuniKorn is very closely related to other Big-Data projects in Apache, such > as Hadoop YARN, Spark, Hive, Flink, etc. > > YuniKorn’s core idea is to support both long-running and batch workloads > like Spark, Hive, Flink, etc, and provide a consistent, unified way to > manage and schedule resources for Big Data workloads across resource > managers like Apache™ Hadoop® YARN / Kubernetes and on-premise and cloud > environments. > > Many of the core ideas for YuniKorn come from the experience of the initial > team building Apache Hadoop YARN’s schedulers - Capacity Scheduler and Fair > Scheduler. > > 2.5.7 An Excessive Fascination with the Apache Brand > Many of the initial developers in YuniKorn project are already experienced > Apache committers, PMC members. We understand the value of the Apache way, > and how to operate the project development on a day to day basis. The > reason for proposing YuniKorn as an Apache project is to build a healthy > community, increasing adoption & the size of the community and end-users, > because we believe the only way to build a highly valuable infrastructure > layer software is to have wide adoption and cater to common use cases. > > 2.6 Documentation > Project summary: > https://github.com/cloudera/yunikorn-core/blob/master/README.md > User guides: > https://github.com/cloudera/yunikorn-core/blob/master/docs/user-guide.md > Developer guides: > https://github.com/cloudera/yunikorn-core/blob/master/docs/developer-guide.md > Roadmap: > https://github.com/cloudera/yunikorn-core/blob/master/docs/roadmap.md > > 2.7 Initial Source > Currently, YuniKorn source code is hosted in several GitHub repositories > - Scheduler interface: > https://github.com/cloudera/yunikorn-scheduler-interface > - Scheduler core: https://github.com/cloudera/yunikorn-core > - K8s Shim: https://github.com/cloudera/yunikorn-k8shim > - Scheduler Web UI: https://github.com/cloudera/yunikorn-web > > 2.8 Source and Intellectual Property Submission Plan > > 2.8.1External Dependencies > External dependencies are listed in below table > - k8s.io/api, K8s API, Apache License 2.0 > - k8s.io/apimachinery, K8s API, Apache License 2.0 > - k8s.io/client-go, K8s client library, Apache License 2.0 > - github.com/looplab/fsm, Go state machine library, MIT License > - github.com/satori/go.uuid, Go UUID library, MIT License > - github.com/uber-go/zap, Go logging library, MIT License > - github.com/golang/protobuf, Go protobuf library, BSD 3-Clause License > - github.com/gorilla/mux, Go network library, BSD 3-Clause License > - google.golang.org/grpc, Go RPC library, Apache License 2.0 > - gopkg.in/yaml.v2, Go YAML library, Apache License 2.0 > - github.com/prometheus/client_golang, Prometheus Client Library, Apache > License 2.0 > - Angular v6.1.x, Angular UI Framework Libraries, MIT License > - TypeScript, TypeScript Language Compiler, Apache License 2.0 > - Chart.js, JavaScript Charting Library, MIT License > - Moment.js, JavaScript Date & Time Library, MIT License > > Build and test only: > - gotest.tools, Test library, Apache License 2.0 > - github.com/stretchr/testify, Test library, MIT License > - Karma, Unit test library, MIT License > - Protactor, End2End test library, MIT License > - Json-server, Test server, MIT License > - Yarn, Dependency manager, BSD 2-Clause License > > 2.8.2 Cryptography > YuniKorn does not currently include any cryptography-related code. > > 2.9 Required Resources > > 2.9.1 Mailing lists: > - priv...@yunikorn.incubator.apache.org (PMC list) > - comm...@yunikorn.incubator.apache.org (git push emails) > - iss...@yunikorn.incubator.apache.org (JIRA issue feed) > - d...@yunikorn.incubator.apache.org (Dev discussion) > - u...@yunikorn.incubator.apache.org (User questions) > > 2.9.2 Git Repositories > Git is the preferred source control system > - git://git.apache.org/yunikorn-* (We have multiple git repositories) > > 2.9.3 Issue Tracking > JIRA YuniKorn (*YUNIKORN-*) > > 2.9.4 Other Resources > We have published a series of demo videos on the Youtube channel: > https://www.youtube.com/channel/UCDSJ2z-lEZcjdK27tTj_hGw > > 2.10 Initial Committers and Affinities > Initial committers and affinities are listed as below: > - Akhil PB (a...@cloudera.com) (Cloudera) > - Sunil Govindan (sun...@apache.org) (Cloudera) > - Vinod Kumar Vavilapalli (vino...@apache.org) (Cloudera) > - Wangda Tan (wan...@apache.org) (Cloudera) > - Weiwei Yang (w...@apache.org) (Cloudera) > - Wilfred Spiegelenburg (wspiegelenb...@cloudera.com) (Cloudera) > - Carlo Curino (cur...@apache.org) (Microsoft) > - Subramaniam Krishnan (su...@apache.org) (Microsoft) > - Arun Suresh (asur...@apache.org) (Microsoft) > - Konstantinos Karanasos (kkarana...@apache.org) (Microsoft) > - Jonathan Hung (jh...@apache.org) (LinkedIn) > - DB Tsai (dbt...@apache.org) (Apple) > - Junping Du (junping...@apache.org) (Tencent) > - Tao Yang (taoy...@apache.org) (Alibaba) > - Jason Lowe (jl...@apache.org) (Nvidia) > > 2.11 Sponsors > Champion > - Vinod Kumar Vavilapalli (vino...@apache.org) > > Nominated Mentors > - Junping Du (Tencent), (junping...@apache.org) > - Felix Cheung (Uber), (felixche...@apache.org) > - Jason Lowe (Nvidia), (jl...@apache.org) > - Holden Karau (Apple), (hol...@apache.org) > > Sponsoring Entity > - The Apache Incubator > > [1] https://cwiki.apache.org/confluence/display/INCUBATOR/YuniKornProposal > > -------------------------------- END OF THE PROPOSAL > ------------------------------- > > Thanks > Weiwei --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
