Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon a...@mesosphere.io wrote: Hello friends, The Myriad team and I would like to propose the Myriad project for inclusion in the Apache Incubator. Full text of the proposal is below. I can add it to the incubator wiki as well, if desired. Please review and discuss. If there are no major concerns, I will call for a Vote after a week. Cheers, -Adam- me@apache == Apache Myriad Proposal * Abstract Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together on the same cluster and allows dynamic resource allocations across both Hadoop and other applications running on the same physical data center infrastructure. * Proposal The vision of Myriad is to provide a comprehensive framework to ensure Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes on either side and prevent the static fragmentation of data center resources. * Background Project Myriad is the first resource management framework that allows big data developers to run YARN-based Hadoop jobs alongside other applications and services in production. ebay Inc., MapR, and Mesosphere jointly built Myriad (available on Github at https://github.com/mesos/myriad) with the vision of freeing big data jobs from siloed clusters and consolidating infrastructure into a single pool of resources for greater utilization and operational efficiency. Several companies including Twitter have expressed interest in Myriad and have begun testing it. * Rationale Many Hadoop users are building larger clusters (data lake/data hub architectures) that support multiple workloads - made possible by the advent of Apache Hadoop YARN. As the clusters grow in size and importance, they become an important application within the broader datacenter. At the same time, Apache Mesos enables efficient resource isolation and sharing across distributed applications for the broader data center, for instance MPI, Spark, long running web services, build/test infrastructure, traditional linux applications/scripts, and others (including arbitrary docker images). Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos on the same physical data center resources, reducing fragmentation of data center resources. * Project Goals ** Initial Goals - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy based allocation of data center resources across Apache Hadoop and other distributed applications - Ensure YARN based execution frameworks work without any changes when running alongside Myriad. YARN Applications will continue to interact and run on top of YARN and can choose to be unaware of Myriad. - Ensure Mesos based execution frameworks work without any changes when running alongside Myriad. Mesos applications will continue to interact and run on Mesos and can choose to be unaware of Myriad. - Provide isolation for multi-tenancy. - Use linux cgroups (and optionally Docker-like technologies to ease packaging, deployment and broader isolation) so that multiple YARN clusters can run in their own space and are isolated from each other. YARN’s RM and NMs are dockerized. - Myriad should be able to manage full YARN lifecycle: - Bring up YARN (RM, NM) - Scale Up/Down YARN - Release resources and shut down YARN ** Longer Term Goals - Allow fine-grained dynamic allocation of resources to Hadoop including the ability to scale up and scale down the cluster. - Provide different policies to allow downsizing running applications on Hadoop when resources are taken away from it. - Provide a framework so the downsizing policy is pluggable and users can write their own implementations. - Allow multiple versions of Apache Hadoop to run on the same physical infrastructure - Allow workload portability - ability to migrate YARN workloads across various cloud infrastructures seamlessly (e.g. GCE, AWS, etc) - Security: - Authentication Requirements: - Support basic CRAM-MD5 password authentication between Myriad and Mesos. Additional authentication mechanisms may be supported in the future. - Traditional user authentication with Hadoop’s HTTP web-consoles should work as usual. - Authorization: - Only authorized users are allowed to launch YARN clusters. Mesos allows to specify which framework principal is allowed to register as a particular role. - Encryption on wire: - All control traffic to/from Myriad/Mesos - Logs - Audits (where to store them) - Log all major activities/events with audit trail - who, what, when, result - Launching YARN/RM - Launching NM’s - Downsizing NM’s - Terminating YARN/RM - What to do with old logs? - Debuggability/Visibility - Hooks to identify different YARN cluster lifecycles (yarn-id?) - GUI: Capability to scale-up and scale-down by selecting nodes and
Re: [DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Luciano, I would expect that you would make an excellent additional mentor. On Fri, Feb 13, 2015 at 5:15 PM, Luciano Resende luckbr1...@gmail.com wrote: On Fri, Feb 13, 2015 at 5:06 PM, Adam Bordelon a...@mesosphere.io wrote: Hello friends, The Myriad team and I would like to propose the Myriad project for inclusion in the Apache Incubator. Full text of the proposal is below. I can add it to the incubator wiki as well, if desired. Please review and discuss. If there are no major concerns, I will call for a Vote after a week. Cheers, -Adam- me@apache == Apache Myriad Proposal * Abstract Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together on the same cluster and allows dynamic resource allocations across both Hadoop and other applications running on the same physical data center infrastructure. * Proposal The vision of Myriad is to provide a comprehensive framework to ensure Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes on either side and prevent the static fragmentation of data center resources. * Background Project Myriad is the first resource management framework that allows big data developers to run YARN-based Hadoop jobs alongside other applications and services in production. ebay Inc., MapR, and Mesosphere jointly built Myriad (available on Github at https://github.com/mesos/myriad) with the vision of freeing big data jobs from siloed clusters and consolidating infrastructure into a single pool of resources for greater utilization and operational efficiency. Several companies including Twitter have expressed interest in Myriad and have begun testing it. * Rationale Many Hadoop users are building larger clusters (data lake/data hub architectures) that support multiple workloads - made possible by the advent of Apache Hadoop YARN. As the clusters grow in size and importance, they become an important application within the broader datacenter. At the same time, Apache Mesos enables efficient resource isolation and sharing across distributed applications for the broader data center, for instance MPI, Spark, long running web services, build/test infrastructure, traditional linux applications/scripts, and others (including arbitrary docker images). Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos on the same physical data center resources, reducing fragmentation of data center resources. * Project Goals ** Initial Goals - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy based allocation of data center resources across Apache Hadoop and other distributed applications - Ensure YARN based execution frameworks work without any changes when running alongside Myriad. YARN Applications will continue to interact and run on top of YARN and can choose to be unaware of Myriad. - Ensure Mesos based execution frameworks work without any changes when running alongside Myriad. Mesos applications will continue to interact and run on Mesos and can choose to be unaware of Myriad. - Provide isolation for multi-tenancy. - Use linux cgroups (and optionally Docker-like technologies to ease packaging, deployment and broader isolation) so that multiple YARN clusters can run in their own space and are isolated from each other. YARN’s RM and NMs are dockerized. - Myriad should be able to manage full YARN lifecycle: - Bring up YARN (RM, NM) - Scale Up/Down YARN - Release resources and shut down YARN ** Longer Term Goals - Allow fine-grained dynamic allocation of resources to Hadoop including the ability to scale up and scale down the cluster. - Provide different policies to allow downsizing running applications on Hadoop when resources are taken away from it. - Provide a framework so the downsizing policy is pluggable and users can write their own implementations. - Allow multiple versions of Apache Hadoop to run on the same physical infrastructure - Allow workload portability - ability to migrate YARN workloads across various cloud infrastructures seamlessly (e.g. GCE, AWS, etc) - Security: - Authentication Requirements: - Support basic CRAM-MD5 password authentication between Myriad and Mesos. Additional authentication mechanisms may be supported in the future. - Traditional user authentication with Hadoop’s HTTP web-consoles should work as usual. - Authorization: - Only authorized users are allowed to launch YARN clusters. Mesos allows to specify which framework principal is allowed to register as a particular role. - Encryption on wire: - All control traffic to/from Myriad/Mesos - Logs - Audits (where to store them) - Log all major activities/events with audit trail - who, what, when, result - Launching
[DISCUSS] [PROPOSAL] Myriad for Apache Incubator
Hello friends, The Myriad team and I would like to propose the Myriad project for inclusion in the Apache Incubator. Full text of the proposal is below. I can add it to the incubator wiki as well, if desired. Please review and discuss. If there are no major concerns, I will call for a Vote after a week. Cheers, -Adam- me@apache == Apache Myriad Proposal * Abstract Myriad enables co-existence of Apache Hadoop YARN and Apache Mesos together on the same cluster and allows dynamic resource allocations across both Hadoop and other applications running on the same physical data center infrastructure. * Proposal The vision of Myriad is to provide a comprehensive framework to ensure Apache Hadoop YARN and Apache Mesos can interoperate with minimal changes on either side and prevent the static fragmentation of data center resources. * Background Project Myriad is the first resource management framework that allows big data developers to run YARN-based Hadoop jobs alongside other applications and services in production. ebay Inc., MapR, and Mesosphere jointly built Myriad (available on Github at https://github.com/mesos/myriad) with the vision of freeing big data jobs from siloed clusters and consolidating infrastructure into a single pool of resources for greater utilization and operational efficiency. Several companies including Twitter have expressed interest in Myriad and have begun testing it. * Rationale Many Hadoop users are building larger clusters (data lake/data hub architectures) that support multiple workloads - made possible by the advent of Apache Hadoop YARN. As the clusters grow in size and importance, they become an important application within the broader datacenter. At the same time, Apache Mesos enables efficient resource isolation and sharing across distributed applications for the broader data center, for instance MPI, Spark, long running web services, build/test infrastructure, traditional linux applications/scripts, and others (including arbitrary docker images). Myriad aims to enable co-existence of Apache Hadoop YARN and Apache Mesos on the same physical data center resources, reducing fragmentation of data center resources. * Project Goals ** Initial Goals - Run Myriad alongside Apache Hadoop YARN and Apache Mesos to allow policy based allocation of data center resources across Apache Hadoop and other distributed applications - Ensure YARN based execution frameworks work without any changes when running alongside Myriad. YARN Applications will continue to interact and run on top of YARN and can choose to be unaware of Myriad. - Ensure Mesos based execution frameworks work without any changes when running alongside Myriad. Mesos applications will continue to interact and run on Mesos and can choose to be unaware of Myriad. - Provide isolation for multi-tenancy. - Use linux cgroups (and optionally Docker-like technologies to ease packaging, deployment and broader isolation) so that multiple YARN clusters can run in their own space and are isolated from each other. YARN’s RM and NMs are dockerized. - Myriad should be able to manage full YARN lifecycle: - Bring up YARN (RM, NM) - Scale Up/Down YARN - Release resources and shut down YARN ** Longer Term Goals - Allow fine-grained dynamic allocation of resources to Hadoop including the ability to scale up and scale down the cluster. - Provide different policies to allow downsizing running applications on Hadoop when resources are taken away from it. - Provide a framework so the downsizing policy is pluggable and users can write their own implementations. - Allow multiple versions of Apache Hadoop to run on the same physical infrastructure - Allow workload portability - ability to migrate YARN workloads across various cloud infrastructures seamlessly (e.g. GCE, AWS, etc) - Security: - Authentication Requirements: - Support basic CRAM-MD5 password authentication between Myriad and Mesos. Additional authentication mechanisms may be supported in the future. - Traditional user authentication with Hadoop’s HTTP web-consoles should work as usual. - Authorization: - Only authorized users are allowed to launch YARN clusters. Mesos allows to specify which framework principal is allowed to register as a particular role. - Encryption on wire: - All control traffic to/from Myriad/Mesos - Logs - Audits (where to store them) - Log all major activities/events with audit trail - who, what, when, result - Launching YARN/RM - Launching NM’s - Downsizing NM’s - Terminating YARN/RM - What to do with old logs? - Debuggability/Visibility - Hooks to identify different YARN cluster lifecycles (yarn-id?) - GUI: Capability to scale-up and scale-down by selecting nodes and providing a scale-up/scale-down factor. * Architectural Overview The following diagram illustrates the high level architecture. YARN (with Myriad) is registered as a
