+ user list as well to hear more feedback from Mesos users. I am +1 on this proposal to create a Mesos framework that exposes k8s API, and provide nodeless <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit> experience to users.
Creating Mesos framework that provides k8s API is not a new idea. For instance, the following are the two prior attempts: 1. https://github.com/kubernetes-retired/kube-mesos-framework 2. https://mesosphere.com/product/kubernetes-engine/ Both of the above solutions will run unmodified kubelets for workloads (i.e., pods). Some users might prefer that way, and we should not preclude that on Mesos. However, the reason this nodeless (aka, virtual kubelet) idea got me very excited is because it provides us an opportunity to create a truly integrated solution to bridge k8s and Mesos. K8s gets popular for reasons. IMO, the followings are the key: (1) API machinery. This includes API extension mechanism (CRD <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit>), simple-to-program client, versioning, authn/authz, etc. (2) It expose basic scheduling primitives, and let users/vendors focus on orchestration (i.e., Operators). In contrast, Mesos framework is significantly harder to program due to the need for doing scheduling also. Although we have scheduling libraries like Fenzo <https://github.com/Netflix/Fenzo>, the whole community suffers from fragmentation because there's no "default" solution. ** Why this proposal is more integrated than prior solutions?* This is because prior solutions are more like installer for k8s. You either need to pre-reserve resources <https://mesosphere.com/product/kubernetes-engine/> for kubelet, or fork k8s scheduler to bring up kubelet on demand <https://github.com/kubernetes-retired/kube-mesos-framework>. Complexity is definitely a concern since both systems are involved. In contrast, the proposal propose to run k8s workloads (pods) directly on Mesos by translating pod spec to tasks/executors in Mesos. It's just another Mesos framework, but you can extend that framework behavior using k8s API extension mechanism (CRD and Operator)! ** Compare to just using k8s?* First of all, IMO, k8s is just an API spec. Any implementation that passes conformance tests is vanilla k8s experience. I understand that by going nodeless, some of the concepts in k8s no longer applies (e.g., NodeAffinity, NodeSelector). I am actually less worried about this for two reasons: 1) Big stakeholders are behind nodeless, including Microsoft, AWS, Alicloud, etc; 2) K8s API is evolving, and nodeless has real use cases (e.g., in public clouds). In fact, we can also choose to implement those k8s APIs that make the most sense first, and maybe define our own APIs, leveraging the extensibility of the k8s API machinery! If we do want to compare to upstream k8s implementation, i think the main benefit is that: 1) You can still run your existing custom Mesos frameworks as it is today, but start to provide your users some k8s API experiences 2) Scalability. Mesos is inherently more scalable than k8s because it takes different trade-offs. You can run multiple copies of the same frameworks (similar to marathon on marathon) to reach large scale if the k8s framework itself cannot scale beyond certain limit. ** Why putting this framework in Mesos repo?* Historically, the problem with Mesos community is fragmentation. People create different solutions for the same set of problems. Having a "blessed" solution in the Mesos repo has the following benefits: 1) License and ownership. It's under Apache already. 2) Attract contributions. Less fragmentation. 3) Established high quality in the repository. **** What's my suggestion for next steps? **** I suggest we create a working group for this. Any other PMC that likes this idea, please chime in here. - Jie On Fri, Nov 23, 2018 at 5:24 AM 张冬冬 <meaglegl...@icloud.com.invalid> wrote: > > > 发自我的 iPhone > > > 在 2018年11月23日,20:37,Alex Rukletsov <a...@mesosphere.com> 写道: > > > > I'm in favour of the proposal, Cameron. Building a bridge between Mesos > and > > Kubernetes will be beneficial for both communities. Virtual kubelet > effort > > looks promising indeed and is definitely a worthwhile approach to build > the > > bridge. > > > > While we will need some sort of a scheduler when implementing a provider > > for mesos, we don't need to implement and use a "default" one: a simple > > mesos-go based scheduler will be fine for the start. We can of course > > consider building a default scheduler, but this will significantly > increase > > the size of the project. > > > > An exercise we will have to do here is determine which parts of a > > kubernetes task specification can be "converted" and hence launched on a > > Mesos cluster. Once we have a working prototype we can start testing and > > collecting data. > > > > Do you want to come up with a plan and maybe a more detailed proposal? > > > > Best, > > Alex >