Cameron and Michal: I would love to understand your motivations and use cases for a k8s Mesos framework in a bit more detail. Looks like you are willing to rewrite your existing app definitions into k8s API spec. At this point, why are you still interested in Mesos as a CAAS backend? Is it because of scalability / reliability? Or is it because you still want to run non-k8s workloads/frameworks in this world? What are these workloads?
In general, I'm in favor of Mesos coming shipped with a default scheduler. I think it might help with the adoption similar to what happened with the command/default executor. In hindsight, we should've done this a long time ago. But, oh well, we were too optimistic that a single "default" scheduler will rule in the ecosystem which didn't quite pan out. However, I'm not sure if re-implementing k8s-scheduler as a Mesos framework is the right approach. I imagine k8s scheduler is significant piece of code which we need to re-implement and on top of it as new API objects are added to k8s API, we need to keep pace with k8s scheduler for parity. The approach we (in the community) took with Spark (and Jenkins to some extent) was for the scheduling innovation happen in Spark community and we just let Spark launch spark executors via Mesos and let Spark launch its tasks out of band of Mesos. We used to have a version of Spark framework (fine grained mode?) where spark tasks were launched via Mesos offers but that was deprecated, partly because of maintainability. Will this k8s framework have similar problem? Sounds like one of the problems with the existing k8s framework implementations it the pre-launching of kubelets; can we use the k8s autoscaler to solve that problem? Also, I think (I might be wrong) most k8s users are not directly creating pods via the API but rather using higher level abstractions like replica sets, stateful sets, daemon sets etc. How will that fit into this architecture? Will the framework need to re-implement those controllers as well? Is there an integration point in k8s ecosystem where we can reuse the existing k8s schedulers and controllers but run the pods with mesos container runtime? All, in all, I'm +1 to explore the ideas in a WG. On Wed, Nov 28, 2018 at 2:05 PM Paulo Pires <pi...@mesosphere.io> wrote: > Hello all, > > As a Kubernetes fan, I am excited about this proposal. > However, I would challenge this community to think more abstractly about > the problem you want to address and any solution requirements before > discussing implementation details, such as adopting VK. > > Don't take me wrong, VK is a great concept: a Kubernetes node that > delegates container management to someone else. > But allow me to clarify a few things about it: > > - VK simply provides a very limited subset of the kubelet functionality, > namely the Kubernetes node registration and the observation of Pods that > have been assigned to it. It doesn't do pod (intra or inter) networking nor > delegates to CNI, doesn't do volume mounting, and so on. > - Like the kubelet, VK doesn't implement scheduling. It also doesn't > understand anything else than a Pod and its dependencies (e.g. ConfigMap or > Secret), meaning other primitives, such as DaemonSet, Deployment, > StatefulSet, or extensions, such as CRDs are unknown to the VK. > - While the kubelet manages containers through CRI API (Container Runtime > Interface), the VK does it through its own Provider API. > - kubelet translates from Kubernetes primitives to CRI primitives, so CRI > implementations only need to understand CRI. However, the VK does no > translation and passes Kubernetes primitives directly to a provider, > requiring the VK provider to understand Kubernetes primitives. > - kubelet talks to CRI implementations through a gRPC socket. VK talks to > providers in-process and is highly-opinionated about the fact a provider > has no lifecycle (there's no _start_ or _stop_, as there would be for a > framework). There are talks about having Provide API over gRPC but it's not > trivial to decide[2]. > > Now, if you are still thinking about implementation details, and having > some experience trying to create a VK provider for Mesos[1], I can tell you > the VK, as is today, is not a seamless fit. > That said, I am willing to help you figure out the design and pick the > right pieces to execute, if this is indeed something you want to do. > > 1 - > https://github.com/pires/virtual-kubelet/tree/mesos_integration/providers/mesos > 2 - https://github.com/virtual-kubelet/virtual-kubelet/issues/160 > > Cheers, > Pires > > On Wed, Nov 28, 2018 at 5:38 AM Jie Yu <yujie....@gmail.com> wrote: > >> + user list as well to hear more feedback from Mesos users. >> >> I am +1 on this proposal to create a Mesos framework that exposes k8s >> API, and provide nodeless >> <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit> >> experience to users. >> >> Creating Mesos framework that provides k8s API is not a new idea. For >> instance, the following are the two prior attempts: >> 1. https://github.com/kubernetes-retired/kube-mesos-framework >> 2. https://mesosphere.com/product/kubernetes-engine/ >> >> Both of the above solutions will run unmodified kubelets for workloads >> (i.e., pods). Some users might prefer that way, and we should not preclude >> that on Mesos. However, the reason this nodeless (aka, virtual kubelet) >> idea got me very excited is because it provides us an opportunity to create >> a truly integrated solution to bridge k8s and Mesos. >> >> K8s gets popular for reasons. IMO, the followings are the key: >> (1) API machinery. This includes API extension mechanism (CRD >> <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit>), >> simple-to-program client, versioning, authn/authz, etc. >> (2) It expose basic scheduling primitives, and let users/vendors focus on >> orchestration (i.e., Operators). In contrast, Mesos framework is >> significantly harder to program due to the need for doing scheduling also. >> Although we have scheduling libraries like Fenzo >> <https://github.com/Netflix/Fenzo>, the whole community suffers from >> fragmentation because there's no "default" solution. >> >> ** Why this proposal is more integrated than prior solutions?* >> >> This is because prior solutions are more like installer for k8s. You >> either need to pre-reserve resources >> <https://mesosphere.com/product/kubernetes-engine/> for kubelet, or fork >> k8s scheduler to bring up kubelet on demand >> <https://github.com/kubernetes-retired/kube-mesos-framework>. Complexity >> is definitely a concern since both systems are involved. In contrast, the >> proposal propose to run k8s workloads (pods) directly on Mesos by >> translating pod spec to tasks/executors in Mesos. It's just another Mesos >> framework, but you can extend that framework behavior using k8s API >> extension mechanism (CRD and Operator)! >> >> ** Compare to just using k8s?* >> >> First of all, IMO, k8s is just an API spec. Any implementation that >> passes conformance tests is vanilla k8s experience. I understand that by >> going nodeless, some of the concepts in k8s no longer applies (e.g., >> NodeAffinity, NodeSelector). I am actually less worried about this for two >> reasons: 1) Big stakeholders are behind nodeless, including Microsoft, AWS, >> Alicloud, etc; 2) K8s API is evolving, and nodeless has real use cases >> (e.g., in public clouds). >> >> In fact, we can also choose to implement those k8s APIs that make the >> most sense first, and maybe define our own APIs, leveraging the >> extensibility of the k8s API machinery! >> >> If we do want to compare to upstream k8s implementation, i think the main >> benefit is that: >> 1) You can still run your existing custom Mesos frameworks as it is >> today, but start to provide your users some k8s API experiences >> 2) Scalability. Mesos is inherently more scalable than k8s because it >> takes different trade-offs. You can run multiple copies of the same >> frameworks (similar to marathon on marathon) to reach large scale if the >> k8s framework itself cannot scale beyond certain limit. >> >> ** Why putting this framework in Mesos repo?* >> >> Historically, the problem with Mesos community is fragmentation. People >> create different solutions for the same set of problems. Having a "blessed" >> solution in the Mesos repo has the following benefits: >> 1) License and ownership. It's under Apache already. >> 2) Attract contributions. Less fragmentation. >> 3) Established high quality in the repository. >> >> **** What's my suggestion for next steps? **** >> >> I suggest we create a working group for this. Any other PMC that likes >> this idea, please chime in here. >> >> - Jie >> >> On Fri, Nov 23, 2018 at 5:24 AM 张冬冬 <meaglegl...@icloud.com.invalid> >> wrote: >> >>> >>> >>> 发自我的 iPhone >>> >>> > 在 2018年11月23日，20:37，Alex Rukletsov <a...@mesosphere.com> 写道： >>> > >>> > I'm in favour of the proposal, Cameron. Building a bridge between >>> Mesos and >>> > Kubernetes will be beneficial for both communities. Virtual kubelet >>> effort >>> > looks promising indeed and is definitely a worthwhile approach to >>> build the >>> > bridge. >>> > >>> > While we will need some sort of a scheduler when implementing a >>> provider >>> > for mesos, we don't need to implement and use a "default" one: a simple >>> > mesos-go based scheduler will be fine for the start. We can of course >>> > consider building a default scheduler, but this will significantly >>> increase >>> > the size of the project. >>> > >>> > An exercise we will have to do here is determine which parts of a >>> > kubernetes task specification can be "converted" and hence launched on >>> a >>> > Mesos cluster. Once we have a working prototype we can start testing >>> and >>> > collecting data. >>> > >>> > Do you want to come up with a plan and maybe a more detailed proposal? >>> > >>> > Best, >>> > Alex >>> >>
