Hello all, As a Kubernetes fan, I am excited about this proposal. However, I would challenge this community to think more abstractly about the problem you want to address and any solution requirements before discussing implementation details, such as adopting VK.
Don't take me wrong, VK is a great concept: a Kubernetes node that delegates container management to someone else. But allow me to clarify a few things about it: - VK simply provides a very limited subset of the kubelet functionality, namely the Kubernetes node registration and the observation of Pods that have been assigned to it. It doesn't do pod (intra or inter) networking nor delegates to CNI, doesn't do volume mounting, and so on. - Like the kubelet, VK doesn't implement scheduling. It also doesn't understand anything else than a Pod and its dependencies (e.g. ConfigMap or Secret), meaning other primitives, such as DaemonSet, Deployment, StatefulSet, or extensions, such as CRDs are unknown to the VK. - While the kubelet manages containers through CRI API (Container Runtime Interface), the VK does it through its own Provider API. - kubelet translates from Kubernetes primitives to CRI primitives, so CRI implementations only need to understand CRI. However, the VK does no translation and passes Kubernetes primitives directly to a provider, requiring the VK provider to understand Kubernetes primitives. - kubelet talks to CRI implementations through a gRPC socket. VK talks to providers in-process and is highly-opinionated about the fact a provider has no lifecycle (there's no _start_ or _stop_, as there would be for a framework). There are talks about having Provide API over gRPC but it's not trivial to decide[2]. Now, if you are still thinking about implementation details, and having some experience trying to create a VK provider for Mesos[1], I can tell you the VK, as is today, is not a seamless fit. That said, I am willing to help you figure out the design and pick the right pieces to execute, if this is indeed something you want to do. 1 - https://github.com/pires/virtual-kubelet/tree/mesos_integration/providers/mesos 2 - https://github.com/virtual-kubelet/virtual-kubelet/issues/160 Cheers, Pires On Wed, Nov 28, 2018 at 5:38 AM Jie Yu <[email protected]> wrote: > + user list as well to hear more feedback from Mesos users. > > I am +1 on this proposal to create a Mesos framework that exposes k8s API, > and provide nodeless > <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit> > experience to users. > > Creating Mesos framework that provides k8s API is not a new idea. For > instance, the following are the two prior attempts: > 1. https://github.com/kubernetes-retired/kube-mesos-framework > 2. https://mesosphere.com/product/kubernetes-engine/ > > Both of the above solutions will run unmodified kubelets for workloads > (i.e., pods). Some users might prefer that way, and we should not preclude > that on Mesos. However, the reason this nodeless (aka, virtual kubelet) > idea got me very excited is because it provides us an opportunity to create > a truly integrated solution to bridge k8s and Mesos. > > K8s gets popular for reasons. IMO, the followings are the key: > (1) API machinery. This includes API extension mechanism (CRD > <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit>), > simple-to-program client, versioning, authn/authz, etc. > (2) It expose basic scheduling primitives, and let users/vendors focus on > orchestration (i.e., Operators). In contrast, Mesos framework is > significantly harder to program due to the need for doing scheduling also. > Although we have scheduling libraries like Fenzo > <https://github.com/Netflix/Fenzo>, the whole community suffers from > fragmentation because there's no "default" solution. > > ** Why this proposal is more integrated than prior solutions?* > > This is because prior solutions are more like installer for k8s. You > either need to pre-reserve resources > <https://mesosphere.com/product/kubernetes-engine/> for kubelet, or fork > k8s scheduler to bring up kubelet on demand > <https://github.com/kubernetes-retired/kube-mesos-framework>. Complexity > is definitely a concern since both systems are involved. In contrast, the > proposal propose to run k8s workloads (pods) directly on Mesos by > translating pod spec to tasks/executors in Mesos. It's just another Mesos > framework, but you can extend that framework behavior using k8s API > extension mechanism (CRD and Operator)! > > ** Compare to just using k8s?* > > First of all, IMO, k8s is just an API spec. Any implementation that passes > conformance tests is vanilla k8s experience. I understand that by going > nodeless, some of the concepts in k8s no longer applies (e.g., > NodeAffinity, NodeSelector). I am actually less worried about this for two > reasons: 1) Big stakeholders are behind nodeless, including Microsoft, AWS, > Alicloud, etc; 2) K8s API is evolving, and nodeless has real use cases > (e.g., in public clouds). > > In fact, we can also choose to implement those k8s APIs that make the most > sense first, and maybe define our own APIs, leveraging the extensibility of > the k8s API machinery! > > If we do want to compare to upstream k8s implementation, i think the main > benefit is that: > 1) You can still run your existing custom Mesos frameworks as it is today, > but start to provide your users some k8s API experiences > 2) Scalability. Mesos is inherently more scalable than k8s because it > takes different trade-offs. You can run multiple copies of the same > frameworks (similar to marathon on marathon) to reach large scale if the > k8s framework itself cannot scale beyond certain limit. > > ** Why putting this framework in Mesos repo?* > > Historically, the problem with Mesos community is fragmentation. People > create different solutions for the same set of problems. Having a "blessed" > solution in the Mesos repo has the following benefits: > 1) License and ownership. It's under Apache already. > 2) Attract contributions. Less fragmentation. > 3) Established high quality in the repository. > > **** What's my suggestion for next steps? **** > > I suggest we create a working group for this. Any other PMC that likes > this idea, please chime in here. > > - Jie > > On Fri, Nov 23, 2018 at 5:24 AM 张冬冬 <[email protected]> > wrote: > >> >> >> 发自我的 iPhone >> >> > 在 2018年11月23日，20:37，Alex Rukletsov <[email protected]> 写道： >> > >> > I'm in favour of the proposal, Cameron. Building a bridge between Mesos >> and >> > Kubernetes will be beneficial for both communities. Virtual kubelet >> effort >> > looks promising indeed and is definitely a worthwhile approach to build >> the >> > bridge. >> > >> > While we will need some sort of a scheduler when implementing a provider >> > for mesos, we don't need to implement and use a "default" one: a simple >> > mesos-go based scheduler will be fine for the start. We can of course >> > consider building a default scheduler, but this will significantly >> increase >> > the size of the project. >> > >> > An exercise we will have to do here is determine which parts of a >> > kubernetes task specification can be "converted" and hence launched on a >> > Mesos cluster. Once we have a working prototype we can start testing and >> > collecting data. >> > >> > Do you want to come up with a plan and maybe a more detailed proposal? >> > >> > Best, >> > Alex >> >
