We now have both Mesos and Kubernetes(not running on Mesos) running in production.As Jie mentioned,with this proposal,I mainly want to solve the static partition issue.I agree to explore the ideas in a WG.
Cameron Jie Yu <yujie....@gmail.com> 于2018年12月6日周四 上午9:48写道： > I'd like to get some feedback on what Mesos users want. I can potentially > see two major use cases: > > (1) I just want k8s to run on Mesos, along with other Mesos frameworks, > sharing the same resources pool. I don't really care about nodeless. > Ideally, i'd like to run upstream k8s (include kubelet). The original k8s > on mesos framework has been retired, and the new Mesosphere MKE is not open > source, and only runs on Mesosphere DC/OS. I need one open source solution > here. > (2) I want nodeless because I believe it has a tighter integration with > Mesos, as compared to (2), and can solve the static partition issue. (1) is > more like a k8s installer, and you can do that without Mesos. > > *Can folks chime in here?* > > However, I'm not sure if re-implementing k8s-scheduler as a Mesos framework > > is the right approach. I imagine k8s scheduler is significant piece of > > code which we need to re-implement and on top of it as new API objects > are > > added to k8s API, we need to keep pace with k8s scheduler for parity. The > > approach we (in the community) took with Spark (and Jenkins to some > extent) > > was for the scheduling innovation happen in Spark community and we just > let > > Spark launch spark executors via Mesos and let Spark launch its tasks out > > of band of Mesos. We used to have a version of Spark framework (fine > > grained mode?) where spark tasks were launched via Mesos offers but that > > was deprecated, partly because of maintainability. Will this k8s > framework > > have similar problem? Sounds like one of the problems with the existing > k8s > > framework implementations it the pre-launching of kubelets; can we use > the > > k8s autoscaler to solve that problem? > > > This is a good concern. It's around 17k lines of code in k8s scheduler. > > Jies-MacBook-Pro:scheduler jie$ pwd > /Users/jie/workspace/kubernetes/pkg/scheduler > Jies-MacBook-Pro:scheduler jie$ loc --exclude .*_test.go > > -------------------------------------------------------------------------------- > Language Files Lines Blank Comment > Code > > -------------------------------------------------------------------------------- > Go 83 17429 2165 3798 > 11466 > > -------------------------------------------------------------------------------- > Total 83 17429 2165 3798 > 11466 > > -------------------------------------------------------------------------------- > > Also, I think (I might be wrong) most k8s users are not directly creating > > pods via the API but rather using higher level abstractions like replica > > sets, stateful sets, daemon sets etc. How will that fit into this > > architecture? Will the framework need to re-implement those controllers > as > > well? > > > This is not true. You can re-use most of the controllers. Those controllers > will create pods as you said, and the mesos framework will be responsible > for scheduling those pods created. > > - Jie > > On Mon, Dec 3, 2018 at 9:56 AM Cecile, Adam <adam.cec...@hitec.lu> wrote: > > > On 12/3/18 5:40 PM, Michał Łowicki wrote: > > > > > > > > On Thu, Nov 29, 2018 at 1:22 AM Vinod Kone <vinodk...@apache.org> wrote: > > > >> Cameron and Michal: I would love to understand your motivations and use > >> cases for a k8s Mesos framework in a bit more detail. Looks like you are > >> willing to rewrite your existing app definitions into k8s API spec. At > >> this > >> point, why are you still interested in Mesos as a CAAS backend? Is it > >> because of scalability / reliability? Or is it because you still want to > >> run non-k8s workloads/frameworks in this world? What are these > workloads? > >> > > > > Mesos with its scalability and ability to run many frameworks (like > > cron-like jobs, spark, proprietary) gives more flexibility in the long > run. > > Right now we're at the stage where Marathon UI in public version isn't > > maintained so looking to have something with better community support. > > Having entity like k8s-compliant scheduler maybe could help with adopting > > other community-driven solutions but I also think that going into that > > direction should be well thought and planned process. > > > > We're sharing the exact same feeling. My next project will probably go > > full k8s because I don't feel confident in mesos future as an opensource > > project. > > > > Marathon UI still not supporting GPUs (even in JSON mode, thanks to > > marshaling) is the tip of the iceberg. I reported the issue ages ago and > I > > can understand nobody cares because DC/OS comes with a different > > (closed-source I bet) UI. > > > > > > > >> > >> In general, I'm in favor of Mesos coming shipped with a default > scheduler. > >> I think it might help with the adoption similar to what happened with > the > >> command/default executor. In hindsight, we should've done this a long > time > >> ago. But, oh well, we were too optimistic that a single "default" > >> scheduler > >> will rule in the ecosystem which didn't quite pan out. > >> > >> However, I'm not sure if re-implementing k8s-scheduler as a Mesos > >> framework > >> is the right approach. I imagine k8s scheduler is significant piece of > >> code which we need to re-implement and on top of it as new API objects > >> are > >> added to k8s API, we need to keep pace with k8s scheduler for parity. > The > >> approach we (in the community) took with Spark (and Jenkins to some > >> extent) > >> was for the scheduling innovation happen in Spark community and we just > >> let > >> Spark launch spark executors via Mesos and let Spark launch its tasks > out > >> of band of Mesos. We used to have a version of Spark framework (fine > >> grained mode?) where spark tasks were launched via Mesos offers but that > >> was deprecated, partly because of maintainability. Will this k8s > framework > >> have similar problem? Sounds like one of the problems with the existing > >> k8s > >> framework implementations it the pre-launching of kubelets; can we use > the > >> k8s autoscaler to solve that problem? > >> > >> Also, I think (I might be wrong) most k8s users are not directly > creating > >> pods via the API but rather using higher level abstractions like replica > >> sets, stateful sets, daemon sets etc. How will that fit into this > >> architecture? Will the framework need to re-implement those controllers > as > >> well? > >> > >> Is there an integration point in k8s ecosystem where we can reuse the > >> existing k8s schedulers and controllers but run the pods with mesos > >> container runtime? > >> > >> All, in all, I'm +1 to explore the ideas in a WG. > >> > >> > >> On Wed, Nov 28, 2018 at 2:05 PM Paulo Pires <pi...@mesosphere.io> > wrote: > >> > >> > Hello all, > >> > > >> > As a Kubernetes fan, I am excited about this proposal. > >> > However, I would challenge this community to think more abstractly > about > >> > the problem you want to address and any solution requirements before > >> > discussing implementation details, such as adopting VK. > >> > > >> > Don't take me wrong, VK is a great concept: a Kubernetes node that > >> > delegates container management to someone else. > >> > But allow me to clarify a few things about it: > >> > > >> > - VK simply provides a very limited subset of the kubelet > functionality, > >> > namely the Kubernetes node registration and the observation of Pods > that > >> > have been assigned to it. It doesn't do pod (intra or inter) > networking > >> nor > >> > delegates to CNI, doesn't do volume mounting, and so on. > >> > - Like the kubelet, VK doesn't implement scheduling. It also doesn't > >> > understand anything else than a Pod and its dependencies (e.g. > >> ConfigMap or > >> > Secret), meaning other primitives, such as DaemonSet, Deployment, > >> > StatefulSet, or extensions, such as CRDs are unknown to the VK. > >> > - While the kubelet manages containers through CRI API (Container > >> Runtime > >> > Interface), the VK does it through its own Provider API. > >> > - kubelet translates from Kubernetes primitives to CRI primitives, so > >> CRI > >> > implementations only need to understand CRI. However, the VK does no > >> > translation and passes Kubernetes primitives directly to a provider, > >> > requiring the VK provider to understand Kubernetes primitives. > >> > - kubelet talks to CRI implementations through a gRPC socket. VK talks > >> to > >> > providers in-process and is highly-opinionated about the fact a > provider > >> > has no lifecycle (there's no _start_ or _stop_, as there would be for > a > >> > framework). There are talks about having Provide API over gRPC but > it's > >> not > >> > trivial to decide[2]. > >> > > >> > Now, if you are still thinking about implementation details, and > having > >> > some experience trying to create a VK provider for Mesos[1], I can > tell > >> you > >> > the VK, as is today, is not a seamless fit. > >> > That said, I am willing to help you figure out the design and pick the > >> > right pieces to execute, if this is indeed something you want to do. > >> > > >> > 1 - > >> > > >> > https://github.com/pires/virtual-kubelet/tree/mesos_integration/providers/mesos > >> > 2 - https://github.com/virtual-kubelet/virtual-kubelet/issues/160 > >> > > >> > Cheers, > >> > Pires > >> > > >> > On Wed, Nov 28, 2018 at 5:38 AM Jie Yu <yujie....@gmail.com> wrote: > >> > > >> >> + user list as well to hear more feedback from Mesos users. > >> >> > >> >> I am +1 on this proposal to create a Mesos framework that exposes k8s > >> >> API, and provide nodeless > >> >> < > >> > https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit > >> > > >> >> experience to users. > >> >> > >> >> Creating Mesos framework that provides k8s API is not a new idea. For > >> >> instance, the following are the two prior attempts: > >> >> 1. https://github.com/kubernetes-retired/kube-mesos-framework > >> >> 2. https://mesosphere.com/product/kubernetes-engine/ > >> >> > >> >> Both of the above solutions will run unmodified kubelets for > workloads > >> >> (i.e., pods). Some users might prefer that way, and we should not > >> preclude > >> >> that on Mesos. However, the reason this nodeless (aka, virtual > kubelet) > >> >> idea got me very excited is because it provides us an opportunity to > >> create > >> >> a truly integrated solution to bridge k8s and Mesos. > >> >> > >> >> K8s gets popular for reasons. IMO, the followings are the key: > >> >> (1) API machinery. This includes API extension mechanism (CRD > >> >> < > >> > https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit > >> >), > >> >> simple-to-program client, versioning, authn/authz, etc. > >> >> (2) It expose basic scheduling primitives, and let users/vendors > focus > >> on > >> >> orchestration (i.e., Operators). In contrast, Mesos framework is > >> >> significantly harder to program due to the need for doing scheduling > >> also. > >> >> Although we have scheduling libraries like Fenzo > >> >> <https://github.com/Netflix/Fenzo>, the whole community suffers from > >> >> fragmentation because there's no "default" solution. > >> >> > >> >> ** Why this proposal is more integrated than prior solutions?* > >> >> > >> >> This is because prior solutions are more like installer for k8s. You > >> >> either need to pre-reserve resources > >> >> <https://mesosphere.com/product/kubernetes-engine/> for kubelet, or > >> fork > >> >> k8s scheduler to bring up kubelet on demand > >> >> <https://github.com/kubernetes-retired/kube-mesos-framework>. > >> Complexity > >> >> is definitely a concern since both systems are involved. In contrast, > >> the > >> >> proposal propose to run k8s workloads (pods) directly on Mesos by > >> >> translating pod spec to tasks/executors in Mesos. It's just another > >> Mesos > >> >> framework, but you can extend that framework behavior using k8s API > >> >> extension mechanism (CRD and Operator)! > >> >> > >> >> ** Compare to just using k8s?* > >> >> > >> >> First of all, IMO, k8s is just an API spec. Any implementation that > >> >> passes conformance tests is vanilla k8s experience. I understand that > >> by > >> >> going nodeless, some of the concepts in k8s no longer applies (e.g., > >> >> NodeAffinity, NodeSelector). I am actually less worried about this > for > >> two > >> >> reasons: 1) Big stakeholders are behind nodeless, including > Microsoft, > >> AWS, > >> >> Alicloud, etc; 2) K8s API is evolving, and nodeless has real use > cases > >> >> (e.g., in public clouds). > >> >> > >> >> In fact, we can also choose to implement those k8s APIs that make the > >> >> most sense first, and maybe define our own APIs, leveraging the > >> >> extensibility of the k8s API machinery! > >> >> > >> >> If we do want to compare to upstream k8s implementation, i think the > >> main > >> >> benefit is that: > >> >> 1) You can still run your existing custom Mesos frameworks as it is > >> >> today, but start to provide your users some k8s API experiences > >> >> 2) Scalability. Mesos is inherently more scalable than k8s because it > >> >> takes different trade-offs. You can run multiple copies of the same > >> >> frameworks (similar to marathon on marathon) to reach large scale if > >> the > >> >> k8s framework itself cannot scale beyond certain limit. > >> >> > >> >> ** Why putting this framework in Mesos repo?* > >> >> > >> >> Historically, the problem with Mesos community is fragmentation. > People > >> >> create different solutions for the same set of problems. Having a > >> "blessed" > >> >> solution in the Mesos repo has the following benefits: > >> >> 1) License and ownership. It's under Apache already. > >> >> 2) Attract contributions. Less fragmentation. > >> >> 3) Established high quality in the repository. > >> >> > >> >> **** What's my suggestion for next steps? **** > >> >> > >> >> I suggest we create a working group for this. Any other PMC that > likes > >> >> this idea, please chime in here. > >> >> > >> >> - Jie > >> >> > >> >> On Fri, Nov 23, 2018 at 5:24 AM 张冬冬 <meaglegl...@icloud.com.invalid> > >> >> wrote: > >> >> > >> >>> > >> >>> > >> >>> 发自我的 iPhone > >> >>> > >> >>> > 在 2018年11月23日，20:37，Alex Rukletsov <a...@mesosphere.com> 写道： > >> >>> > > >> >>> > I'm in favour of the proposal, Cameron. Building a bridge between > >> >>> Mesos and > >> >>> > Kubernetes will be beneficial for both communities. Virtual > kubelet > >> >>> effort > >> >>> > looks promising indeed and is definitely a worthwhile approach to > >> >>> build the > >> >>> > bridge. > >> >>> > > >> >>> > While we will need some sort of a scheduler when implementing a > >> >>> provider > >> >>> > for mesos, we don't need to implement and use a "default" one: a > >> simple > >> >>> > mesos-go based scheduler will be fine for the start. We can of > >> course > >> >>> > consider building a default scheduler, but this will significantly > >> >>> increase > >> >>> > the size of the project. > >> >>> > > >> >>> > An exercise we will have to do here is determine which parts of a > >> >>> > kubernetes task specification can be "converted" and hence > launched > >> on > >> >>> a > >> >>> > Mesos cluster. Once we have a working prototype we can start > testing > >> >>> and > >> >>> > collecting data. > >> >>> > > >> >>> > Do you want to come up with a plan and maybe a more detailed > >> proposal? > >> >>> > > >> >>> > Best, > >> >>> > Alex > >> >>> > >> >> > >> > > > > > > -- > > BR, > > Michał Łowicki > > > > > > >
