Hey Yue, Very interesting idea. I am not a kubernetes expert, but this seems like a neat concept. I guess the idea is only one MM would be needed? (Or maybe a handful, if one can't manage every pod?) If so, great. Hopefully someone that is more of a kubernetes expert will be able to chime in on the PR.
On Fri, Feb 26, 2021 at 12:46 AM Yue Zhang <zhangyue921...@163.com> wrote: > Hi Druid, > > I’d like to contribute a new Community Extension named > druid-kubernetes-middlemanager-extensions. Proposal Link: > https://github.com/apache/druid/issues/10824 and PR link : > https://github.com/apache/druid/pull/10910 > > When enable this feature, Druid can launch Peon as a pod to do data > ingestion. The advantages are as follows: > > Cost saving: there is no need to let MiddleManager take up a lot of > resources in advance, and just require resources whenever it will use. Now > we can use a single 2Cores and 2Gi Memory middelmanager pod to control > Dozens or even hundreds peon pods. > Improve resource utilization: Now different kinds of tasks can use > different configs including CPU resources and Memory resources if necessary. > > If you are interested in this feature, please let me know. Looking > forward to your reply. > > Best wishes, > > Yue Zhang > > >
