Re: Spark on Kubernetes scheduler variety

2021-06-23 Thread Klaus Ma
Hi team, I'm kube-batch/Volcano founder, and I'm excited to hear that the spark community also has such requirements :) Volcano provides several features for batch workload, e.g. fair-share, queue, reservation, preemption/reclaim and so on. It has been used in several product environments with

Re: Build customized resource manager

2019-11-10 Thread Klaus Ma
: > I don't know if it all works but some work was done to make cluster > manager pluggable, see SPARK-13904. > > Tom > > On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma < > klaus1982...@gmail.com> wrote: > > > Any suggestions? > > - Klaus > >

Re: Build customized resource manager

2019-11-06 Thread Klaus Ma
Any suggestions? - Klaus On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma wrote: > Hi team, > > AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did > some enhancement to them, e.g. integrate with Volcano > <http://github.com/volcano-sh/volcano> in k8s. I

How to enable Spark mesos docker executor?

2015-10-15 Thread Klaus Ma
park home in docker for this case? Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer Platform Symphony/DCOS Development & Support, STG, IBM GCG +86-10-8245 4084 | mad...@cn.ibm.com | http://www.cguru.net

RE: How to enable Spark mesos docker executor?

2015-10-15 Thread Klaus Ma
Hi Timothy, Thanks for your feedback, I logged https://issues.apache.org/jira/browse/SPARK-11143 to trace this issue. If any more suggestions, please let me know :). Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer Platform Symphony/DCOS Development & Support, STG, IBM GCG +8