Hi, On Fri, Mar 1, 2019 at 9:48 AM Xingbo Jiang <jiangxb1...@gmail.com> wrote: > > Hi Sean, > > To support GPU scheduling with YARN cluster, we have to update the hadoop > version to 3.1.2+. However, if we decide to not upgrade hadoop to beyond that > version for Spark 3.0, then we just have to disable/fallback the GPU > scheduling with YARN, users shall still be able to have that feature with > Standalone or Kubernetes cluster. > > We didn't include the Mesos support in current SPIP because we didn't receive > use cases that require GPU scheduling on Mesos cluster, however, we can still > add Mesos support in the future if we observe valid use cases.
First time caller, long time listener. We have GPUs in our Mesos-based Spark cluster, and it would be nice to use them with Spark-based GPU-enabled frameworks (our use case is deep learning applications). Cheers Andrew > > Thanks! > > Xingbo > > Sean Owen <sro...@gmail.com> 于2019年3月1日周五 下午10:39写道： >> >> Two late breaking questions: >> >> This basically requires Hadoop 3.1 for YARN support? >> Mesos support is listed as a non goal but it already has support for >> requesting GPUs in Spark. That would be 'harmonized' with this >> implementation even if it's not extended? >> >> On Fri, Mar 1, 2019, 7:48 AM Xingbo Jiang <jiangxb1...@gmail.com> wrote: >>> >>> I think we are aligned on the commitment, I'll start a vote thread for this >>> shortly. >>> >>> Xiangrui Meng <men...@gmail.com> 于2019年2月27日周三 上午6:47写道： >>>> >>>> In case there are issues visiting Google doc, I attached PDF files to the >>>> JIRA. >>>> >>>> On Tue, Feb 26, 2019 at 7:41 AM Xingbo Jiang <jiangxb1...@gmail.com> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I want send a revised SPIP on implementing Accelerator(GPU)-aware >>>>> Scheduling. It improves Spark by making it aware of GPUs exposed by >>>>> cluster managers, and hence Spark can match GPU resources with user task >>>>> requests properly. If you have scenarios that need to run >>>>> workloads(DL/ML/Signal Processing etc.) on Spark cluster with GPU nodes, >>>>> please help review and check how it fits into your use cases. Your >>>>> feedback would be greatly appreciated! >>>>> >>>>> # Links to SPIP and Product doc: >>>>> >>>>> * Jira issue for the SPIP: >>>>> https://issues.apache.org/jira/browse/SPARK-24615 >>>>> * Google Doc: >>>>> https://docs.google.com/document/d/1C4J_BPOcSCJc58HL7JfHtIzHrjU0rLRdQM3y7ejil64/edit?usp=sharing >>>>> * Product Doc: >>>>> https://docs.google.com/document/d/12JjloksHCdslMXhdVZ3xY5l1Nde3HRhIrqvzGnK_bNE/edit?usp=sharing >>>>> >>>>> Thank you! >>>>> >>>>> Xingbo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
