Speaking from experience, if you're already operating a kubernetes cluster. Getting a spark workload operating there is nearly an order of magnitude simpler than working with / around EMR.
That's not say EMR is excessively hard, just that Kubernetes is easier, all the steps to getting your application deployed are well documented and ultimately the whole process is more visible. Also, thanks for the link Yinan! I'll be investigating that project! We have in the past used EMR for larger workloads and as soon as we announced our users could run those workloads on our k8s cluster everyone immediately moved their workloads over. This was despite having Spark on K8s still in an infant state. No one has expressed interest in moving back. G On 21 March 2018 at 07:32, Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Hi, > > just out of curiosity, but since it in AWS, is there any specific reason > not to use EMR? Or any particular reason to use Kubernetes? > > > Regards, > Gourav Sengupta > > On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <purna2prad...@gmail.com> > wrote: > >> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3 >> ,now i want to run spark-submit from AWS lambda function to k8s >> master,would like to know if there is any REST interface to run Spark >> submit on k8s Master > > >
