Purna, It's a bit tangental to your original question but heads up that Amazon EKS is in Preview right now: https://aws.amazon.com/eks/
I don't know if it actually allows a nice interface between k8s hosted Spark & Lamda functions (my suspicion is it won't fix your problem), but might be something worth taking a peak at, since you're already doing AWS hosted K8s stuff. On Wed, Mar 21, 2018 at 9:37 AM, lucas.g...@gmail.com <lucas.g...@gmail.com> wrote: > Speaking from experience, if you're already operating a kubernetes > cluster. Getting a spark workload operating there is nearly an order of > magnitude simpler than working with / around EMR. > > That's not say EMR is excessively hard, just that Kubernetes is easier, > all the steps to getting your application deployed are well documented and > ultimately the whole process is more visible. > > Also, thanks for the link Yinan! I'll be investigating that project! > > We have in the past used EMR for larger workloads and as soon as we > announced our users could run those workloads on our k8s cluster everyone > immediately moved their workloads over. This was despite having Spark on > K8s still in an infant state. No one has expressed interest in moving back. > > G > > On 21 March 2018 at 07:32, Gourav Sengupta <gourav.sengu...@gmail.com> > wrote: > >> Hi, >> >> just out of curiosity, but since it in AWS, is there any specific reason >> not to use EMR? Or any particular reason to use Kubernetes? >> >> >> Regards, >> Gourav Sengupta >> >> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <purna2prad...@gmail.com> >> wrote: >> >>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3 >>> ,now i want to run spark-submit from AWS lambda function to k8s >>> master,would like to know if there is any REST interface to run Spark >>> submit on k8s Master >> >> >> >
