Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster

Hi Lucas, Thanks a ton for responding. have you used livy and SPARK in EMR? I am genuinely not sure how adding a spark-submit in EMR is hard, it is just one line of code. I must be missing something here Regards, Gourav Sengupta On Wed, Mar 21, 2018 at 2:37 PM, lucas.g...@gmail.com

Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster

Purna, It's a bit tangental to your original question but heads up that Amazon EKS is in Preview right now: https://aws.amazon.com/eks/ I don't know if it actually allows a nice interface between k8s hosted Spark & Lamda functions (my suspicion is it won't fix your problem), but might be

Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster

Speaking from experience, if you're already operating a kubernetes cluster. Getting a spark workload operating there is nearly an order of magnitude simpler than working with / around EMR. That's not say EMR is excessively hard, just that Kubernetes is easier, all the steps to getting your

Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster

Hi, just out of curiosity, but since it in AWS, is there any specific reason not to use EMR? Or any particular reason to use Kubernetes? Regards, Gourav Sengupta On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep wrote: > Im using kubernetes cluster on AWS to run spark

Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster

Thanks Yinan, Looks like this is stil in alpha version. Would like to know if there is any rest-interface for spark2.3 job submission similar to spark 2.2 as I need to submit spark applications to k8 master based on different events (cron or s3 file based trigger) On Tue, Mar 20, 2018 at 11:50

Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster

One option is the Spark Operator . It allows specifying and running Spark applications on Kubernetes using Kubernetes custom resources objects. It takes SparkApplication CRD objects and automatically submits the applications to run on a

Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster

Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3 ,now i want to run spark-submit from AWS lambda function to k8s master,would like to know if there is any REST interface to run Spark submit on k8s Master