Re: One click to run Spark on Kubernetes

Hi, There are two distinct actions here; namely Deploy and Run. Deployment can be done by command line script with autoscaling. In the newer versions of Kubernnetes you don't even need to specify the node types, you can leave it to the Kubernetes cluster to scale up and down and decide on node

Re: One click to run Spark on Kubernetes

Merging another email from Prasad. It could co-exist with livy. Livy is similar like the REST Service + Spark Operator. Unfortunately Livy is not very active right now. To Amihay, the link is: https://github.com/datapunchorg/punch. On Tue, Feb 22, 2022 at 8:53 PM amihay gonen wrote: > Can you

Re: One click to run Spark on Kubernetes

We do not have SaaS yet. Now it is an open source project we build in our part time , and we welcome more people working together on that. You could specify cluster size (EC2 instance type and number of instances) and run it for 1 hour. Then you could run one click command to destroy the cluster.

Re: One click to run Spark on Kubernetes

Hi Bo Yang, Would it be something along the lines of Apache livy? Thanks, Prasad On Tue, Feb 22, 2022 at 10:22 PM bo yang wrote: > It is not a standalone spark cluster. In some details, it deploys a Spark > Operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) > and an extra

Re: One click to run Spark on Kubernetes

It is not a standalone spark cluster. In some details, it deploys a Spark Operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) and an extra REST Service. When people submit Spark application to that REST Service, the REST Service will create a CRD inside the Kubernetes cluster.

One click to run Spark on Kubernetes

Hi Spark Community, We built an open source tool to deploy and run Spark on Kubernetes with a one click command. For example, on AWS, it could automatically create an EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will be able to use curl or a CLI tool to submit Spark

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

I understand that, and we do so, but news about official images were breaking me Ok, I will follow you on those activities. Thanks for the quick response. On Tue, 22 Feb 2022 at 22:03, Holden Karau wrote: > So your more than welcome to still build your own Spark docker containers > with

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

So your more than welcome to still build your own Spark docker containers with the docker image tool, these are provided to make it easier for folks without specific needs. In the future well hopefully have published Spark containers tagged for different JDKs but that work has not yet been done.

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

Hello Holden, Could you please provide more details and plan for docker images support? So far I see that there are only two tags, I get from them spark version, but there is no information about java, hadoop, scala versions. Also there is no description on docker hub, probably it would be nice

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

Well that is just a recommendation. The onus is on me the user to download and go through dev and test running suite of batch jobs to ensure that all work ok, especially on the edge, sign the release off and roll it in out into production. It won’t be prudent otherwise. HHH On Tue, 22 Feb 2022

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

"Spark 3.1.3 is a maintenance release containing stability fixes. This release is based on the branch-3.1 maintenance branch of Spark. We strongly recommend all 3.1.3 users to upgrade to this stable release." https://spark.apache.org/releases/spark-release-3-1-3.html Do we have another 3.13 or do

Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

Hi, seems - [SPARK-35391] : Memory leak in ExecutorAllocationListener breaks dynamic allocation under high load Links to wrong jira ticket? Mich Talebzadeh 于2022年2月22日周二 15:49写道： > Well, that is pretty easy to do. > > However, a