Re: Deploying Spark on Google Kubernetes (GKE) autopilot, preliminary findings

Hi, sorry in case it appeared otherwise, Mich's takes are super interesting. Just that while applying solutions on commercial undertakings things are quite different from research/ development scenarios . Regards, Gourav Sengupta On Mon, Feb 14, 2022 at 5:02 PM

Re: Deploying Spark on Google Kubernetes (GKE) autopilot, preliminary findings

Thanks Mich. Very insightful. AKOn Monday, 14 February 2022, 11:18:19 GMT, Mich Talebzadeh wrote: Good question. However, we ought to look at what options we have so to speak.  Let us consider Spark on Dataproc, Spark on Kubernetes and Spark on Dataflow Spark on DataProc is proven

Re: Deploying Spark on Google Kubernetes (GKE) autopilot, preliminary findings

Hi, I would still not build any custom solution, and if in GCP use serverless Dataproc. I think that it is always better to be hands on with AWS Glue before commenting on it. Regards, Gourav Sengupta On Mon, Feb 14, 2022 at 11:18 AM Mich Talebzadeh wrote: > Good question. However, we ought to

Re: Deploying Spark on Google Kubernetes (GKE) autopilot, preliminary findings

Good question. However, we ought to look at what options we have so to speak. Let us consider Spark on Dataproc, Spark on Kubernetes and Spark on Dataflow Spark on DataProc is proven and it is in use at many organizations, I have deployed it extensively. It

Re: Deploying Spark on Google Kubernetes (GKE) autopilot, preliminary findings

Hi, may be this is useful in case someone is testing SPARK in containers for developing SPARK. *From a production scale work point of view:* But if I am in AWS, I will just use GLUE if I want to use containers for SPARK, without massively increasing my costs for operations unnecessarily. Also,

Deploying Spark on Google Kubernetes (GKE) autopilot, preliminary findings

The equivalent of Google GKE autopilot in AWS is AWS Fargate I have not used the AWS Fargate so I can only mension Google's GKE Autopilot. This is developed from the concept of