Hey Jayabindu,
We use thriftserver for on K8S. May I ask why you are not going for Trino
instead? I know it didn't support autoscaling when we tested it in the past but
not sure if it does now.
Autoscaling also means that users might have to wait for the cluster to
autoscale but that usually happens not so slow and once its done then other
queries have the new nodes available.
Also the workload on our thriftserver is not so large so it solves the purpose
for now.
You can also take a look at Apache Kyuubi.
I can put in some details below and attach the config we use for spark
thriftserver, you can pick whatever is relevant for you:
* We run thriftserver on default(stable) nodes and its executors on
preemptible(spot) nodes
* We use driver and executor templates to make above possible by using node
selectors
* We use fair scheduling to manage workload
Mvg/Regards
Saurabh
From: Jayabindu Singh
Sent: 09 March 2023 06:31
To: u...@spark.incubator.apache.org
Subject: [EXTERNAL] Spark Thrift Server - Autoscaling on K8
Caution! This email originated outside of FedEx. Please do not open attachments
or click links from an unknown or suspicious origin.
Hi All,
We are in the process of moving our workloads to K8 and looking for some
guidance to run Spark Thrift Server on K8.
We need the executor pods to autoscale based on the workload vs running it with
a static number of executors.
If any one has done it and can share the details, it will be really appreciated.
Regards
Jayabindu Singh
spark-defaults.conf
Description: spark-defaults.conf
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
