Hi all, I am learning about the performance difference of Spark when performing a JOIN problem on Serverless (K8S) and Serverful (Traditional server) environments.
Through experiment, Spark on K8s tends to run slower than Serverful. Through understanding the architecture, I know that Spark runs on K8s as Containers (Pods) so it takes a certain time to initialize, but when I look at each job, stage, and task, Spark K8s tends to be slower. Serverful. *I have some questions:* Q1: What are the causes and reasons for Spark on K8s to be slower than Serverful? Q2: How or is there a scenario to show the most apparent difference in performance and cost of these two environments (Serverless (K8S) and Serverful (Traditional server)? Thank you so much! Best regards, Truong
