Hi, In k8s the driver is responsible for executor creation. The likelihood of your problem is that Insufficient memory allocated for executors in the K8s cluster. Even with dynamic allocation, k8s won't schedule executor pods if there is not enough free memory to fulfill their resource requests.
My suggestions - Increase Executor Memory: Allocate more memory per executor (e.g., 2GB or 3GB) to allow for multiple executors within available cluster memory. - Adjust Driver Pod Resources: Ensure the driver pod has enough memory to run Spark and manage executors. - Optimize Resource Management: Explore on-demand allocation or adjusting allocation granularity for better resource utilization. For example look at documents for Executor On-Demand Allocation (spark.executor.cores=0): and spark.dynamicAllocation.minExecutors & spark.dynamicAllocation.maxExecutors HTH Mich Talebzadeh, Technologist | Architect | Data Engineer | Generative AI | FinCrime London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Tue, 30 Apr 2024 at 04:29, Tarun raghav <tarunraghavsrika...@gmail.com> wrote: > Respected Sir/Madam, > I am Tarunraghav. I have a query regarding spark on kubernetes. > > We have an eks cluster, within which we have spark installed in the pods. > We set the executor memory as 1GB and set the executor instances as 2, I > have also set dynamic allocation as true. So when I try to read a 3 GB CSV > file or parquet file, it is supposed to increase the number of pods by 2. > But the number of executor pods is zero. > I don't know why executor pods aren't being created, even though I set > executor instance as 2. Please suggest a solution for this. > > Thanks & Regards, > Tarunraghav > >
