Respected Sir/Madam, I am Tarunraghav. I have a query regarding spark on kubernetes.
We have an eks cluster, within which we have spark installed in the pods. We set the executor memory as 1GB and set the executor instances as 2, I have also set dynamic allocation as true. So when I try to read a 3 GB CSV file or parquet file, it is supposed to increase the number of pods by 2. But the number of executor pods is zero. I don't know why executor pods aren't being created, even though I set executor instance as 2. Please suggest a solution for this. Thanks & Regards, Tarunraghav
