I am running many Spark 3.3 batch processing jobs on YARN. How can we configure the appropriate executor memory and off-heap memory sizes for a Spark 3 offline batch processing job? Setting them too large wastes resources, while setting them too small may cause insufficient memory and job failures.
