Shuffle service with more than one executor

Hi, I have a spark standalone cluster running on Kubernetes With anti-affinity for network performance. I’d like to enable spark dynamic allocation and for this I need to enable shuffle services but Looks like I can’t do that running more than one worker instance on the same worker. Is there a

Terminate job without killing

I have a python spark job that runs successfully but never ends (releases the prompt). I got messages like "releasing accumulator" but never the shutdown message (expected) and the prompt release. In order to handle this I used sys.exit(0), now it works but the tasks always appears as KILLED a

Best approach to schedule Spark jobs

I have a standalone Spark cluster and have some jobs scheduled using crontab. It works but I don't have all the real time monitoring to get emails or to control a flow for example. Thought about using the Spark "hidden" API to have a better control but seems the API is not officially documented

Using sparkContext.stop()

Hey all, I have created a Spark Job that runs successfully but if I do not use sc.stop() at the end, the job hangs. It shows some "cleaned accumulator 0" messages but never finishes. I intent to use these jobs in production via spark-submit and schedule it in cron. Is that the best practice u