Hi Raja,
A little atypical way to respond to your question - please check out the
most recent Spark AMA where we discuss this:
https://www.linkedin.com/posts/apachespark_apachespark-ama-committers-activity-6989052811397279744-jpWH?utm_source=share_medium=member_ios
HTH!
Denny
On Tue, Oct 25,
Hi Nikhil,
Spark on Kubernetes supports dynamic allocation using shuffle tracking
feature instead of the external shuffle service.
In order to enable dynamic allocation, you should set these two configs as
true:
spark.dynamicAllocation.enabled and
spark.dynamicAllocation.shuffleTracking.enabled
