Hi Spark Community, I am reaching out to see if there are current large scale production or pre-production deployment of Spark on k8s for batch and micro batch jobs. Large scale means running 100s of thousand spark jobs daily and 1000s of concurrent spark jobs on a single k8s cluster and 10s of millions of spark executor pods daily (not concurrently).
If you happen to run and develop Spark on k8s at such scale, I'd want to learn about your experience and scaling challenges and solutions. Thank you, Li
