Hi Spark community, I’m recently working on migrating Spark workloads from Yarn to K8s, and one of the issue I found is that there is no out-of-box log solution for Spark applications on K8s as Yarn has, and Spark even does not provide a way to integrate w/ external log service.
Based on the above situation, I propose a way[1][2] to allow Spark to integrate w/ external log service when running on K8s, and PR[3] is available for reviewing and testing. I also found there was a PR[4] wants to the similar thing. Please let me know if you have any feedback on this idea, and other log solutions/ideas/suggestions for Spark on K8s are welcome as well. [1] https://docs.google.com/document/d/1MfB39LD4B4Rp7MDRxZbMKMbdNSe6V6mBmMQ-gkCnM-0/edit?usp=sharing [2] https://issues.apache.org/jira/browse/SPARK-40887 [3] https://github.com/apache/spark/pull/38357 [4] https://github.com/apache/spark/pull/32456 Thanks, Cheng Pan
