Meta At the weekly K8s Big Data SIG meeting today, we agreed to experiment with publishing a brief summary of noteworthy Spark-related topics from the weekly meeting to dev@spark, as a reference for interested members of the Apache Spark community.
The format is a brief summary, including a link to the SIG minutes (we also post a link to the meeting recording on the minutes when it becomes available). With that, here are the first SIG meeting notes: Spark Based on initial feedback on the remote shuffle service storage design exploration <https://issues.apache.org/jira/browse/SPARK-25299>, further work on design options is going to continue prior to attempting a POC implementation. The group is consulting Facebook and Baidu for additional input from their independent work in this area. We reviewed the new fractional CPU <https://issues.apache.org/jira/browse/SPARK-23285> support on the K8s back-end (landing in Spark 2.4) and discussed options for configuring fractional CPUs on the driver pod. A consensus was reached to proceed with the current PR <https://github.com/apache/spark/pull/22146> on apache/spark for the upcoming user supplied pod template feature on the K8s back-end. Link to meeting minutes https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85 Vms4V2uTsGZvSp8MNIA
