Hi Spark users, We would like to announce the release of Spark on MR3, which is Apache Spark using MR3 as the execution backend. MR3 is a general purpose execution engine for Hadoop and Kubernetes, and Hive on MR3 has been its main application. Spark on MR3 is a new application of MR3.
The main motivation for developing Spark on MR3 is to allow multiple Spark applications to share compute resources such as Yarn containers or Kubernetes Pods. It can be particularly useful in cloud environments where Spark applications are created and destroyed frequently. We wrote a blog article introducing Spark on MR3: https://www.datamonad.com/post/2021-08-18-spark-mr3/ Currently we have released Spark 3.0.3 on MR3 1.3. For more details on Spark on MR3, you can check out the user guide: https://mr3docs.datamonad.com/docs/spark/ Thanks, --- Sungwoo
