I think it’s reasonable to drop 3.0 at 1.9, thanks for driving this discussion
Thanks, Cheng Pan > On Oct 11, 2023, at 10:30, Kent Yao <[email protected]> wrote: > > Hi dev, > > Since v1.6, the Spark AuthZ Plugin has been introduced as a core > extension for our Spark SQL engine and other standalone Spark SQL > applications. > > Although authz heavily depends on private modules of Spark SQL and > Spark Catalyst, we have well designed an extensible serde layer for > spark cross-version compatibilities, which makes us easily support all > Spark 3 releases. The main reasons for not bringing Spark 2, which is > not that hard, are Scala version incompatibilities, lack of use cases > and contributions. > > The development of Spark 4 is currently underway. It then brings critical > changes such as JDK and scala compiler, which require a lot of efforts > for our community to catch up with. > > Supporting mainstream data lake projects, such as Iceberg, Hudi, and > Deltalake is one of the major goals for authz. These projects all heavily > depend on private modules of Spark and handle cross-version compatibilities > in different ways, or maybe they don't particularly care. > > Thus, considering the maintenance burden of the Kyuubi community and > easy cross-support for data lake projects, it looks like it's time to > remove Spark 3.0 in v1.9 for the Spark AuthZ Plugin. > > Note that for Spark 3.0 users, AuthZ will still bring bug fixes to > v1.7.x and v1.8.0. > > FYI, https://github.com/apache/kyuubi/pull/5363, Yi Zhu is helping with this > and > comments in this thread and PR 5363 are welcome. > > Kent
