Hi, Will. According to the following JIRA, as of now, there is no plan or on-going discussion to switch it.
https://issues.apache.org/jira/browse/SPARK-44111 (Prepare Apache Spark 4.0.0) Thanks, Dongjoon. On Wed, Sep 13, 2023 at 9:02 AM Will Raschkowski <wraschkow...@palantir.com.invalid> wrote: > Hey everyone, > > > > I was wondering what the plans are for Spark's built-in v2 file data > sources in Spark 4. > > > > Concretely, is the plan for Spark 4 to continue defaulting to the built-in > v1 data sources? And if yes, what are the blockers for defaulting to v2? I > see, just as example, that writing Hive-partitions is not supported in v2. > Are there other blockers or outstanding discussions? > > > > Regards, > > Will > > >
