Good point, though we did mark Spark 3.4 as end-of-life in the docs and removed it from main, meaning no future releases will include it. The 1.11.x branch is really just a byproduct of the release process rather than active maintenance.
On Wed, May 20, 2026 at 9:20 AM Manu Zhang <[email protected]> wrote: > Thanks Kevin. Spark 3.4 is not yet "End of Life" since we will maintain it > on the 1.11.x branch and might have more releases with it contained. > > On Thu, May 21, 2026 at 12:05 AM Kevin Liu <[email protected]> wrote: > >> The Spark 3.4 removal PR has been merged! ( >> https://github.com/apache/iceberg/pull/14122) >> Spark 3.4 is now considered "End of Life", >> https://iceberg.apache.org/multi-engine-support/#apache-spark >> And the last Iceberg support is version 1.11.0 >> >> Thanks everyone! >> >> On Thu, Jan 8, 2026 at 6:49 PM Manu Zhang <[email protected]> >> wrote: >> >>> Totally agreed. I'd propose we formalize this process in the doc. >>> >>> 1. Mark Spark X / Flink Y as deprecated for Iceberg Z in multi-engine >>> support doc[1]. >>> 2. Release Z with Spark X / Flink Y. >>> 3. Remove Spark X / Flink Y support as part of post action of release Z. >>> 4. Update multi-engine support doc accordingly. >>> >>> Regards, >>> Manu >>> >>> On Fri, Jan 9, 2026 at 3:19 AM Kevin Liu <[email protected]> wrote: >>> >>>> Hey folks, >>>> >>>> Picking this thread up again. I've gathered feedback from the >>>> community, and here's what I think is the best course of action. >>>> >>>> >>>> *Current State*We've already marked Spark 3.4 as "Deprecated" at >>>> https://iceberg.apache.org/multi-engine-support/#apache-spark. This >>>> means Spark 3.4 is no longer actively maintained as of 1.10.x. Backports >>>> can still be added, but the *community will not invest effort in >>>> achieving feature parity*. We recommend users move to a newer version: >>>> Spark 3.5 or Spark 4.x. >>>> >>>> Since the 1.10 release, there have been 34 backport PRs [1] currently >>>> on the main branch. Since Spark 3.4 is already marked as "Deprecated," >>>> *there's >>>> no obligation for the community to ensure all changes are backported to >>>> Spark 3.4.* I propose we release 1.11 with the available changes and >>>> fully remove Spark 3.4 support after the release. If further changes are >>>> needed for Spark 3.4 afterward, they can go into the 1.11.x branch as patch >>>> releases. >>>> >>>> >>>> *Proposed Actions*1. Allow Spark 3.4 backports with no obligation to >>>> ensure all features are backported >>>> 2. Release 1.11 with Spark 3.4 support >>>> 3. Fully remove the `spark/v3.4` folder immediately after 1.11 >>>> 4. Mark Spark 3.4 as "End-of-life" with 1.11 as the last supported >>>> version >>>> >>>> Please share your thoughts. I'll also link this to the vote thread >>>> on removing Spark 3.4. >>>> >>>> Best, >>>> Kevin Liu >>>> >>>> >>>> [1] git log --pretty=oneline apache-iceberg-1.10.1..main -- spark/v3.4/ >>>> | nl >>>> >>>> On Mon, Oct 13, 2025 at 10:03 PM huaxin gao <[email protected]> >>>> wrote: >>>> >>>>> Instead of removing Spark 3.4 in 1.11, could we keep it for a couple >>>>> more versions? >>>>> >>>>> Thanks, >>>>> Huaxin >>>>> >>>>> On Mon, Oct 13, 2025 at 9:57 AM Parth Chandra <[email protected]> >>>>> wrote: >>>>> >>>>>> I'm wondering if it is too late to request just one more release of >>>>>> Iceberg with Spark 3.4 support? I have a PR open >>>>>> https://github.com/apache/iceberg/pull/13786 that addresses a whole >>>>>> host of issues with using Comet and Iceberg. This completes the work >>>>>> started by Huaxin in https://github.com/apache/iceberg/pull/13378. >>>>>> At the moment Comet still supports Spark 3.4 and the PR accordingly >>>>>> updates the Spark 3.4 module as well. If we deprecate Spark 3.4, users of >>>>>> Spark 3.4 who want to use Comet will have to build their own. It's not >>>>>> the >>>>>> end of the world, Comet already provides a patch and instructions on how >>>>>> to >>>>>> do this. It is, however, less than ideal. >>>>>> I know that voting has already started (and is practically over) on >>>>>> this issue, but I figured it doesn't hurt to try. >>>>>> >>>>>> Parth >>>>>> >>>>>> On Tue, Oct 7, 2025 at 7:49 AM Kevin Liu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> The PR has been merged (https://github.com/apache/iceberg/pull/14099), >>>>>>> and I can see the change reflected on the website, >>>>>>> https://iceberg.apache.org/multi-engine-support/#apache-spark >>>>>>> Thanks everyone for the review. >>>>>>> >>>>>>> I'll start a separate thread for removing spark 3.4 from the repo. >>>>>>> >>>>>>> Best, >>>>>>> Kevin Liu >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 6, 2025 at 3:03 PM Kevin Liu <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hey folks, >>>>>>>> >>>>>>>> All (except 1) recent contributors to spark/v3.4 have responded on >>>>>>>> the slack thread above and support removing spark 3.4. The last >>>>>>>> contributor >>>>>>>> has not responded yet. >>>>>>>> >>>>>>>> If there are no objections, I plan to merge this PR to change the >>>>>>>> status for Spark 3.4 from "Maintained" -> "Deprecated" . >>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>> Thanks Eduard for the review. >>>>>>>> >>>>>>>> Best, >>>>>>>> Kevin Liu >>>>>>>> >>>>>>>> On Mon, Sep 29, 2025 at 9:52 AM Kevin Liu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Started a slack thread, >>>>>>>>> https://apache-iceberg.slack.com/archives/C03LG1D563F/p1759163920874249 >>>>>>>>> Based on the recently commit activities, I messaged (on slack and >>>>>>>>> pr) 4 contributors who are actively backporting Spark 3.4. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Kevin Liu >>>>>>>>> >>>>>>>>> On Mon, Sep 29, 2025 at 9:32 AM Kevin Liu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Bumping this thread. >>>>>>>>>> Since there's consensus to mark Spark 3.4 as deprecated, let's >>>>>>>>>> merge this PR to change the status on the docs, >>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>> I'll start a new thread for the decision to mark Spark 3.4 as >>>>>>>>>> end-of-life and for removal from the codebase. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Kevin Liu >>>>>>>>>> >>>>>>>>>> On Mon, Sep 22, 2025 at 11:13 AM Steve Loughran >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> making hadoop 3.4.1 the base hadoop version has sort of made >>>>>>>>>>> iceberg incompatible with spark 3.4, it just doesn't surface very >>>>>>>>>>> much. >>>>>>>>>>> >>>>>>>>>>> My PR, https://github.com/apache/iceberg/pull/10233 , which >>>>>>>>>>> uses a hadoop 3.4.1 BulkDelete call not in the spark 3.4 classpath >>>>>>>>>>> flags >>>>>>>>>>> this: the spark 3.4 tests fail with classpath errors. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, 19 Sept 2025 at 22:02, Kevin Liu <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks Anton and Eduard. I'm ok with being more aggressive with >>>>>>>>>>>> the deprecation schedule. >>>>>>>>>>>> Looking at the git history for `spark/v3.4/` >>>>>>>>>>>> <https://github.com/apache/iceberg/commits/main/spark/v3.4>, >>>>>>>>>>>> there are 5 new commits since the 1.10 release. Only 1 commit ( >>>>>>>>>>>> 3bbdee9 >>>>>>>>>>>> <https://github.com/apache/iceberg/commit/3bbdee97b0f33794df091c3292de8d54a05dcc1e>) >>>>>>>>>>>> is to backport specifically for Spark 3.4. >>>>>>>>>>>> >>>>>>>>>>>> From Engine Version Lifecycle >>>>>>>>>>>> <https://iceberg.apache.org/multi-engine-support/#engine-version-lifecycle> >>>>>>>>>>>> , >>>>>>>>>>>> > Deprecated: an engine version is no longer actively >>>>>>>>>>>> maintained. People who are still interested in the version can >>>>>>>>>>>> backport any >>>>>>>>>>>> necessary feature or bug fix from newer versions, but the >>>>>>>>>>>> community will >>>>>>>>>>>> not spend effort in achieving feature parity. Iceberg recommends >>>>>>>>>>>> users to >>>>>>>>>>>> move towards a newer version. Contributions to a deprecated >>>>>>>>>>>> version is >>>>>>>>>>>> expected to diminish over time, so that eventually no change is >>>>>>>>>>>> added to a >>>>>>>>>>>> deprecated version. >>>>>>>>>>>> and >>>>>>>>>>>> > End-of-life: a vote can be initiated in the community to >>>>>>>>>>>> fully remove a deprecated version out of the Iceberg repository to >>>>>>>>>>>> mark as >>>>>>>>>>>> its end of life. >>>>>>>>>>>> >>>>>>>>>>>> Let's change the status for Spark 3.4 to "Deprecated". I have >>>>>>>>>>>> prepared a PR already >>>>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>>>> >>>>>>>>>>>> I can start another vote thread for removal of Spark 3.4 in the >>>>>>>>>>>> upcoming 1.11 release and mark it as "End-of-life". I already >>>>>>>>>>>> prepared a PR >>>>>>>>>>>> to fully remove Spark 3.4 from the codebase, >>>>>>>>>>>> https://github.com/apache/iceberg/pull/14122 >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Kevin Liu >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 19, 2025 at 11:29 AM Eduard Tudenhöfner < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I agree with Anton and I would be in favor of just removing it >>>>>>>>>>>>> in the next release. By updating the docs now we can already >>>>>>>>>>>>> signal >>>>>>>>>>>>> immediately that Spark 3.4 is deprecated and people can always >>>>>>>>>>>>> use Iceberg >>>>>>>>>>>>> 1.10 when needing Spark 3.4 support. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 19, 2025 at 7:06 PM Anton Okolnychyi < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I know we followed this rule of deprecating a Spark version >>>>>>>>>>>>>> in one release and then removing it in the next one. Shall we >>>>>>>>>>>>>> ask ourselves >>>>>>>>>>>>>> whether it is still the model we want to follow? >>>>>>>>>>>>>> >>>>>>>>>>>>>> My problem like before is that we release a new Iceberg jar >>>>>>>>>>>>>> that is supposed to contain the latest and greatest features but >>>>>>>>>>>>>> the >>>>>>>>>>>>>> functionality for older Spark versions is severely lagging. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We initially kept older Spark modules in main to give folks >>>>>>>>>>>>>> in the community a place to maintain these older integrations and >>>>>>>>>>>>>> collaborate. I don’t see a lot of interest in that, if I am >>>>>>>>>>>>>> being honest. >>>>>>>>>>>>>> Instead, it became a liability for devs as all major format >>>>>>>>>>>>>> features now >>>>>>>>>>>>>> have to work with those old Spark integrations. It is hurting >>>>>>>>>>>>>> the velocity >>>>>>>>>>>>>> of the project and recent row ID work is an example of that. >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Anton >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 6:33 PM Kevin Liu < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > why not just remove Spark 3.4 for the next 1.11 release? >>>>>>>>>>>>>>> Or do we usually wait for one more release and remove it in the >>>>>>>>>>>>>>> 1.12 >>>>>>>>>>>>>>> release after marking 3.4 as deprecated in the engine status >>>>>>>>>>>>>>> doc page? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My preference is to mark as deprecated for one release and >>>>>>>>>>>>>>> remove in the following. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To quote JB: >>>>>>>>>>>>>>> "announce" the deprecation in 1.11 and remove 1.12, it >>>>>>>>>>>>>>> gives time for users to "adapt". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Kevin Liu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:26 AM Steven Wu < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Following up on Manu's question, why not just remove Spark >>>>>>>>>>>>>>>> 3.4 for the next 1.11 release? Or do we usually wait for one >>>>>>>>>>>>>>>> more release >>>>>>>>>>>>>>>> and remove it in the 1.12 release after marking 3.4 as >>>>>>>>>>>>>>>> deprecated in the >>>>>>>>>>>>>>>> engine status doc page? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:12 AM Kevin Liu < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Given the many +1's here, I've moved the PR to deprecate >>>>>>>>>>>>>>>>> 3.4 to "ready for review", >>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> > Does it mean we will stop back-porting PRs to Spark 3.4 >>>>>>>>>>>>>>>>> for 1.11? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Not necessarily. There's a lot of Spark 3.4 backports >>>>>>>>>>>>>>>>> already, >>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/commits/main/spark/v3.4 >>>>>>>>>>>>>>>>> I suggest we continue to backport for consistency and then >>>>>>>>>>>>>>>>> stop right after the 1.11 release. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Kevin Liu >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 6:18 AM Amogh Jahagirdar < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 2:03 AM Péter Váry < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Eduard Tudenhöfner <[email protected]> ezt írta >>>>>>>>>>>>>>>>>>> (időpont: 2025. szept. 19., P, 8:56): >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> +1 on deprecating Spark 3.4 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Sep 18, 2025 at 8:36 AM Steve < >>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 22:52 Jean-Baptiste Onofré < >>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I agree about the plan to "announce" the deprecation >>>>>>>>>>>>>>>>>>>>>> in 1.11 and >>>>>>>>>>>>>>>>>>>>>> remove 1.12, it gives time for users to "adapt". >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>>> JB >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 10:31 PM Kevin Liu < >>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > Hi everyone, >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > I’d like to bring up the topic of deprecating Spark >>>>>>>>>>>>>>>>>>>>>> 3.4 in an upcoming release. Anton initially suggested >>>>>>>>>>>>>>>>>>>>>> this during our >>>>>>>>>>>>>>>>>>>>>> previous dev list discussion about maintaining feature >>>>>>>>>>>>>>>>>>>>>> parity across the >>>>>>>>>>>>>>>>>>>>>> Spark versions we support for 1.10. >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > Currently, we support two different Spark 3.x >>>>>>>>>>>>>>>>>>>>>> versions, 3.4 and 3.5. Spark 3.4’s last maintenance >>>>>>>>>>>>>>>>>>>>>> release was in October >>>>>>>>>>>>>>>>>>>>>> 2024, and it is now considered end-of-life. >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > What are your thoughts on marking Spark 3.4 as >>>>>>>>>>>>>>>>>>>>>> deprecated in 1.11 and removing it in 1.12? >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > For reference, here's the previous discussion >>>>>>>>>>>>>>>>>>>>>> thread on deprecating Spark 3.3. >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > Best, >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > Kevin Liu >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>
