Thanks Kevin. Spark 3.4 is not yet "End of Life" since we will maintain it on the 1.11.x branch and might have more releases with it contained.
On Thu, May 21, 2026 at 12:05 AM Kevin Liu <[email protected]> wrote: > The Spark 3.4 removal PR has been merged! ( > https://github.com/apache/iceberg/pull/14122) > Spark 3.4 is now considered "End of Life", > https://iceberg.apache.org/multi-engine-support/#apache-spark > And the last Iceberg support is version 1.11.0 > > Thanks everyone! > > On Thu, Jan 8, 2026 at 6:49 PM Manu Zhang <[email protected]> wrote: > >> Totally agreed. I'd propose we formalize this process in the doc. >> >> 1. Mark Spark X / Flink Y as deprecated for Iceberg Z in multi-engine >> support doc[1]. >> 2. Release Z with Spark X / Flink Y. >> 3. Remove Spark X / Flink Y support as part of post action of release Z. >> 4. Update multi-engine support doc accordingly. >> >> Regards, >> Manu >> >> On Fri, Jan 9, 2026 at 3:19 AM Kevin Liu <[email protected]> wrote: >> >>> Hey folks, >>> >>> Picking this thread up again. I've gathered feedback from the community, >>> and here's what I think is the best course of action. >>> >>> >>> *Current State*We've already marked Spark 3.4 as "Deprecated" at >>> https://iceberg.apache.org/multi-engine-support/#apache-spark. This >>> means Spark 3.4 is no longer actively maintained as of 1.10.x. Backports >>> can still be added, but the *community will not invest effort in >>> achieving feature parity*. We recommend users move to a newer version: >>> Spark 3.5 or Spark 4.x. >>> >>> Since the 1.10 release, there have been 34 backport PRs [1] currently on >>> the main branch. Since Spark 3.4 is already marked as "Deprecated," *there's >>> no obligation for the community to ensure all changes are backported to >>> Spark 3.4.* I propose we release 1.11 with the available changes and >>> fully remove Spark 3.4 support after the release. If further changes are >>> needed for Spark 3.4 afterward, they can go into the 1.11.x branch as patch >>> releases. >>> >>> >>> *Proposed Actions*1. Allow Spark 3.4 backports with no obligation to >>> ensure all features are backported >>> 2. Release 1.11 with Spark 3.4 support >>> 3. Fully remove the `spark/v3.4` folder immediately after 1.11 >>> 4. Mark Spark 3.4 as "End-of-life" with 1.11 as the last supported >>> version >>> >>> Please share your thoughts. I'll also link this to the vote thread >>> on removing Spark 3.4. >>> >>> Best, >>> Kevin Liu >>> >>> >>> [1] git log --pretty=oneline apache-iceberg-1.10.1..main -- spark/v3.4/ >>> | nl >>> >>> On Mon, Oct 13, 2025 at 10:03 PM huaxin gao <[email protected]> >>> wrote: >>> >>>> Instead of removing Spark 3.4 in 1.11, could we keep it for a couple >>>> more versions? >>>> >>>> Thanks, >>>> Huaxin >>>> >>>> On Mon, Oct 13, 2025 at 9:57 AM Parth Chandra <[email protected]> >>>> wrote: >>>> >>>>> I'm wondering if it is too late to request just one more release of >>>>> Iceberg with Spark 3.4 support? I have a PR open >>>>> https://github.com/apache/iceberg/pull/13786 that addresses a whole >>>>> host of issues with using Comet and Iceberg. This completes the work >>>>> started by Huaxin in https://github.com/apache/iceberg/pull/13378. >>>>> At the moment Comet still supports Spark 3.4 and the PR accordingly >>>>> updates the Spark 3.4 module as well. If we deprecate Spark 3.4, users of >>>>> Spark 3.4 who want to use Comet will have to build their own. It's not the >>>>> end of the world, Comet already provides a patch and instructions on how >>>>> to >>>>> do this. It is, however, less than ideal. >>>>> I know that voting has already started (and is practically over) on >>>>> this issue, but I figured it doesn't hurt to try. >>>>> >>>>> Parth >>>>> >>>>> On Tue, Oct 7, 2025 at 7:49 AM Kevin Liu <[email protected]> >>>>> wrote: >>>>> >>>>>> The PR has been merged (https://github.com/apache/iceberg/pull/14099), >>>>>> and I can see the change reflected on the website, >>>>>> https://iceberg.apache.org/multi-engine-support/#apache-spark >>>>>> Thanks everyone for the review. >>>>>> >>>>>> I'll start a separate thread for removing spark 3.4 from the repo. >>>>>> >>>>>> Best, >>>>>> Kevin Liu >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Oct 6, 2025 at 3:03 PM Kevin Liu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hey folks, >>>>>>> >>>>>>> All (except 1) recent contributors to spark/v3.4 have responded on >>>>>>> the slack thread above and support removing spark 3.4. The last >>>>>>> contributor >>>>>>> has not responded yet. >>>>>>> >>>>>>> If there are no objections, I plan to merge this PR to change the >>>>>>> status for Spark 3.4 from "Maintained" -> "Deprecated" . >>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>> Thanks Eduard for the review. >>>>>>> >>>>>>> Best, >>>>>>> Kevin Liu >>>>>>> >>>>>>> On Mon, Sep 29, 2025 at 9:52 AM Kevin Liu <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Started a slack thread, >>>>>>>> https://apache-iceberg.slack.com/archives/C03LG1D563F/p1759163920874249 >>>>>>>> Based on the recently commit activities, I messaged (on slack and >>>>>>>> pr) 4 contributors who are actively backporting Spark 3.4. >>>>>>>> >>>>>>>> Best, >>>>>>>> Kevin Liu >>>>>>>> >>>>>>>> On Mon, Sep 29, 2025 at 9:32 AM Kevin Liu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Bumping this thread. >>>>>>>>> Since there's consensus to mark Spark 3.4 as deprecated, let's >>>>>>>>> merge this PR to change the status on the docs, >>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>> I'll start a new thread for the decision to mark Spark 3.4 as >>>>>>>>> end-of-life and for removal from the codebase. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Kevin Liu >>>>>>>>> >>>>>>>>> On Mon, Sep 22, 2025 at 11:13 AM Steve Loughran >>>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> making hadoop 3.4.1 the base hadoop version has sort of made >>>>>>>>>> iceberg incompatible with spark 3.4, it just doesn't surface very >>>>>>>>>> much. >>>>>>>>>> >>>>>>>>>> My PR, https://github.com/apache/iceberg/pull/10233 , which uses >>>>>>>>>> a hadoop 3.4.1 BulkDelete call not in the spark 3.4 classpath flags >>>>>>>>>> this: >>>>>>>>>> the spark 3.4 tests fail with classpath errors. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, 19 Sept 2025 at 22:02, Kevin Liu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Anton and Eduard. I'm ok with being more aggressive with >>>>>>>>>>> the deprecation schedule. >>>>>>>>>>> Looking at the git history for `spark/v3.4/` >>>>>>>>>>> <https://github.com/apache/iceberg/commits/main/spark/v3.4>, >>>>>>>>>>> there are 5 new commits since the 1.10 release. Only 1 commit ( >>>>>>>>>>> 3bbdee9 >>>>>>>>>>> <https://github.com/apache/iceberg/commit/3bbdee97b0f33794df091c3292de8d54a05dcc1e>) >>>>>>>>>>> is to backport specifically for Spark 3.4. >>>>>>>>>>> >>>>>>>>>>> From Engine Version Lifecycle >>>>>>>>>>> <https://iceberg.apache.org/multi-engine-support/#engine-version-lifecycle> >>>>>>>>>>> , >>>>>>>>>>> > Deprecated: an engine version is no longer actively >>>>>>>>>>> maintained. People who are still interested in the version can >>>>>>>>>>> backport any >>>>>>>>>>> necessary feature or bug fix from newer versions, but the community >>>>>>>>>>> will >>>>>>>>>>> not spend effort in achieving feature parity. Iceberg recommends >>>>>>>>>>> users to >>>>>>>>>>> move towards a newer version. Contributions to a deprecated version >>>>>>>>>>> is >>>>>>>>>>> expected to diminish over time, so that eventually no change is >>>>>>>>>>> added to a >>>>>>>>>>> deprecated version. >>>>>>>>>>> and >>>>>>>>>>> > End-of-life: a vote can be initiated in the community to >>>>>>>>>>> fully remove a deprecated version out of the Iceberg repository to >>>>>>>>>>> mark as >>>>>>>>>>> its end of life. >>>>>>>>>>> >>>>>>>>>>> Let's change the status for Spark 3.4 to "Deprecated". I have >>>>>>>>>>> prepared a PR already >>>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>>> >>>>>>>>>>> I can start another vote thread for removal of Spark 3.4 in the >>>>>>>>>>> upcoming 1.11 release and mark it as "End-of-life". I already >>>>>>>>>>> prepared a PR >>>>>>>>>>> to fully remove Spark 3.4 from the codebase, >>>>>>>>>>> https://github.com/apache/iceberg/pull/14122 >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Kevin Liu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 19, 2025 at 11:29 AM Eduard Tudenhöfner < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I agree with Anton and I would be in favor of just removing it >>>>>>>>>>>> in the next release. By updating the docs now we can already signal >>>>>>>>>>>> immediately that Spark 3.4 is deprecated and people can always use >>>>>>>>>>>> Iceberg >>>>>>>>>>>> 1.10 when needing Spark 3.4 support. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 19, 2025 at 7:06 PM Anton Okolnychyi < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I know we followed this rule of deprecating a Spark version in >>>>>>>>>>>>> one release and then removing it in the next one. Shall we ask >>>>>>>>>>>>> ourselves >>>>>>>>>>>>> whether it is still the model we want to follow? >>>>>>>>>>>>> >>>>>>>>>>>>> My problem like before is that we release a new Iceberg jar >>>>>>>>>>>>> that is supposed to contain the latest and greatest features but >>>>>>>>>>>>> the >>>>>>>>>>>>> functionality for older Spark versions is severely lagging. >>>>>>>>>>>>> >>>>>>>>>>>>> We initially kept older Spark modules in main to give folks in >>>>>>>>>>>>> the community a place to maintain these older integrations and >>>>>>>>>>>>> collaborate. >>>>>>>>>>>>> I don’t see a lot of interest in that, if I am being honest. >>>>>>>>>>>>> Instead, it >>>>>>>>>>>>> became a liability for devs as all major format features now have >>>>>>>>>>>>> to work >>>>>>>>>>>>> with those old Spark integrations. It is hurting the velocity of >>>>>>>>>>>>> the >>>>>>>>>>>>> project and recent row ID work is an example of that. >>>>>>>>>>>>> >>>>>>>>>>>>> - Anton >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 19, 2025 at 6:33 PM Kevin Liu < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> > why not just remove Spark 3.4 for the next 1.11 release? >>>>>>>>>>>>>> Or do we usually wait for one more release and remove it in the >>>>>>>>>>>>>> 1.12 >>>>>>>>>>>>>> release after marking 3.4 as deprecated in the engine status doc >>>>>>>>>>>>>> page? >>>>>>>>>>>>>> >>>>>>>>>>>>>> My preference is to mark as deprecated for one release and >>>>>>>>>>>>>> remove in the following. >>>>>>>>>>>>>> >>>>>>>>>>>>>> To quote JB: >>>>>>>>>>>>>> "announce" the deprecation in 1.11 and remove 1.12, it >>>>>>>>>>>>>> gives time for users to "adapt". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Kevin Liu >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:26 AM Steven Wu < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Following up on Manu's question, why not just remove Spark >>>>>>>>>>>>>>> 3.4 for the next 1.11 release? Or do we usually wait for one >>>>>>>>>>>>>>> more release >>>>>>>>>>>>>>> and remove it in the 1.12 release after marking 3.4 as >>>>>>>>>>>>>>> deprecated in the >>>>>>>>>>>>>>> engine status doc page? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:12 AM Kevin Liu < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Given the many +1's here, I've moved the PR to deprecate >>>>>>>>>>>>>>>> 3.4 to "ready for review", >>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14099 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> > Does it mean we will stop back-porting PRs to Spark 3.4 >>>>>>>>>>>>>>>> for 1.11? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Not necessarily. There's a lot of Spark 3.4 backports >>>>>>>>>>>>>>>> already, >>>>>>>>>>>>>>>> https://github.com/apache/iceberg/commits/main/spark/v3.4 >>>>>>>>>>>>>>>> I suggest we continue to backport for consistency and then >>>>>>>>>>>>>>>> stop right after the 1.11 release. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> Kevin Liu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 6:18 AM Amogh Jahagirdar < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 2:03 AM Péter Váry < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Eduard Tudenhöfner <[email protected]> ezt írta >>>>>>>>>>>>>>>>>> (időpont: 2025. szept. 19., P, 8:56): >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> +1 on deprecating Spark 3.4 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Thu, Sep 18, 2025 at 8:36 AM Steve < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 22:52 Jean-Baptiste Onofré < >>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I agree about the plan to "announce" the deprecation >>>>>>>>>>>>>>>>>>>>> in 1.11 and >>>>>>>>>>>>>>>>>>>>> remove 1.12, it gives time for users to "adapt". >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>> JB >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 17, 2025 at 10:31 PM Kevin Liu < >>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > Hi everyone, >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > I’d like to bring up the topic of deprecating Spark >>>>>>>>>>>>>>>>>>>>> 3.4 in an upcoming release. Anton initially suggested >>>>>>>>>>>>>>>>>>>>> this during our >>>>>>>>>>>>>>>>>>>>> previous dev list discussion about maintaining feature >>>>>>>>>>>>>>>>>>>>> parity across the >>>>>>>>>>>>>>>>>>>>> Spark versions we support for 1.10. >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > Currently, we support two different Spark 3.x >>>>>>>>>>>>>>>>>>>>> versions, 3.4 and 3.5. Spark 3.4’s last maintenance >>>>>>>>>>>>>>>>>>>>> release was in October >>>>>>>>>>>>>>>>>>>>> 2024, and it is now considered end-of-life. >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > What are your thoughts on marking Spark 3.4 as >>>>>>>>>>>>>>>>>>>>> deprecated in 1.11 and removing it in 1.12? >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > For reference, here's the previous discussion thread >>>>>>>>>>>>>>>>>>>>> on deprecating Spark 3.3. >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > Best, >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > Kevin Liu >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>
