Updates for the remaining 3 open PRs for 1.10.0 milestone https://github.com/apache/iceberg/milestone/54
Core: Keep track of data files to be removed for orphaned DV detection <https://github.com/apache/iceberg/pull/13222> * got 2 approvals. should be merged in a day or two Spark 4.0: Migrate Iceberg Stored Procedures to Spark built-in implementations <https://github.com/apache/iceberg/pull/13106> * Got 3 approvals. expect it to be merged in a day or two Core: Fix numeric overflow of timestamp nano literal <https://github.com/apache/iceberg/pull/11775> * It breaks the use case of long literal value for nano timestamp column. But there is no correctness issue. Hence I am favoring moving it out of the 1.10.0 milestone * there is no consensus on the path forward yet. On Thu, Jul 3, 2025 at 2:28 PM Szehon Ho <szehon.apa...@gmail.com> wrote: > Thanks Steven! > > > On Jul 3, 2025, at 1:56 PM, Steven Wu <stevenz...@gmail.com> wrote: > > > > because it makes a bit of backward incompatible change by using the > Spark behavior (for argument case sensitivity and type coercion support), > we should do it in 1.10 release as we will have Spark 4.0 as part of it > iiuc. Else 1.11 will be backward incompatible for the Spark 4.0 jar. > > Szehon, thanks for bringing up PR 13106. I agree with your argument here. > It will be great to include this in the 1.10.0 release to avoid a behavior > change later because Spark 4.0 support was released in this version. It > seems close as you already approved it. Can we get a couple more approvals? > > > > On Thu, Jul 3, 2025 at 1:37 PM Szehon Ho <szehon.apa...@gmail.com> wrote: > >> Hi Steven >> >> Thanks. One more, what do we think about having >> https://github.com/apache/iceberg/pull/13106/ as part of 1.10 release? >> It's migrating Spark procedure to use Spark 4's new DSV2 Procedure >> framework. >> >> Its not a blocker but I feel, because it makes a bit of backward >> incompatible change by using the Spark behavior (for argument case >> sensitivity and type coercion support), we should do it in 1.10 release as >> we will have Spark 4.0 as part of it iiuc. Else 1.11 will be backward >> incompatible for the Spark 4.0 jar. >> >> Thanks, >> Szehon >> >> >> >> On Thu, Jul 3, 2025 at 1:17 PM Steven Wu <stevenz...@gmail.com> wrote: >> >>> Szehon's backport PR has been merged. Another blocker (dangling DVs for >>> rewrite) was also merged. >>> Core, Spark: Propagate orphaned delete files when rewriting data files >>> <https://github.com/apache/iceberg/pull/13245> >>> >>> There are still 3 blockers left, as a new PR (also danglinge DV related) >>> is identified as a blocker. >>> >>> Core: Keep track of data files to be removed for orphaned DV detection >>> <https://github.com/apache/iceberg/pull/13222> >>> * being actively reviewed and worked on. >>> >>> S3: Add LegacyMd5Plugin to S3 client builder >>> <https://github.com/apache/iceberg/pull/12264> >>> *being actively reviewed and worked on. >>> >>> Core: Fix numeric overflow of timestamp nano literal >>> <https://github.com/apache/iceberg/pull/11775> >>> * It causes the number overflow error for engines who have nano >>> precision long literal. Engines have to find workarounds currently. >>> Potential solutions are discussed in the PR. E.g. provide a public >>> method to construct a TimestampNanoLiteral so that engines/clients can use >>> it. >>> * I think it is a non-blocker for the 1.10 release. Treating the long >>> literal always as micro (current behavior) is not a correctness bug. It is >>> an important issue to be fixed so that engines can support nano timestamp >>> literal without going through the hoops. >>> >>> Thanks, >>> Steven >>> >>> On Wed, Jul 2, 2025 at 11:53 AM Szehon Ho <szehon.apa...@gmail.com> >>> wrote: >>> >>>> Thanks Steven for driving the release. >>>> >>>> I like to get in one more bug fix: >>>> https://github.com/apache/iceberg/pull/13448, it is a backport of >>>> https://github.com/apache/iceberg/pull/13435 (merged by Amogh) as I >>>> missed to do Spark 3.4, so also should be close. >>>> >>>> Thanks >>>> Szehon >>>> >>>> >>>> >>>> On Wed, Jul 2, 2025 at 11:17 AM Steven Wu <stevenz...@gmail.com> wrote: >>>> >>>>> During today's community sync meeting, there are 2 PRs that are >>>>> flagged as potential blockers. I have pushed other PRs to the next 1.11.0 >>>>> milestone. These 3 PRs are the only open items left in the 1.10.0 >>>>> milestone. >>>>> https://github.com/apache/iceberg/milestone/54 >>>>> >>>>> S3: Add LegacyMd5Plugin to S3 client builder >>>>> <https://github.com/apache/iceberg/pull/12264> >>>>> * Need community's help on reviewing this PR. Does the current >>>>> change solve the issue? >>>>> * If this still requires more significant work, we may want to roll >>>>> back the aws-sdk version (like we did for the 1.9.0 release) >>>>> >>>>> Core: Fix numeric overflow of timestamp nano literal >>>>> <https://github.com/apache/iceberg/pull/11775> >>>>> * The overflow error can be confusing. Is there any metadata or >>>>> correctness issue due to this bug? If not, we can consider it as a >>>>> non-blocker. >>>>> >>>>> Core, Spark: Propagate orphaned delete files when rewriting data files >>>>> <https://github.com/apache/iceberg/pull/13245> >>>>> * Very close. should be merged very soon >>>>> >>>>> It is not saying that we can't get other changes in the 1.10.0 >>>>> release. Just that these 3 PRs are considered potential blockers. >>>>> >>>>> Thanks, >>>>> Steven >>>>> >>>>> On Tue, Jul 1, 2025 at 10:38 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>>>> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> I'm updated the PR about multi-args transforms today, but not sure I >>>>>> will have reviews before 1.10.0. Let's try as best effort for 1.10, >>>>>> else we will include in 1.11. >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On Tue, Jul 1, 2025 at 6:42 PM Steven Wu <stevenz...@gmail.com> >>>>>> wrote: >>>>>> > >>>>>> > Hi, >>>>>> > >>>>>> > I plan to cut a release branch in the next 1 or 2 days. >>>>>> > >>>>>> > Waiting for this row lineage related PR (and its 3.4 backport >>>>>> afterwards) >>>>>> > https://github.com/apache/iceberg/pull/13070 >>>>>> > >>>>>> > Other items in the 1.10.0 milestone will probably have to be pushed >>>>>> to the next 1.11.0 release >>>>>> > https://github.com/apache/iceberg/milestone/54 >>>>>> > >>>>>> > Thanks, >>>>>> > Steven >>>>>> >>>>>
