Thanks Steven, I agree that we can defer the timestamp nano fix, I had to write some tests to prove it to myself but every long microseconds or string timestamp value that's within range would always fail with an exception when converting to nanos; so I think we're OK as is though that is a long standing issue that'd be good to address to unblock Trino on supporting the new precision.
I also added the Spark 4.0 row lineage support PR <https://github.com/apache/iceberg/pull/13310> to the milestone since folks who may be upgrading from 3.4/3.5 to 4.0 would go from a system which is correctly capturing lineage information to a system which will just start dropping those. My biggest concern is that for these users, this is effectively a regression where they would stop trusting lineage data. I've asked folks to take a look, and encourage everyone to leave their feedback on the PR so I can quickly address it and we can move forward with the release. Thanks, Amogh Jahagirdar On Mon, Jul 7, 2025 at 1:07 PM Steven Wu <stevenz...@gmail.com> wrote: > Updates for the remaining 3 open PRs for 1.10.0 milestone > https://github.com/apache/iceberg/milestone/54 > > Core: Keep track of data files to be removed for orphaned DV detection > <https://github.com/apache/iceberg/pull/13222> > * got 2 approvals. should be merged in a day or two > > Spark 4.0: Migrate Iceberg Stored Procedures to Spark built-in > implementations <https://github.com/apache/iceberg/pull/13106> > * Got 3 approvals. expect it to be merged in a day or two > > Core: Fix numeric overflow of timestamp nano literal > <https://github.com/apache/iceberg/pull/11775> > * It breaks the use case of long literal value for nano timestamp column. > But there is no correctness issue. Hence I am favoring moving it out of the > 1.10.0 milestone > * there is no consensus on the path forward yet. > > On Thu, Jul 3, 2025 at 2:28 PM Szehon Ho <szehon.apa...@gmail.com> wrote: > >> Thanks Steven! >> >> >> On Jul 3, 2025, at 1:56 PM, Steven Wu <stevenz...@gmail.com> wrote: >> >> >> > because it makes a bit of backward incompatible change by using the >> Spark behavior (for argument case sensitivity and type coercion support), >> we should do it in 1.10 release as we will have Spark 4.0 as part of it >> iiuc. Else 1.11 will be backward incompatible for the Spark 4.0 jar. >> >> Szehon, thanks for bringing up PR 13106. I agree with your argument here. >> It will be great to include this in the 1.10.0 release to avoid a behavior >> change later because Spark 4.0 support was released in this version. It >> seems close as you already approved it. Can we get a couple more approvals? >> >> >> >> On Thu, Jul 3, 2025 at 1:37 PM Szehon Ho <szehon.apa...@gmail.com> wrote: >> >>> Hi Steven >>> >>> Thanks. One more, what do we think about having >>> https://github.com/apache/iceberg/pull/13106/ as part of 1.10 release? >>> It's migrating Spark procedure to use Spark 4's new DSV2 Procedure >>> framework. >>> >>> Its not a blocker but I feel, because it makes a bit of backward >>> incompatible change by using the Spark behavior (for argument case >>> sensitivity and type coercion support), we should do it in 1.10 release as >>> we will have Spark 4.0 as part of it iiuc. Else 1.11 will be backward >>> incompatible for the Spark 4.0 jar. >>> >>> Thanks, >>> Szehon >>> >>> >>> >>> On Thu, Jul 3, 2025 at 1:17 PM Steven Wu <stevenz...@gmail.com> wrote: >>> >>>> Szehon's backport PR has been merged. Another blocker (dangling DVs for >>>> rewrite) was also merged. >>>> Core, Spark: Propagate orphaned delete files when rewriting data files >>>> <https://github.com/apache/iceberg/pull/13245> >>>> >>>> There are still 3 blockers left, as a new PR (also danglinge DV >>>> related) is identified as a blocker. >>>> >>>> Core: Keep track of data files to be removed for orphaned DV detection >>>> <https://github.com/apache/iceberg/pull/13222> >>>> * being actively reviewed and worked on. >>>> >>>> S3: Add LegacyMd5Plugin to S3 client builder >>>> <https://github.com/apache/iceberg/pull/12264> >>>> *being actively reviewed and worked on. >>>> >>>> Core: Fix numeric overflow of timestamp nano literal >>>> <https://github.com/apache/iceberg/pull/11775> >>>> * It causes the number overflow error for engines who have nano >>>> precision long literal. Engines have to find workarounds currently. >>>> Potential solutions are discussed in the PR. E.g. provide a public >>>> method to construct a TimestampNanoLiteral so that engines/clients can use >>>> it. >>>> * I think it is a non-blocker for the 1.10 release. Treating the long >>>> literal always as micro (current behavior) is not a correctness bug. It is >>>> an important issue to be fixed so that engines can support nano timestamp >>>> literal without going through the hoops. >>>> >>>> Thanks, >>>> Steven >>>> >>>> On Wed, Jul 2, 2025 at 11:53 AM Szehon Ho <szehon.apa...@gmail.com> >>>> wrote: >>>> >>>>> Thanks Steven for driving the release. >>>>> >>>>> I like to get in one more bug fix: >>>>> https://github.com/apache/iceberg/pull/13448, it is a backport of >>>>> https://github.com/apache/iceberg/pull/13435 (merged by Amogh) as I >>>>> missed to do Spark 3.4, so also should be close. >>>>> >>>>> Thanks >>>>> Szehon >>>>> >>>>> >>>>> >>>>> On Wed, Jul 2, 2025 at 11:17 AM Steven Wu <stevenz...@gmail.com> >>>>> wrote: >>>>> >>>>>> During today's community sync meeting, there are 2 PRs that are >>>>>> flagged as potential blockers. I have pushed other PRs to the next 1.11.0 >>>>>> milestone. These 3 PRs are the only open items left in the 1.10.0 >>>>>> milestone. >>>>>> https://github.com/apache/iceberg/milestone/54 >>>>>> >>>>>> S3: Add LegacyMd5Plugin to S3 client builder >>>>>> <https://github.com/apache/iceberg/pull/12264> >>>>>> * Need community's help on reviewing this PR. Does the current >>>>>> change solve the issue? >>>>>> * If this still requires more significant work, we may want to roll >>>>>> back the aws-sdk version (like we did for the 1.9.0 release) >>>>>> >>>>>> Core: Fix numeric overflow of timestamp nano literal >>>>>> <https://github.com/apache/iceberg/pull/11775> >>>>>> * The overflow error can be confusing. Is there any metadata or >>>>>> correctness issue due to this bug? If not, we can consider it as a >>>>>> non-blocker. >>>>>> >>>>>> Core, Spark: Propagate orphaned delete files when rewriting data files >>>>>> <https://github.com/apache/iceberg/pull/13245> >>>>>> * Very close. should be merged very soon >>>>>> >>>>>> It is not saying that we can't get other changes in the 1.10.0 >>>>>> release. Just that these 3 PRs are considered potential blockers. >>>>>> >>>>>> Thanks, >>>>>> Steven >>>>>> >>>>>> On Tue, Jul 1, 2025 at 10:38 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>>>>> wrote: >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I'm updated the PR about multi-args transforms today, but not sure I >>>>>>> will have reviews before 1.10.0. Let's try as best effort for 1.10, >>>>>>> else we will include in 1.11. >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>> On Tue, Jul 1, 2025 at 6:42 PM Steven Wu <stevenz...@gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > Hi, >>>>>>> > >>>>>>> > I plan to cut a release branch in the next 1 or 2 days. >>>>>>> > >>>>>>> > Waiting for this row lineage related PR (and its 3.4 backport >>>>>>> afterwards) >>>>>>> > https://github.com/apache/iceberg/pull/13070 >>>>>>> > >>>>>>> > Other items in the 1.10.0 milestone will probably have to be >>>>>>> pushed to the next 1.11.0 release >>>>>>> > https://github.com/apache/iceberg/milestone/54 >>>>>>> > >>>>>>> > Thanks, >>>>>>> > Steven >>>>>>> >>>>>>
