Thanks for the input folks. I agree we want to keep this discussion focused on parquet-java's use of parquet.thrift rather than the versioning cadence of parquet-format. I'll start a POC so we can discuss specifics. My thinking is that there should be some script to update the inlined version to match a commit hash in parquet-format. The main branch can track parquet-format commits without any issue (this will allow reference implementations for new features to actually get merged). I can look into a release-time check to ensure that released versions of parquet-java strictly track released versions of parquet.format.
-- Div On Fri, Jul 24, 2026 at 7:47 PM Julien Le Dem <[email protected]> wrote: > On Fri, Jul 24, 2026 at 10:30 AM Ryan Blue <[email protected]> wrote: > > > > I think we should also make more frequent releases of parquet-format. > > > > I'd like to keep this a separate topic from how we work with the thrift > > file in parquet-java so that we can move forward independently. We don't > > need to know how parquet-format is released to agree that we want to be > > able to build against newer structures without a formal release, just > like > > the other implementations do. > > > > Fair enough. > > > > > > > What is the mechanism to track what version of parquet-format, > > parquet-java > > depends on? > > > > Like the other projects, I don't think we track a specific version of the > > one from parquet-format, since we may pick changes that have not been > > released (as that's the pain point we're trying to solve). > > > > That said, I suggest that we sync the format into parquet-java when the > > upstream format is released, if not sooner. We should always use a > version > > that corresponds to a parquet-format commit to avoid drift. > > > > > Either in the main branch of parquet-java or at a minimum in releases of > parquet-java we should have an automated check that this file is in sync > with an official release of parquet-format. > > > > > Ryan > > > > On Fri, Jul 24, 2026 at 10:21 AM Julien Le Dem <[email protected]> > wrote: > > > > > Hello, > > > I'm strongly in favor of enabling a passing build on PRs like #3610 > > > What is the mechanism to track what version of parquet-format, > > > parquet-java depends on? Is there a check to track discrepancies or > > drift? > > > A > > > POC would help clarify this. > > > > > > Separately, I think we should also make more frequent releases of > > > parquet-format. > > > That could be formalized in our new versioning process. > > > In particular every time we merge a new change to the thrift file, we > > could > > > systematically create a new minor-patch release of Parquet format with > > the > > > addition in preview or experimental depending where we're at. > > > The documentation would clarify whether a new optional field is part of > > the > > > official version yet or not. > > > That would decouple publishing versioned artifacts and declaring > > something > > > part of the next official version. > > > (and independently of that we can either depend on a tagged version of > > the > > > file rather than the maven artifact) > > > > > > On Fri, Jul 24, 2026 at 7:42 AM Russell Spitzer < > > [email protected] > > > > > > > wrote: > > > > > > > +1 for all the reasons stated above. > > > > > > > > On Thu, Jul 23, 2026 at 10:21 PM Gang Wu <[email protected]> wrote: > > > > > > > > > I'm in favor of this proposal. > > > > > > > > > > PoCing a Java implementation in the current setup is painful. CI > > builds > > > > > will > > > > > always fail, and PRs cannot merge before releasing a new > > parquet-format > > > > > jar. > > > > > > > > > > Best, > > > > > Gang > > > > > > > > > > On Tue, Jul 21, 2026 at 8:46 PM Antoine Pitrou <[email protected] > > > > > > wrote: > > > > > > > > > > > Le 21/07/2026 à 14:05, Andrew Lamb a écrit : > > > > > > > As a maintainer of arrow-rs 's Parquet implementation, my > > > experience > > > > > has > > > > > > > been that our copy of parquet.thrift (rather than a dependency) > > has > > > > > > worked > > > > > > > great, and we have never had an issue with this scheme that I > can > > > > > > remember > > > > > > > > > > > > We do the same in Arrow C++ and never had any problem with it > > either. > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 7:19 AM Divjot Arora via dev < > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > >> Hi folks, > > > > > > >> > > > > > > >> The parquet-java project currently declares a dependency on a > > > pinned > > > > > > >> version of parquet-format and dynamically pulls in the > > > > parquet.thrift > > > > > > file > > > > > > >> for the pinned version. > > > > > > >> This makes it impossible to validate and merge PRs to support > > new > > > > > > features > > > > > > >> that have been merged into parquet-format but are not yet part > > of > > > an > > > > > > >> official parquet-format > > > > > > >> release (https://github.com/apache/parquet-java/pull/3610 is > an > > > > > > example). > > > > > > >> In contrast, the arrow-rs and arrow-cpp projects have a > vendored > > > > copy > > > > > of > > > > > > >> parquet.thrift in their > > > > > > >> repos, which allows them to bypass this problem. What do folks > > > think > > > > > > about > > > > > > >> adopting this vendored approach in parquet-java? > > > > > > >> > > > > > > >> -- Div Arora > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
