Ah sorry, my reply above is **NOT** blocking On Thu, Sep 3, 2026 at 10:16 AM Gang Wu <[email protected]> wrote:
> I'm in favor of inlining parquet.thrift overall so my reply below is > blocking. > > I'm not sure if we have explored making parquet-format as a git submodule > in the parquet-java. Then we can point to a commit hash without any script > to > validate the thrift file. It would also be easy to validate if that commit > belongs to a release tag when releasing parquet-java. > > On Thu, Sep 3, 2026 at 6:29 AM Ryan Blue <[email protected]> wrote: > >> I just approved the PR that Div opened (linked in his email). >> >> Right now, the PR updates the build to use a local copy of the thrift file >> and adds a script to download new copies from parquet-format. New copies >> are specified by commit hash or by a ref, and the resolved version is >> tracked in a parquet-format.version file. >> >> The script makes it easy to update to the latest copy, but rather than >> having scripts to check and diff the local copy I suggested that we use >> `git` instead. Since the file is version controlled, we have very good >> tools to view changes and history already. >> >> I'd like to merge this soon, unless there are objections. Please take a >> look at the PR if you're interested. >> >> Ryan >> >> On Wed, Aug 5, 2026 at 2:02 PM Divjot Arora via dev < >> [email protected]> >> wrote: >> >> > I've put up a PR [1] to inline parquet.thrift into parquet-java and >> > remove the upstream >> > dependency on parquet-format. There are some scripts to help manage >> > upgrading the inlined >> > copy as well as to check whether it references a released parquet-format >> > version. Please take >> > a look if interested, I look forward to any feedback. >> > >> > Best, >> > Div >> > >> > [1] https://github.com/apache/parquet-java/pull/3709 >> > >> > On Mon, Jul 27, 2026 at 12:25 PM Divjot Arora <[email protected] >> > >> > wrote: >> > >> > > Thanks for the input folks. I agree we want to keep this discussion >> > > focused on >> > > parquet-java's use of parquet.thrift rather than the versioning >> cadence >> > of >> > > parquet-format. >> > > I'll start a POC so we can discuss specifics. My thinking is that >> there >> > > should be some >> > > script to update the inlined version to match a commit hash in >> > > parquet-format. The main >> > > branch can track parquet-format commits without any issue (this will >> > allow >> > > reference >> > > implementations for new features to actually get merged). I can look >> into >> > > a release-time >> > > check to ensure that released versions of parquet-java strictly track >> > > released versions of >> > > parquet.format. >> > > >> > > -- Div >> > > >> > > On Fri, Jul 24, 2026 at 7:47 PM Julien Le Dem <[email protected]> >> wrote: >> > > >> > >> On Fri, Jul 24, 2026 at 10:30 AM Ryan Blue <[email protected]> wrote: >> > >> >> > >> > > I think we should also make more frequent releases of >> > parquet-format. >> > >> > >> > >> > I'd like to keep this a separate topic from how we work with the >> > thrift >> > >> > file in parquet-java so that we can move forward independently. We >> > don't >> > >> > need to know how parquet-format is released to agree that we want >> to >> > be >> > >> > able to build against newer structures without a formal release, >> just >> > >> like >> > >> > the other implementations do. >> > >> > >> > >> >> > >> Fair enough. >> > >> >> > >> >> > >> > >> > >> > > What is the mechanism to track what version of parquet-format, >> > >> > parquet-java >> > >> > depends on? >> > >> > >> > >> > Like the other projects, I don't think we track a specific version >> of >> > >> the >> > >> > one from parquet-format, since we may pick changes that have not >> been >> > >> > released (as that's the pain point we're trying to solve). >> > >> > >> > >> > That said, I suggest that we sync the format into parquet-java when >> > the >> > >> > upstream format is released, if not sooner. We should always use a >> > >> version >> > >> > that corresponds to a parquet-format commit to avoid drift. >> > >> > >> > >> > >> > >> Either in the main branch of parquet-java or at a minimum in >> releases of >> > >> parquet-java we should have an automated check that this file is in >> sync >> > >> with an official release of parquet-format. >> > >> >> > >> >> > >> >> > >> > Ryan >> > >> > >> > >> > On Fri, Jul 24, 2026 at 10:21 AM Julien Le Dem <[email protected]> >> > >> wrote: >> > >> > >> > >> > > Hello, >> > >> > > I'm strongly in favor of enabling a passing build on PRs like >> #3610 >> > >> > > What is the mechanism to track what version of parquet-format, >> > >> > > parquet-java depends on? Is there a check to track discrepancies >> or >> > >> > drift? >> > >> > > A >> > >> > > POC would help clarify this. >> > >> > > >> > >> > > Separately, I think we should also make more frequent releases of >> > >> > > parquet-format. >> > >> > > That could be formalized in our new versioning process. >> > >> > > In particular every time we merge a new change to the thrift >> file, >> > we >> > >> > could >> > >> > > systematically create a new minor-patch release of Parquet format >> > with >> > >> > the >> > >> > > addition in preview or experimental depending where we're at. >> > >> > > The documentation would clarify whether a new optional field is >> part >> > >> of >> > >> > the >> > >> > > official version yet or not. >> > >> > > That would decouple publishing versioned artifacts and declaring >> > >> > something >> > >> > > part of the next official version. >> > >> > > (and independently of that we can either depend on a tagged >> version >> > of >> > >> > the >> > >> > > file rather than the maven artifact) >> > >> > > >> > >> > > On Fri, Jul 24, 2026 at 7:42 AM Russell Spitzer < >> > >> > [email protected] >> > >> > > > >> > >> > > wrote: >> > >> > > >> > >> > > > +1 for all the reasons stated above. >> > >> > > > >> > >> > > > On Thu, Jul 23, 2026 at 10:21 PM Gang Wu <[email protected]> >> > wrote: >> > >> > > > >> > >> > > > > I'm in favor of this proposal. >> > >> > > > > >> > >> > > > > PoCing a Java implementation in the current setup is >> painful. CI >> > >> > builds >> > >> > > > > will >> > >> > > > > always fail, and PRs cannot merge before releasing a new >> > >> > parquet-format >> > >> > > > > jar. >> > >> > > > > >> > >> > > > > Best, >> > >> > > > > Gang >> > >> > > > > >> > >> > > > > On Tue, Jul 21, 2026 at 8:46 PM Antoine Pitrou < >> > >> [email protected]> >> > >> > > > wrote: >> > >> > > > > >> > >> > > > > > Le 21/07/2026 à 14:05, Andrew Lamb a écrit : >> > >> > > > > > > As a maintainer of arrow-rs 's Parquet implementation, my >> > >> > > experience >> > >> > > > > has >> > >> > > > > > > been that our copy of parquet.thrift (rather than a >> > >> dependency) >> > >> > has >> > >> > > > > > worked >> > >> > > > > > > great, and we have never had an issue with this scheme >> that >> > I >> > >> can >> > >> > > > > > remember >> > >> > > > > > >> > >> > > > > > We do the same in Arrow C++ and never had any problem with >> it >> > >> > either. >> > >> > > > > > >> > >> > > > > > Regards >> > >> > > > > > >> > >> > > > > > Antoine. >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > >> > >> > > > > > > Andrew >> > >> > > > > > > >> > >> > > > > > > On Tue, Jul 21, 2026 at 7:19 AM Divjot Arora via dev < >> > >> > > > > > [email protected]> >> > >> > > > > > > wrote: >> > >> > > > > > > >> > >> > > > > > >> Hi folks, >> > >> > > > > > >> >> > >> > > > > > >> The parquet-java project currently declares a dependency >> > on a >> > >> > > pinned >> > >> > > > > > >> version of parquet-format and dynamically pulls in the >> > >> > > > parquet.thrift >> > >> > > > > > file >> > >> > > > > > >> for the pinned version. >> > >> > > > > > >> This makes it impossible to validate and merge PRs to >> > support >> > >> > new >> > >> > > > > > features >> > >> > > > > > >> that have been merged into parquet-format but are not >> yet >> > >> part >> > >> > of >> > >> > > an >> > >> > > > > > >> official parquet-format >> > >> > > > > > >> release ( >> https://github.com/apache/parquet-java/pull/3610 >> > >> is an >> > >> > > > > > example). >> > >> > > > > > >> In contrast, the arrow-rs and arrow-cpp projects have a >> > >> vendored >> > >> > > > copy >> > >> > > > > of >> > >> > > > > > >> parquet.thrift in their >> > >> > > > > > >> repos, which allows them to bypass this problem. What do >> > >> folks >> > >> > > think >> > >> > > > > > about >> > >> > > > > > >> adopting this vendored approach in parquet-java? >> > >> > > > > > >> >> > >> > > > > > >> -- Div Arora >> > >> > > > > > >> >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > > >> > >> >
