Thanks Gang, I had the same thought, and I think it can work. However, in practice, I find submodules to be clunky and user-unfriendly. For example, you need to run additional git commands to fetch the submodules. Sometimes it is stuck on an older version, and I always find it awkward to work with.
Another possibility I suggested in the PR itself is to use nightly snapshots. What I like is that we don't duplicate the thrift definitions in the Java repo, and we can work on something that's unreleased but on the parquet-format main branch. I noticed that we stopped producing nightly snapshots a while ago. I created a PR to reinstate that [1]. On the PR, we ended up on the fact that working from snapshots isn't as flexible as Div's setup, where you can make the changes in the same PR. At all costs, I think we want to avoid forcing folks to write Perl scripts to amend the definitions [2] 🤣. I believe inlining is the most flexible approach, since it makes the changes transparent. When releasing, we should double-check that the current version of the definitions matches the latest released version of parquet-format. Kind regards, Fokko [1] https://github.com/apache/parquet-format/pull/614 [2] https://github.com/apache/parquet-java/pull/3397/changes#diff-07fef3853dd3df8f18fa5b6192e36d1f60a081591a529cf0c3558ab4c093322fR69-R97 On 2026/09/03 02:16:42 Gang Wu wrote: > Ah sorry, my reply above is **NOT** blocking > > On Thu, Sep 3, 2026 at 10:16 AM Gang Wu <[email protected]> wrote: > > > I'm in favor of inlining parquet.thrift overall so my reply below is > > blocking. > > > > I'm not sure if we have explored making parquet-format as a git submodule > > in the parquet-java. Then we can point to a commit hash without any script > > to > > validate the thrift file. It would also be easy to validate if that commit > > belongs to a release tag when releasing parquet-java. > > > > On Thu, Sep 3, 2026 at 6:29 AM Ryan Blue <[email protected]> wrote: > > > >> I just approved the PR that Div opened (linked in his email). > >> > >> Right now, the PR updates the build to use a local copy of the thrift file > >> and adds a script to download new copies from parquet-format. New copies > >> are specified by commit hash or by a ref, and the resolved version is > >> tracked in a parquet-format.version file. > >> > >> The script makes it easy to update to the latest copy, but rather than > >> having scripts to check and diff the local copy I suggested that we use > >> `git` instead. Since the file is version controlled, we have very good > >> tools to view changes and history already. > >> > >> I'd like to merge this soon, unless there are objections. Please take a > >> look at the PR if you're interested. > >> > >> Ryan > >> > >> On Wed, Aug 5, 2026 at 2:02 PM Divjot Arora via dev < > >> [email protected]> > >> wrote: > >> > >> > I've put up a PR [1] to inline parquet.thrift into parquet-java and > >> > remove the upstream > >> > dependency on parquet-format. There are some scripts to help manage > >> > upgrading the inlined > >> > copy as well as to check whether it references a released parquet-format > >> > version. Please take > >> > a look if interested, I look forward to any feedback. > >> > > >> > Best, > >> > Div > >> > > >> > [1] https://github.com/apache/parquet-java/pull/3709 > >> > > >> > On Mon, Jul 27, 2026 at 12:25 PM Divjot Arora <[email protected] > >> > > >> > wrote: > >> > > >> > > Thanks for the input folks. I agree we want to keep this discussion > >> > > focused on > >> > > parquet-java's use of parquet.thrift rather than the versioning > >> cadence > >> > of > >> > > parquet-format. > >> > > I'll start a POC so we can discuss specifics. My thinking is that > >> there > >> > > should be some > >> > > script to update the inlined version to match a commit hash in > >> > > parquet-format. The main > >> > > branch can track parquet-format commits without any issue (this will > >> > allow > >> > > reference > >> > > implementations for new features to actually get merged). I can look > >> into > >> > > a release-time > >> > > check to ensure that released versions of parquet-java strictly track > >> > > released versions of > >> > > parquet.format. > >> > > > >> > > -- Div > >> > > > >> > > On Fri, Jul 24, 2026 at 7:47 PM Julien Le Dem <[email protected]> > >> wrote: > >> > > > >> > >> On Fri, Jul 24, 2026 at 10:30 AM Ryan Blue <[email protected]> wrote: > >> > >> > >> > >> > > I think we should also make more frequent releases of > >> > parquet-format. > >> > >> > > >> > >> > I'd like to keep this a separate topic from how we work with the > >> > thrift > >> > >> > file in parquet-java so that we can move forward independently. We > >> > don't > >> > >> > need to know how parquet-format is released to agree that we want > >> to > >> > be > >> > >> > able to build against newer structures without a formal release, > >> just > >> > >> like > >> > >> > the other implementations do. > >> > >> > > >> > >> > >> > >> Fair enough. > >> > >> > >> > >> > >> > >> > > >> > >> > > What is the mechanism to track what version of parquet-format, > >> > >> > parquet-java > >> > >> > depends on? > >> > >> > > >> > >> > Like the other projects, I don't think we track a specific version > >> of > >> > >> the > >> > >> > one from parquet-format, since we may pick changes that have not > >> been > >> > >> > released (as that's the pain point we're trying to solve). > >> > >> > > >> > >> > That said, I suggest that we sync the format into parquet-java when > >> > the > >> > >> > upstream format is released, if not sooner. We should always use a > >> > >> version > >> > >> > that corresponds to a parquet-format commit to avoid drift. > >> > >> > > >> > >> > > >> > >> Either in the main branch of parquet-java or at a minimum in > >> releases of > >> > >> parquet-java we should have an automated check that this file is in > >> sync > >> > >> with an official release of parquet-format. > >> > >> > >> > >> > >> > >> > >> > >> > Ryan > >> > >> > > >> > >> > On Fri, Jul 24, 2026 at 10:21 AM Julien Le Dem <[email protected]> > >> > >> wrote: > >> > >> > > >> > >> > > Hello, > >> > >> > > I'm strongly in favor of enabling a passing build on PRs like > >> #3610 > >> > >> > > What is the mechanism to track what version of parquet-format, > >> > >> > > parquet-java depends on? Is there a check to track discrepancies > >> or > >> > >> > drift? > >> > >> > > A > >> > >> > > POC would help clarify this. > >> > >> > > > >> > >> > > Separately, I think we should also make more frequent releases of > >> > >> > > parquet-format. > >> > >> > > That could be formalized in our new versioning process. > >> > >> > > In particular every time we merge a new change to the thrift > >> file, > >> > we > >> > >> > could > >> > >> > > systematically create a new minor-patch release of Parquet format > >> > with > >> > >> > the > >> > >> > > addition in preview or experimental depending where we're at. > >> > >> > > The documentation would clarify whether a new optional field is > >> part > >> > >> of > >> > >> > the > >> > >> > > official version yet or not. > >> > >> > > That would decouple publishing versioned artifacts and declaring > >> > >> > something > >> > >> > > part of the next official version. > >> > >> > > (and independently of that we can either depend on a tagged > >> version > >> > of > >> > >> > the > >> > >> > > file rather than the maven artifact) > >> > >> > > > >> > >> > > On Fri, Jul 24, 2026 at 7:42 AM Russell Spitzer < > >> > >> > [email protected] > >> > >> > > > > >> > >> > > wrote: > >> > >> > > > >> > >> > > > +1 for all the reasons stated above. > >> > >> > > > > >> > >> > > > On Thu, Jul 23, 2026 at 10:21 PM Gang Wu <[email protected]> > >> > wrote: > >> > >> > > > > >> > >> > > > > I'm in favor of this proposal. > >> > >> > > > > > >> > >> > > > > PoCing a Java implementation in the current setup is > >> painful. CI > >> > >> > builds > >> > >> > > > > will > >> > >> > > > > always fail, and PRs cannot merge before releasing a new > >> > >> > parquet-format > >> > >> > > > > jar. > >> > >> > > > > > >> > >> > > > > Best, > >> > >> > > > > Gang > >> > >> > > > > > >> > >> > > > > On Tue, Jul 21, 2026 at 8:46 PM Antoine Pitrou < > >> > >> [email protected]> > >> > >> > > > wrote: > >> > >> > > > > > >> > >> > > > > > Le 21/07/2026 à 14:05, Andrew Lamb a écrit : > >> > >> > > > > > > As a maintainer of arrow-rs 's Parquet implementation, my > >> > >> > > experience > >> > >> > > > > has > >> > >> > > > > > > been that our copy of parquet.thrift (rather than a > >> > >> dependency) > >> > >> > has > >> > >> > > > > > worked > >> > >> > > > > > > great, and we have never had an issue with this scheme > >> that > >> > I > >> > >> can > >> > >> > > > > > remember > >> > >> > > > > > > >> > >> > > > > > We do the same in Arrow C++ and never had any problem with > >> it > >> > >> > either. > >> > >> > > > > > > >> > >> > > > > > Regards > >> > >> > > > > > > >> > >> > > > > > Antoine. > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > > >> > >> > > > > > > Andrew > >> > >> > > > > > > > >> > >> > > > > > > On Tue, Jul 21, 2026 at 7:19 AM Divjot Arora via dev < > >> > >> > > > > > [email protected]> > >> > >> > > > > > > wrote: > >> > >> > > > > > > > >> > >> > > > > > >> Hi folks, > >> > >> > > > > > >> > >> > >> > > > > > >> The parquet-java project currently declares a dependency > >> > on a > >> > >> > > pinned > >> > >> > > > > > >> version of parquet-format and dynamically pulls in the > >> > >> > > > parquet.thrift > >> > >> > > > > > file > >> > >> > > > > > >> for the pinned version. > >> > >> > > > > > >> This makes it impossible to validate and merge PRs to > >> > support > >> > >> > new > >> > >> > > > > > features > >> > >> > > > > > >> that have been merged into parquet-format but are not > >> yet > >> > >> part > >> > >> > of > >> > >> > > an > >> > >> > > > > > >> official parquet-format > >> > >> > > > > > >> release ( > >> https://github.com/apache/parquet-java/pull/3610 > >> > >> is an > >> > >> > > > > > example). > >> > >> > > > > > >> In contrast, the arrow-rs and arrow-cpp projects have a > >> > >> vendored > >> > >> > > > copy > >> > >> > > > > of > >> > >> > > > > > >> parquet.thrift in their > >> > >> > > > > > >> repos, which allows them to bypass this problem. What do > >> > >> folks > >> > >> > > think > >> > >> > > > > > about > >> > >> > > > > > >> adopting this vendored approach in parquet-java? > >> > >> > > > > > >> > >> > >> > > > > > >> -- Div Arora > >> > >> > > > > > >> > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > > > >> > > >> > > >
