Thanks for the pointer! The `unknown-logical-type.parquet` file is very helpful.
I have written up a more detailed github issue on the forward compatibility issue: https://github.com/apache/parquet-format/issues/599 And I also opened a PR to fix in parquet-java: https://github.com/apache/parquet-java/pull/3669 Please take a look! Best, Kevin Liu On 2026/07/02 14:45:19 Ed Seidl wrote: > FWIW, there is a file "unknown-logical-type.parquet" in the parquet-testing > data directory. My version of parquet-cli (1.14.4) fails to read this file. > > Cheers, > Ed > > On 2026/07/02 05:14:58 Kevin Liu wrote: > > We've discussed this issue during the parquet sync today. > > > > To summarize: > > - We agreed that this is a bug and should be fixed. I'll put up a PR for > > it. > > - I proposed to backport the fix and create patch releases so older > > parquet-java versions can be fixed without being forced to upgrade to newer > > versions. > > - I’ll also do some follow-up testing across other popular implementations, > > since this behavior may not be limited to parquet-java. Will mentioned that > > Go and FastParquet will crash with new logical types. > > - There was a suggestion to add the “unknown logical type” case to to the > > parquet testing suite. > > > > Best, > > Kevin Liu > > > > > > On 2026/07/01 04:28:14 Kevin Liu wrote: > > > Thanks Micah! I wanted to confirm my understanding. > > > I would love to fix older readers so they can still open parquet files > > > with the variant type. Do you think its worth backporting a fix to older > > > readers as a patch release? > > > > > > I'll also do some research for other parquet clients based on when they > > > added support for the variant type and their corresponding behavior for > > > older readers. > > > > > > Best, > > > Kevin Liu > > > > > > On 2026/06/30 19:13:45 Micah Kornfield wrote: > > > > Hi Kevin, > > > > > > > > This sounds like a bug. > > > > > > > > if an older reader sees a logical type > > > > > it does not know how to decode, should it treat that logical > > > > > annotation as > > > > > absent, preserve the physical schema, and continue as long as the > > > > > unsupported column is not interpreted semantically? > > > > > > > > > > > > I think most implementations that support it, go even further and still > > > > allow the column to be projected. The main other thing that unknown > > > > logical types should do is not surface statistics (as that is a form of > > > > semantic interpretation). > > > > > > > > Cheers, > > > > Micah > > > > > > > > > > > > > > > > On Sat, Jun 27, 2026 at 11:01 AM Kevin Liu <[email protected]> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I wanted to ask for guidance on the expected behavior for older > > > > > parquet-java readers when reading a file that contains a newer logical > > > > > type. > > > > > > > > > > In PARQUET-3633 [1], parquet-java 1.15.x fails while opening a file > > > > > with a > > > > > VARIANT column, even when the read projection only requests a > > > > > separate id > > > > > column. The failure happens during footer schema conversion, before > > > > > projection is applied. I put up a small repro PR here [2]. > > > > > > > > > > VARIANT was added in parquet-format 2.12.0 and picked up by > > > > > parquet-java > > > > > starting in the 1.16 line. parquet-java 1.15.x uses parquet-format > > > > > 2.10.0, > > > > > so this is an older-reader/newer-logical-type compatibility case. I’m > > > > > looking at the parquet-format compatibility guidance [3], which calls > > > > > out > > > > > adding new logical types as a forward-compatible change. In this > > > > > case, I’d > > > > > expect parquet-java 1.15.x not to understand VARIANT semantics, but > > > > > still > > > > > to tolerate the annotation in the footer, preserve the physical > > > > > schema, and > > > > > allow reads that only project non-VARIANT columns. > > > > > > > > > > The behavior I’m wondering about is: if an older reader sees a > > > > > logical type > > > > > it does not know how to decode, should it treat that logical > > > > > annotation as > > > > > absent, preserve the physical schema, and continue as long as the > > > > > unsupported column is not interpreted semantically? That would allow > > > > > projected reads of unrelated columns to succeed without implying that > > > > > the > > > > > older reader understands VARIANT. > > > > > > > > > > I also sketched one possible fix/regression test here [4]. It treats > > > > > unknown LogicalType union fields as absent during footer conversion, > > > > > keeps > > > > > the physical schema usable, and verifies that reading only the > > > > > projected id > > > > > column succeeds for the shredded_variant/case-001.parquet fixture from > > > > > apache/parquet-testing. > > > > > > > > > > Does that match the intended forward-compatibility behavior for new > > > > > logical > > > > > types? > > > > > > > > > > Thanks, > > > > > > > > > > Kevin > > > > > > > > > > [1] https://github.com/apache/parquet-java/issues/3633 > > > > > [2] https://github.com/kevinjqliu/parquet-java/pull/1 > > > > > [3] > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement > > > > > [4] https://github.com/kevinjqliu/parquet-java/pull/2 > > > > > > > > > > > > > > >
