> > Thanks for raising the issue! You are right that the version is always > 1 written by parquet-mr.
Last I checked at least Impala fails if the version is not set to 1 (not sure if there are other engines). On Fri, Mar 15, 2024 at 9:07 AM Gang Wu <[email protected]> wrote: > Hi Stephen, > > Thanks for raising the issue! You are right that the version is always > 1 written by parquet-mr. This is something we need to fix. However, > IMHO, the community does not have a clear answer on the definition > of parquet format v2. Which feature are you referring to specifically in > the version 2.6? It seems that you don't have to bother with the version > and just set the config to enable it. > > Best, > Gang > > On Fri, Mar 15, 2024 at 6:02 PM Stephen Colebourne <[email protected]> > wrote: > > > Hi all, > > I'm trying to use the parquet-mr library to set format_version=2.6 (or > > higher). > > > > When I review a file that is produced by the library, it appears that > > the version is set to 1.0. Looking at the code in > > org.apache.parquet.hadoop.ParquetFileWriter CURRENT_VERSION is hard > > coded to 1.0. > > > > Is it a bug to hard code the version there? Am I missing something > > obvious to select the format_version? > > > > thanks > > Stephen > > >
