Hi Prem, Wes' comment on the thread you posted on the arrow dev list should clear up your confusion: https://lists.apache.org/thread/72qwr66wf3xyrl5cozgojz88ct23qzxx. There is a difference between the "standard" itself (parquet-format) and the implementation (parquet-mr, etc...).
Parquet-format (https://github.com/apache/parquet-format) contains mostly just the docs and thrift definition now that a PR to clean up the remaining deprecated code was just merged. Releases of this just format, which again, is mostly just docs, is what Gang was referring to in [2]. We just started conversations about how a Parquet 2.0 release may look in the meeting yesterday. As these conversations progress, the dev list will be kept updated. On Wed, Apr 24, 2024 at 11:10 AM Prem Sahoo <prem.re...@gmail.com> wrote: > Hello Vinoo/Team, > As per pyarrow Team , They don't see any concern , please check below. > Please let us know *where it says Parquet V2 is not official * > > "> *As per Apache Parquet Community Parquet V2 is not final yet so it is > not > > official . They are advising not to use Parquet V2 for writing (though > code > > is available ) .* > > This would be news to me. Parquet releases are listed (by the parquet > community) at [1] > > The vote to release parquet 2.10 is here: [2] > > > *Neither of these links mention anything about this being an > experimental,unofficial, or non-finalized release.* > > I understand your concern. I believe your quotes are coming from your > discussion on the parquet mailing list here [3]. This communication is > unfortunate and confusing to me as well. > > [1] https://parquet.apache.org/blog/ > [2] https://lists.apache.org/thread/fdf1zz0f3xzz5zpvo6c811xjswhm1zy6 > [3] https://lists.apache.org/thread/4nzroc68czwxnp0ndqz15kp1vhcd7vg3"; > > > On Mon, Apr 22, 2024 at 4:56 PM Prem Sahoo <prem.re...@gmail.com> wrote: > >> Hello Vinoo/Team,. >> I was going through pyarrow and they have started using V2 as default . >> isn't it they should avoid it as it is not official. >> >> >> https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table >> >> version{“1.0”, “2.4”, “2.6”}, default “2.6” >> >> Determine which Parquet logical types are available for use, whether the >> reduced set from the Parquet 1.x.x format or the expanded logical types >> added in later format versions. Files written with version=’2.4’ or ‘2.6’ >> may not be readable in all Parquet implementations, so version=’1.0’ is >> likely the choice that maximizes file compatibility. UINT32 and some >> logical types are only available with version ‘2.4’. Nanosecond timestamps >> are only available with version ‘2.6’. Other features such as compression >> algorithms or the new serialized data page format must be enabled >> separately (see ‘compression’ and ‘data_page_version’). >> >
