I will defer here as per Parquet community Parquet V2 encoding is not final yet so they haven’t make it official. I have no clue how pyarrow is supporting it ? I thought parquet used by pyarrow and Spark should have same flavor but unfortunately it is not 😞which is very concerning. Spark doesn’t support write in Parquet V2 and pyarrow do support in V2. Sent from my iPhone
> On Apr 23, 2024, at 11:23 AM, Gang Wu <ust...@gmail.com> wrote: > > I would expect so. parquet-mr has a complete implementation of all v2 > encodings > and some other Parquet implementations (e.g. Apache Arrow C++ and arrow-rs) > have already supported most (if not all) v2 encodings for a long time. > > Best, > Gang > >> On Tue, Apr 23, 2024 at 11:02 PM Prem Sahoo <prem.re...@gmail.com> wrote: >> >> Are we planning to put Parquet V2 encoding in 2.0 ? >> Sent from my iPhone >> >>> On Apr 23, 2024, at 10:31 AM, Xinli shang <sha...@uber.com.invalid> >> wrote: >>> >>> 4/23/2024 >>> >>> Attendee Fokko Driesprong, Vinoo Ganesh, Xinli Shang >>> >>> >>> Parquet-mr 1.14 release: >>> >>> 1. Fokko and Gang will discuss starting the release soon >>> >>> 2. There are a few breaking changes we need to make to ensure backward >>> compatibility and do proper testing >>> >>> 2. Vinoo will shadow and do some testing >>> >>> 3. Ideas on the release of Parquet 2.0. We start collecting thoughts and >>> welcome everybody to share opinions. >>> -- >>> Xinli Shang >>
