On Thu, 15 Jun 2023 at 19:08, Ian Cook <[email protected]> wrote: > > It will still be possible to write files using Parquet 2.4 by > explicitly specifying the 2.4 version to the Parquet writer, correct? > If yes, that provides a simple workaround for users who encounter > compatibility issues.
Indeed. Using the pyarrow API, it would be something like `pq.write_parquet(table, path, version="2.4")` > > However we should take care to document this as a potentially breaking > change, and document the workaround in release notes, release blog, > etc. Certainly! > > Ian > > On Thu, Jun 15, 2023 at 12:25 PM Joris Van den Bossche > <[email protected]> wrote: > > > > Hi all, > > > > Bringing up https://github.com/apache/arrow/issues/35746 to the > > mailing list: this issue proposes to bump the default Parquet version > > we use for writing to Parquet files in the C++ library (and in the > > various bindings including pyarrow and R arrow) from the current > > default of "2.4" to "2.6". > > > > In practice, the only change is that the writer will, by default, > > write the Timestamp LogicalType with NANOS unit > > (https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#timestamp) > > if your data uses timestamp("ns") (currently, such data gets coerced > > to microsecond resolution when writing to Parquet). > > > > In theory this could cause compatibility issues if the files you are > > writing need to be read by other Parquet implementations which don't > > yet support nanoseconds. But the Parquet format 2.6 was released in > > Sept 2018, and parquet-mr added support for it in 2018 as well. > > > > Unless there is pushback on this, we are currently planning to make > > this change for the upcoming Arrow 13.0.0 release. > > > > Best, > > Joris
