Hello,
I think it might be useful to have a comparison point for the versioning
discussion we're having.
In the Arrow IPC format there are two versioning mechanisms:
1. A "metadata version", which records semantically breaking changes to
the IPC metadata format (such as: a change of layout for some datatype).
The metadata version (currently V5) is recorded in the
Flatbuffers-serialized metadata, such that a reader can reject a file
with an unsupported metadata version.
2. A "format version", which records non-breaking feature additions,
such as new Arrow datatypes. The format version (currently 1.5) is *not*
recorded in the Flatbuffers-serialized metadata, and this doesn't seem
to have bothered anyone until now.
Of course, Parquet is not Arrow, and our constraints and use cases are
different. For example, Parquet is much more often used for long-term
storage and publishing of datasets, while the Arrow IPC format is better
suited for short-term transmission and storage (caches etc.).
Regards
Antoine.
