emkornfield commented on PR #598: URL: https://github.com/apache/parquet-format/pull/598#issuecomment-4970348523
>> It potentially allows readers to read incorrect data. Consider the scenario of a new major version with 2 breaking changes. A reader implements the first of the features (the second one has been discussed yet). For the first feature the reader must "accept" the new major version (i.e. not fail). The reader is then released. The second feature is then added to the specification, this feature is not detectable by other information in the footer but changes how to interpret data on disk. When writers write out the full version, the old reader would silently return incorrect data. > My understanding is that the second feature would be classified as "forward incompatible" and thus could not be written to a new file without changing the embedded version / feature listing @alamb I think this only works with a version if you guarantee every forward incompatible feature has its own release (and therefore version). This was not intended as a strict requirement of this proposal. IMO two forward incompatible features could end up in the same release if they land close enough together or the community moves away from on-demand releases a regular time based cadence. This would not be a problem with "feature lists". Ideally, how we communicate versions at a physical level should not need to change if the release process changes. > This is true -- on the other hand writers could reduce the impact by emitting the lowest version number required to read all features in the file (rather than the most recent feature level the writer supports). The feature set could be restricted as necessary. Definitely, this is an alternative. I think for writers that want to be smart and pick the lowest version, it ends up being extra work. The feature list already needs to be tracked, but they then need to properly calculate the lowest possible version as well. > I agree this is more coarse grained than a feature-by-feature (or buckets of features) listing, but it is simpler to communicate and implement, which is why I think it is worthc onsidering Yes, this is a nice to have and version numbers could work here with the trade-off you mention. My perspective is colored by operating a multitenant service reading and writing parquet files with a wide variety of producers and consumers. The more things we can do to make parquet files compatible with old readers the better. My primary concern would be ensuring the first failure mode never happens. For communication at a user/debugging level I agree version number should always be present in the footer. If implementers don't want to be smart about tracking individual features then I think they could simply keep a map of Version number to feature list. This would not add a lot of complexity but bias implementations towards allowing finer grained compatibility. In other words, users should not need to think about the subtlety here but I think it is reasonable for implementators to do so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
