Following with some scenarios I just thought up to illustrate: Imagine a reader which knows version N encounters a file that contains a new feature.
If the feature is not breaking the footer and if the version in the footer is still N then the reader will ignore benign new features but fail if it needs to access the feature etc. Like today. If the feature is a breaking change to the footer then we’d better have bumped the version number in the footer and we need the reader to fail. It the feature is not breaking the footer but we bumped the version number anyway then the reader has to fail- there is no correct way to do graceful degradation. This final case is surprising but it’s useful to illustrate it by making up a scenario where the reader proceeding to parse a footer with a version number it doesn’t understand would corrupt data: The new feature in the file is a new kind of schema. The reader proceeds but doesn’t recognise the new schema so wrongly believes the columns being accessed are not present. Most engines using the reader are used to encountering parquet files with missing columns and interprets them as NULLs. Ergo not versioning the footer means we can’t do breaking changes to the footer itself, and if we commingle features and footer version in the same footer version then readers can’t distinguish and must fail. So my conclusion is that footer version we have in the footer thrift today should be the footer version, only bumped if we have breaking changes to the footer itself, and the current mechanism of thrift field ids continues to version the features used in the rest of the file. Which is what we have today?
