julienledem commented on code in PR #258: URL: https://github.com/apache/parquet-format/pull/258#discussion_r1674344985
########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). Review Comment: I agree with this section. Open source implementations are the first requirement. I think we should add that validation by proprietary engines is not required but welcome and useful. Third party vendors can publish reports of their analysis of the gains they observe in their proprietary non-open code base. As Parquet is an open source standard that helps using open source and non-opensource solutions together in the wider industry, I think it is important to cover both. (but open source is still the requirement as you describe above) ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). Review Comment: do we mean that we expect only bugfixes to be backported to older version? ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forward + compatibility (old readers can still read files). + +2. New encodings should be fully specified in this repository and ideally not + rely on an external dependencies for implementation (i.e. `parquet-format` is + the source of truth for the encoding). + +3. New compression mechanisms must have a pure Java implementation that can be + used as a dependency in `parquet-java`. + +### Releases + +The Parquet PMC aims to do releases of the format package only as needed when +new features are introduced. If multiple new features are being proposed +simultaneously some features might be consolidated into the same release. +Guidance is provided below on when implementations should enable features added +to the specification. Due to confusion in the past over Parquet versioning it +is not expected that there will be a 3.x release of the specification in the +foreseeable future. + +### Compatibility and Feature Enablement + +For the purposes of this discussion we classify features into the following buckets: + +1. Backward compatible. A file written under an older version of the format + should be readable under a newer version of the format. + +2. Forward compatible. A file written under a newer version of the format with + the feature enabled can be read under an older version of the format, but + some information might be missing or performance might be suboptimal. + +3. Forward incompatible. A file written under a newer version of the format with + the feature enabled cannot be read under an older version of the format (e.g. + adding and using a new compression algorithm). It is expected any feature in + this category will provide a signal to older readers, so they can + unambiguously determine that they cannot properly read the file (e.g. via + adding a new value to an existing enum). + +New features are intended to be widely beneficial to users of Parquet, and +therefore it is hoped third-party implementations will adopt them quickly after +they are introduced. It is assumed that writing new parts of the format, and +especially forward incompatible features, will be configured with a feature flag +defaulted to "off", and at some future point the feature is turned on by default +(reading of the new feature will typically be enabled without configuration or +defaulted to on). Some amount of lead time is desirable to ensure a critical +mass of Parquet implementations support a feature to avoid compatibility issues +across the ecosystem. Therefore, the Parquet PMC gives the following +recommendations for managing features: + +1. Backward compatibility is the concern of implementations but given the + ubiquity of Parquet and the length of time it has been used, libraries should + support reading older versions of the format to the greatest extent possible. + +2. Forward compatible features/changes may be enabled and used by default in + implementations once the parquet-format containing those changes has been + formally released. For features that may pose a significant performance + regression to older format readers, libaries should consider delaying default + enablement until 1 year after the release of the parquet-java implementation + that contains the feature implementation. + +3. Forward incompatible features/changes should not be turned on by default + until 2 years after the parquet-java implementation containing the feature is + released. It is recommended that changing the default value for a forward + incompatible feature flag should be clearly advertised to consumers (e.g. via + a major version release if using Semantic Versioning, or highlighed in + release notes). Review Comment: Using Major version release to advertise this is difficult. We are talking of a Parquet V3, but are we thinking to increment a new major version more often for the purpose of making a feature that already exists on by default? There is a dissonance between releasing new functionality that you would expect to make it a V3 even if it's not on by default yet and waiting until it's on by default to make it v3 when it's been available in the format for a while. There will be also confusion between the versioning of the format and the versioning of the library. I would suggest that we decouple enabling by default from the major versioning and instead rely on the release notes. ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forward + compatibility (old readers can still read files). + +2. New encodings should be fully specified in this repository and ideally not + rely on an external dependencies for implementation (i.e. `parquet-format` is + the source of truth for the encoding). + +3. New compression mechanisms must have a pure Java implementation that can be + used as a dependency in `parquet-java`. Review Comment: Is this true today? In the past we have relied on jni with a native lib properly wrapped in a jar (snappy, zstd, ...). ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forward + compatibility (old readers can still read files). Review Comment: I think we should be more prescriptive here. We can assume that users will have an environment with producers/consumers that do not get updated simultaneously. A few thoughts: When updgrading a producer, one should not unknowingly break consumers that have not been updated yet. If a new functionality makes the generated file not readable (ex: new encoding, new compression, ...), it should have a flag to turn it off by default. If it is purely additive (ex: add bloom filters) then it can be on by default. After a given amount of time it can be considered more widely adopted and be on by default. ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). Review Comment: There is always a trade-off in adding complexity to the format vs adding improvement in term of performance and/or compression. In that phase we should have a discussion about this. Often, a prototype helps confirm some of the assumptions. ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST Review Comment: The goal of phase 1 is to confirm this is a functionality (encoding, ...) we want to add to the format. The goal of phase 2 is to guarantee some level of completeness before we release. Being a file format, everything we release stays forever (or at least for a very long time). It would help to add titles to each section to clarify the goal. ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forward + compatibility (old readers can still read files). + +2. New encodings should be fully specified in this repository and ideally not + rely on an external dependencies for implementation (i.e. `parquet-format` is + the source of truth for the encoding). + +3. New compression mechanisms must have a pure Java implementation that can be + used as a dependency in `parquet-java`. + +### Releases + +The Parquet PMC aims to do releases of the format package only as needed when +new features are introduced. If multiple new features are being proposed +simultaneously some features might be consolidated into the same release. +Guidance is provided below on when implementations should enable features added +to the specification. Due to confusion in the past over Parquet versioning it +is not expected that there will be a 3.x release of the specification in the +foreseeable future. + +### Compatibility and Feature Enablement + +For the purposes of this discussion we classify features into the following buckets: + +1. Backward compatible. A file written under an older version of the format + should be readable under a newer version of the format. + +2. Forward compatible. A file written under a newer version of the format with + the feature enabled can be read under an older version of the format, but + some information might be missing or performance might be suboptimal. Review Comment: semantically we mean that we can read the data back without loss. ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). Review Comment: I agree that we shouldn't release changes to the format until we are convinced the change is stable. I think we need to figure out how we do this transition in a practical way. Implementing the functionality without some release of the metadata is a bit inconvenient. (this is a bit of an implementation detail, maybe we skip this discussion for now). ########## CONTRIBUTING.md: ########## @@ -29,3 +29,138 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long a comitter feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it sufficient to provide 2 working + implementations as outlined in step 2 or if demonstration of the feature with + a down-stream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in Arrow's DataSet library or Apache + Data Fusion or another open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [parquet-java](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [parquet-cpp](https://github.com/apache/arrow) or + [parquet-rs](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on parquet-site) are more likely + to be considered. If discussed as a requirement in step one, demonstration + of integration with a query engine is also required for this step. The + implementations must be made available publicly (e.g. as a pull request + against the target repository). + +Unless otherwise discussed, it is expected the implementations will develop from +the main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on the Parquet + mailing list to officially ratify the feature. After the vote passes the + format change is merged into the parquet-format repository and it is expected + the changes from step 2 will also be merged soon after. Before merging into + Parquet-java a parquet-format release must be performed. + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forwards + compatibility (old readers can still read files). + +2. New encodings should be fully specified in this repository and ideally not + rely on an external dependencies for implementation (i.e. `parquet-format` is + the source of truth for the encoding). + +3. New compression mechanisms must have a pure Java implementation that can be + used as dependency in parquet-java. + +### Releases + +The Parquet PMC aims to do releases of the format package only as needed when +new features are introduced. If multiple new features are being proposed +simultaneously some features might be consolidated into the same release. +Guidance is provided below on when implementations should enable features added +to the specification. Due to confusion in the past over parquet versioning it +is not expected that there will be a 3.x release of the specification in the +foreseeable future. + +### Compatibility and Feature Enablement + +For the purposes of this discussion we classify features into the following buckets: + +1. Backwards compatible. A file written under an older version of the format + should be readable under a newer version of the format. + +2. Forwards compatible. A file written under a newer version of the format with + the enabled feature can be read under an older version of the format, but + some information might be missing or performance might be suboptimal. + +3. Forward incompatible. A file written under a new version of the format with + the feature enabled cannot be read under an older version of the format (e.g. + Adding a new compression algorithm). + +New features are intended to be widely beneficial to users of Parquet, and +therefore it is hoped third-party implementations will adopt them quickly after +they are introduced. It is assumed that writing new parts of the format, and +especially forward incompatible features, will be configured with feature flag +defaulted to "off" and at some future point the features are turned on by default +(reading of the new feature will typically be enabled without configuration or +defaulted to on). Some amount of lead time is desirable to ensure a critical +mass of Parquet implementations support a feature to avoid compatability issues +across the ecosystem. Therefore, the Parquet PMC gives the following +recommendations for managing features: + +1. Backwards compatibility is the concern of implementations but given the + ubiquity of Parquet and the length of time it has been used, libraries should + support reading older version of the format to the greatest extent possible. + +2. Forward compatible features/changes may be used by default in implementations + once the parquet-format containing those changes has been formally released. + For features that may pose a significant performance regression to older + format readers, libaries should consider delaying default enablement until 1 + year after the release of the parquet-java implementation that contains the + feature implementation. + +3. Forwards incompatible features/changes should not be turned on by default + until 2 years after the parquet-java implementation containing the feature is Review Comment: What @emkornfield is saying [above](https://github.com/apache/parquet-format/pull/258/files#r1645520086) makes sense to me. We should always have new forward incompatible features off by default at first. Otherwise, this will cause breakages every time we add a new encoding (or similarly incompatible feature). How long is debatable. 2 years seems long to me. I don't think we should give a duration guidance to other implementers/users. We should document why we are doing this and explain the constraints. How long is very dependent on their circumstances. In an environment where there is a single system reading and writing Parquet files, they can enable it right away without problems. In a "data lakehouse" environment where people have multiple systems reading and writing they want to make sure they have upgraded other consumers before they turn it on. The more time passes, the more it becomes acceptable to have such new features on by default and ask the user to turn them off because they have some legacy system that can only read old files. We could have a setting to adjust this in environments where there is a single system reading/writing parquet: ```setForwardIncompatibleFeaturesOnByDefault(boolean)``` ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forward + compatibility (old readers can still read files). + +2. New encodings should be fully specified in this repository and ideally not + rely on an external dependencies for implementation (i.e. `parquet-format` is + the source of truth for the encoding). Review Comment: I would replace "idealy by something more specific. IE: If it relies on dependencies it should be fully specified in the format. (ex: zstandard, brotli, ...) ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it is sufficient to provide two working + implementations as outlined in step 2, or if demonstration of the feature with + a downstream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in the Apache Arrow C++ Dataset library, + the Apache DataFusion query engine, or any other open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step 1 above, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forward + compatibility (old readers can still read files). + +2. New encodings should be fully specified in this repository and ideally not + rely on an external dependencies for implementation (i.e. `parquet-format` is + the source of truth for the encoding). + +3. New compression mechanisms must have a pure Java implementation that can be + used as a dependency in `parquet-java`. + +### Releases + +The Parquet PMC aims to do releases of the format package only as needed when +new features are introduced. If multiple new features are being proposed +simultaneously some features might be consolidated into the same release. +Guidance is provided below on when implementations should enable features added +to the specification. Due to confusion in the past over Parquet versioning it +is not expected that there will be a 3.x release of the specification in the +foreseeable future. + +### Compatibility and Feature Enablement + +For the purposes of this discussion we classify features into the following buckets: + +1. Backward compatible. A file written under an older version of the format + should be readable under a newer version of the format. + +2. Forward compatible. A file written under a newer version of the format with + the feature enabled can be read under an older version of the format, but + some information might be missing or performance might be suboptimal. + +3. Forward incompatible. A file written under a newer version of the format with + the feature enabled cannot be read under an older version of the format (e.g. + adding and using a new compression algorithm). It is expected any feature in + this category will provide a signal to older readers, so they can + unambiguously determine that they cannot properly read the file (e.g. via + adding a new value to an existing enum). + +New features are intended to be widely beneficial to users of Parquet, and +therefore it is hoped third-party implementations will adopt them quickly after +they are introduced. It is assumed that writing new parts of the format, and +especially forward incompatible features, will be configured with a feature flag +defaulted to "off", and at some future point the feature is turned on by default +(reading of the new feature will typically be enabled without configuration or +defaulted to on). Some amount of lead time is desirable to ensure a critical +mass of Parquet implementations support a feature to avoid compatibility issues +across the ecosystem. Therefore, the Parquet PMC gives the following +recommendations for managing features: + +1. Backward compatibility is the concern of implementations but given the + ubiquity of Parquet and the length of time it has been used, libraries should + support reading older versions of the format to the greatest extent possible. + +2. Forward compatible features/changes may be enabled and used by default in + implementations once the parquet-format containing those changes has been + formally released. For features that may pose a significant performance + regression to older format readers, libaries should consider delaying default + enablement until 1 year after the release of the parquet-java implementation + that contains the feature implementation. + +3. Forward incompatible features/changes should not be turned on by default + until 2 years after the parquet-java implementation containing the feature is + released. It is recommended that changing the default value for a forward + incompatible feature flag should be clearly advertised to consumers (e.g. via + a major version release if using Semantic Versioning, or highlighed in + release notes). + +For forward compatible changes which have a high chance of performance +regression for older readers and forward incompatible changes, implementations +should clearly document the compatibility issues. Additionally, while it is up +to maintainers of individual implementations to make the best decision to serve +their ecosystem, they are encouraged to start enabling features by default along +the same timelines as `parquet-java`. Parquet-java will wait to enable features +by default until the most conservative timelines outlined above have been +exceeded. Review Comment: I think they should be encourage implementations to enable it as soon as they are comfortable with it and no later than the library unless they have legacy constraints. The conservative approach in parquet-java is because it has much less visible potential impact. Using the compatibility matrix to turn things on by default more aggressively when something is quickly adopted is a great idea in my opinion ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long as a committer feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part Review Comment: I think we need a template for Proposal docs. IMO Draft PRs can be a component of a proposal but they do not substitute to a proper explanation of what we are trying to achieve and why. ########## CONTRIBUTING.md: ########## @@ -29,3 +29,142 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +Note: This section applies to actual functional changes to the specification. +Fixing typos, grammar, and clarifying concepts that would not change the +semantics of the specification can be done as long a comitter feels comfortable +to merge them. When in doubt starting a discussion on the dev mailing list is +encouraged. + +The general steps for adding features to the format are as follows: + +1. Discuss changes on the developer mailing list ([email protected]). + Often times it is helpful to link to a draft pull request to make the + discussion concrete. This step is complete when there is lazy consensus. Part + of the consensus is whether it sufficient to provide 2 working + implementations as outlined in step 2 or if demonstration of the feature with + a down-stream query engine is necessary to justify the feature (e.g. + demonstrate performance improvements in Arrow's DataSet library or Apache + Data Fusion or another open source engine). + +2. Once a change has lazy consensus, two implementations of the feature + demonstrating interopability must also be provided. One implementation MUST + be [`parquet-java`](http://github.com/apache/parquet-java). It is preferred + that the second implementation be + [`parquet-cpp`](https://github.com/apache/arrow) or + [`parquet-rs`](https://github.com/apache/arrow-rs), however at the discretion + of the PMC any open source Parquet implementation may be acceptable. + Implementations whose contributors actively participate in the community + (e.g. keep their feature matrix up-to-date on the Parquet website) are more + likely to be considered. If discussed as a requirement in step one, + demonstration of integration with a query engine is also required for this + step. The implementations must be made available publicly, and they should be + fit for inclusion (for example, they were submitted as a pull request against + the target repository and committers gave positive reviews). + +Unless otherwise discussed, it is expected the implementations will be developed +from their respective main branch (i.e. backporting is not expected). + +3. After the first two steps are complete a formal vote is held on + [email protected] to officially ratify the feature. After the vote + passes the format change is merged into the `parquet-format` repository and + it is expected the changes from step 2 will also be merged soon after + (implementations should not be merged until the addition has been merged to + `parquet-format`). + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forward + compatibility (old readers can still read files). + +2. New encodings should be fully specified in this repository and ideally not + rely on an external dependencies for implementation (i.e. `parquet-format` is + the source of truth for the encoding). + +3. New compression mechanisms must have a pure Java implementation that can be + used as a dependency in `parquet-java`. + +### Releases + +The Parquet PMC aims to do releases of the format package only as needed when +new features are introduced. If multiple new features are being proposed +simultaneously some features might be consolidated into the same release. +Guidance is provided below on when implementations should enable features added +to the specification. Due to confusion in the past over Parquet versioning it +is not expected that there will be a 3.x release of the specification in the +foreseeable future. + +### Compatibility and Feature Enablement + +For the purposes of this discussion we classify features into the following buckets: + +1. Backward compatible. A file written under an older version of the format + should be readable under a newer version of the format. + +2. Forward compatible. A file written under a newer version of the format with + the feature enabled can be read under an older version of the format, but + some information might be missing or performance might be suboptimal. + +3. Forward incompatible. A file written under a newer version of the format with + the feature enabled cannot be read under an older version of the format (e.g. + adding and using a new compression algorithm). It is expected any feature in + this category will provide a signal to older readers, so they can + unambiguously determine that they cannot properly read the file (e.g. via + adding a new value to an existing enum). + +New features are intended to be widely beneficial to users of Parquet, and +therefore it is hoped third-party implementations will adopt them quickly after +they are introduced. It is assumed that writing new parts of the format, and +especially forward incompatible features, will be configured with a feature flag +defaulted to "off", and at some future point the feature is turned on by default +(reading of the new feature will typically be enabled without configuration or +defaulted to on). Some amount of lead time is desirable to ensure a critical +mass of Parquet implementations support a feature to avoid compatibility issues +across the ecosystem. Therefore, the Parquet PMC gives the following +recommendations for managing features: + +1. Backward compatibility is the concern of implementations but given the + ubiquity of Parquet and the length of time it has been used, libraries should + support reading older versions of the format to the greatest extent possible. + +2. Forward compatible features/changes may be enabled and used by default in + implementations once the parquet-format containing those changes has been + formally released. For features that may pose a significant performance + regression to older format readers, libaries should consider delaying default + enablement until 1 year after the release of the parquet-java implementation + that contains the feature implementation. + +3. Forward incompatible features/changes should not be turned on by default + until 2 years after the parquet-java implementation containing the feature is + released. It is recommended that changing the default value for a forward + incompatible feature flag should be clearly advertised to consumers (e.g. via + a major version release if using Semantic Versioning, or highlighed in + release notes). + +For forward compatible changes which have a high chance of performance +regression for older readers and forward incompatible changes, implementations +should clearly document the compatibility issues. Additionally, while it is up +to maintainers of individual implementations to make the best decision to serve +their ecosystem, they are encouraged to start enabling features by default along +the same timelines as `parquet-java`. Parquet-java will wait to enable features +by default until the most conservative timelines outlined above have been +exceeded. + +For features released prior to October 2024, target dates for each of these Review Comment: Am I correct in understanding that We intend to: - increment the major release of parquet-format when it has the definition of the new (forward incompatible) feature in it. - implement the above new feature in parquet-java under a minor release in the previous major but turning it off by default - increment the major release of parquet-java when we turn on the new encodings by default? I think this is not unsound but it might be confusing. We should then expect to have several major releases coming: one for the new footer and one for each new encoding. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
