Hi, All. After 2.4.5 RC1 vote failure, I asked your opinions about correctness/dataloss issues (at mailing lists/JIRAs/PRs) in order to collect the current status and public opinion widely in the community to build a consensus on this at this time.
Before talking about those issues, please remind that - Apache Spark 2.4.x is the only live version because 2.3.x is EOL and 3.0.0 is not released. - Apache Spark community has the following rule: "Correctness and data loss issues should be considered Blockers." Unfortunately, we didn't build a consensus on what is really blocked by that. In reality, it was just our resolution for the quality and it works a little differently. In this email, I want to talk about correctness/dataloss issues and observed public opinions. They fall into the following categories roughly. 1. Resolved in both 3.0.0 and 2.4.x - ex) SPARK-30447 Constant propagation nullability issue - No problem. However, this case sometimes goes to (2) 2. Resolved in both 3.0.0 and 2.4.x. But, reverted in 2.4.x later. - ex) SPARK-26021 -0.0 and 0.0 not treated consistently, doesn't match Hive - "We don't want to change the behavior in the maintenence release" 3. Resolved in 3.0.0 and not backported because this is 3.0.0-specific. - ex) SPARK-29906 Reading of csv file fails with adaptive execution turned on - No problem. 4. Resolved in 3.0.0 and not backported due to technical difficulty. - ex) SPARK-26154 Stream-stream joins - left outer join gives inconsistent output - "This is not backported due to the technical difficulty" 5. Resolved in 3.0.0 and not backported because this is not public API. - ex) SPARK-29503 MapObjects doesn't copy Unsafe data when nested under Safe data - "Since `catalyst` is not public, it's less worth backporting this." 6. Resolved in 3.0.0 and not backported because we forget since there was a no Target Version. - ex) SPARK-28375 Make pullupCorrelatedPredicate idempotent - "Adding the 'correctness' label so we remember to backport this fix to 2.4.x." - "This is possible, if users add the rule into postHocOptimizationBatches" 7. Open with Target Version 3.0.0. - ex) SPARK-29701 Correct behaviours of group analytical queries when empty input given - "We aren't fully SQL compliant there and I think that has been true since the beginning of spark sql" - "This is not a regression" 8. Open without Target Version. - I removed this case last week to give more visibility on them. Here, I want to focus that Apache Spark is a very healthy community because we have diverse opinions and reevaluating JIRA issues are the results of the community decision based on the discusson. I believe that it will go well eventually. In the above, I added those example JIRA IDs and the collected reasons just to give some colors to illustrate all cases are the real cases. There is no case to be blamed in the above. Although some JIRA issues will jump from one category into another category time to time, the categories will remain there. I want to propose a small additional work on `Target Version` to distinguish the above categories easily to communicate clearly in the community. This should be done by committers because we have the following policy on `Target Version`. "Target Version. This is assigned by committers to indicate a PR has been accepted for possible fix by the target version." Proposed Idea: A. To reduce the mismatch between `Target Version` vs `Affected Version`: When a committer set `correctness` or `data-loss` label, `Target Version` should be set together according to the `Affected Versions`. In case of the insufficient `Target Version` (e.g. `Target Version`=`3.0.0` for `Affected Version`=`2.4.4,3.0.0`), he/she need to add a comment on the JIRA. For example, "This is 3.0.0-specific issue" B. To reduce the mismatch between `Target Version` vs `Fixed Version`: When a committer resolve `correctness` or `data-loss` labeled issue, `Target Version` should be compared with `Fixed Version`. In case of the insufficient `Fixed Version` (e.g. `Target Version`=`2.4.4,3.0.0` and `Fixed Version`=`3.0.0`), he/she need to add a comment on the JIRA and adjust `Target Version` according to his/her decision. For example, "This is not backported due to the technical difficulty. I'll remove `2.4.4` from `Target Version`." With the above rules, the combination of `Affected Version` / `Target Version` / `Fixed Version` will serve us with much easier way in searching them, understanding categories, and discussing how to handle properly. Bests, Dongjoon.