Hi, All.
After 2.4.5 RC1 vote failure, I asked your opinions about
correctness/dataloss issues (at mailing lists/JIRAs/PRs) in order to
collect the current status and public opinion widely in the community to
build a consensus on this at this time.
Before talking about those issues, please remind that
- Apache Spark 2.4.x is the only live version because 2.3.x is EOL and
3.0.0 is not released.
- Apache Spark community has the following rule: "Correctness and data
loss issues should be considered Blockers."
Unfortunately, we didn't build a consensus on what is really blocked by
that. In reality, it was just our resolution for the quality and it works a
little differently.
In this email, I want to talk about correctness/dataloss issues and
observed public opinions. They fall into the following categories roughly.
1. Resolved in both 3.0.0 and 2.4.x
- ex) SPARK-30447 Constant propagation nullability issue
- No problem. However, this case sometimes goes to (2)
2. Resolved in both 3.0.0 and 2.4.x. But, reverted in 2.4.x later.
- ex) SPARK-26021 -0.0 and 0.0 not treated consistently, doesn't match
Hive
- "We don't want to change the behavior in the maintenence release"
3. Resolved in 3.0.0 and not backported because this is 3.0.0-specific.
- ex) SPARK-29906 Reading of csv file fails with adaptive execution
turned on
- No problem.
4. Resolved in 3.0.0 and not backported due to technical difficulty.
- ex) SPARK-26154 Stream-stream joins - left outer join gives
inconsistent output
- "This is not backported due to the technical difficulty"
5. Resolved in 3.0.0 and not backported because this is not public API.
- ex) SPARK-29503 MapObjects doesn't copy Unsafe data when nested under
Safe data
- "Since `catalyst` is not public, it's less worth backporting this."
6. Resolved in 3.0.0 and not backported because we forget since there was a
no Target Version.
- ex) SPARK-28375 Make pullupCorrelatedPredicate idempotent
- "Adding the 'correctness' label so we remember to backport this fix to
2.4.x."
- "This is possible, if users add the rule into
postHocOptimizationBatches"
7. Open with Target Version 3.0.0.
- ex) SPARK-29701 Correct behaviours of group analytical queries when
empty input given
- "We aren't fully SQL compliant there and I think that has been true
since the beginning of spark sql"
- "This is not a regression"
8. Open without Target Version.
- I removed this case last week to give more visibility on them.
Here, I want to focus that Apache Spark is a very healthy community because
we have diverse opinions and reevaluating JIRA issues are the results of
the community decision based on the discusson. I believe that it will go
well eventually. In the above, I added those example JIRA IDs and the
collected reasons just to give some colors to illustrate all cases are the
real cases. There is no case to be blamed in the above.
Although some JIRA issues will jump from one category into another category
time to time, the categories will remain there. I want to propose a small
additional work on `Target Version` to distinguish the above categories
easily to communicate clearly in the community. This should be done by
committers because we have the following policy on `Target Version`.
"Target Version. This is assigned by committers to indicate a PR has
been accepted for possible fix by the target version."
Proposed Idea:
A. To reduce the mismatch between `Target Version` vs `Affected
Version`:
When a committer set `correctness` or `data-loss` label, `Target
Version` should be set together according to the `Affected Versions`.
In case of the insufficient `Target Version` (e.g. `Target
Version`=`3.0.0` for `Affected Version`=`2.4.4,3.0.0`), he/she need to add
a comment on the JIRA.
For example, "This is 3.0.0-specific issue"
B. To reduce the mismatch between `Target Version` vs `Fixed Version`:
When a committer resolve `correctness` or `data-loss` labeled issue,
`Target Version` should be compared with `Fixed Version`.
In case of the insufficient `Fixed Version` (e.g. `Target
Version`=`2.4.4,3.0.0` and `Fixed Version`=`3.0.0`), he/she need to add a
comment on the JIRA and adjust `Target Version` according to his/her
decision.
For example, "This is not backported due to the technical
difficulty. I'll remove `2.4.4` from `Target Version`."
With the above rules, the combination of `Affected Version` / `Target
Version` / `Fixed Version` will serve us with much easier way in searching
them, understanding categories, and discussing how to handle properly.
Bests,
Dongjoon.
