andygrove opened a new pull request, #6248:
URL: https://github.com/apache/datafusion-comet/pull/6248
## Which issue does this PR close?
There is no dedicated issue. This relates to #6169, which lists the #6066
key removals as a 1.1.0 follow-up. #6089 settled the policy side of that item,
and this PR adds the user-facing note.
## Rationale for this change
The "Upgrading to Comet 1.1.0" section says only that no legacy key is
needed and that JDK 17 is required. I went through the config changes between
1.0.0 and main and checked them against the versioning policy. None of them
needs a `spark.comet.legacy.*` key, but a few can still change what an existing
deployment does, and anyone upgrading should know about them:
- #6191 fixed how Comet reads `spark.comet.shuffle.native.writeBufferSize`,
`spark.comet.maxTempDirectorySize` and `spark.memory.offHeap.size`. As a result:
- A spill limit written with a unit is now enforced.
- A write buffer written with a unit is now that many bytes rather than a
millionth of it.
- A bare-byte off-heap size now bounds `fair_unified`'s shares.
I'm treating these as bug fixes rather than behavior changes, because each
setting now does what its documentation always said, and the setting itself
restores the old effect. A legacy key that brought back "ignore my unit"
wouldn't make much sense.
- #6195 made Comet disable itself when it is registered through
`spark.sql.extensions` without off-heap memory. Before, it ran on-heap.
- #6163 deprecated `spark.comet.exec.memoryPool.fraction`.
- #6066 removed three on-heap settings that the 1.0.0 docs listed.
The config docs for the fraction and for
`spark.comet.exec.sortMergeJoinWithJoinFilter.enabled`, and the fraction's
driver warning, say the key "will be removed in a future release". The
versioning policy removes a covered key only in a major release.
This should be picked onto `branch-1.1` for the 1.1.0 docs. #6191 is also on
`branch-1.0` (#6209), so if 1.0.1 ships, its release docs need the #6191 entry
too.
## What changes are included in this PR?
- `migration-guide.md`:
- Three subsections under "Upgrading to Comet 1.1.0": settings that now
take effect as documented, the conditions for enabling Comet, and deprecated
and removed settings.
- A sentence in the introduction, so that the guide can list changes that
need no legacy key.
- `CometConf.scala` and `Plugins.scala`: "a future release" becomes "a
future major release" for the two deprecated keys. Nothing else in the rendered
text changes; the fraction's lines are just re-wrapped.
## How are these changes tested?
This PR changes only docs and message text.
- I built the docs locally with Sphinx. There are no new warnings, and the
four new anchor links resolve.
- Prettier 3.9.9 is clean, `spotless:check` and `scalastyle:check` pass, and
the JVM modules compile.
- `CometPluginsMemoryPoolFractionWarningSuite` asserts only the `...=0.8 is
deprecated` prefix of the warning, and that prefix hasn't changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
