andygrove opened a new pull request, #5244: URL: https://github.com/apache/datafusion-comet/pull/5244
## Which issue does this PR close? N/A - release process task (the ["Generate Release Documentation"](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/release_process.md#generate-release-documentation) step). ## Rationale for this change On `main`, `docs/source/user-guide/latest/` holds only template markers, and CI fills them at publish time. A release branch instead commits the generated content so the archived docs for 1.0.0 render real tables. ## What changes are included in this PR? Output of `./dev/generate-release-docs.sh`, with no hand edits: - `configs.md` — config reference tables filled in (all 10 `CONFIG_TABLE` markers populated, +416 lines) - `compatibility/expressions/spark-{3.4,3.5,4.0,4.1}/*.md` — 40 new per-Spark-version compatibility pages generated from `_category_template/` - `expressions.md` — Implementation column refreshed ## Note for reviewers: six rows regressed to em-dash in `expressions.md` `hour`, `minute`, `second`, `make_timestamp`, `to_char`, and `to_varchar` now show `—` (no implementation) instead of `Native` / `Hybrid` / `Codegen dispatch`. This is a pre-existing `GenerateDocs` limitation, not a real loss of support. `GenerateDocs.buildFunctionNameToKind` resolves each function name via `FunctionRegistry.expressions` and looks up `info.getClassName` in `QueryPlanSerde`'s serde maps. In Spark 4.1 these six are registered with `expressionBuilder(...)` rather than `expression[T](...)`: ```scala expressionBuilder("to_char", ToCharacterBuilder), expressionBuilder("to_varchar", ToCharacterBuilder, setAlias = true, Some("3.5.0")), expressionBuilder("hour", HourExpressionBuilder), expressionBuilder("minute", MinuteExpressionBuilder), expressionBuilder("second", SecondExpressionBuilder), expressionBuilder("make_timestamp", MakeTimestampExpressionBuilder), ``` `expressionBuilder` builds its `ExpressionInfo` from the *builder* type, so `getClassName` returns e.g. `HourExpressionBuilder`, which has no serde entry, and the lookup falls through to the em-dash placeholder. `spark-4.1` is the last profile in the loop and `expressions.md` is not per-version, so 4.1's answer wins. Because `docs/build.sh` runs the same loop, the published `latest` docs already show these six as `—` today — freezing just makes it visible in a diff. I'd suggest fixing the builder resolution in `GenerateDocs` on `main` and backporting it, then regenerating this PR, so the archived 1.0.0 docs don't record these as unimplemented. Two genuine improvements are also visible in the same diff: `cast` correctly becomes `Hybrid` (it gained a codegen-dispatch fallback in #5079), and `<<` / `>>` / `>>>` correctly become `Native` (registered as operator aliases in Spark 4.0+). ## How are these changes tested? `./dev/generate-release-docs.sh` completed successfully for all four Spark profiles, and `npx prettier "**/*.md" --check` reports no issues in any file this PR touches. (Note: JDK 17+ is required for the script to get through the `spark-4.0` / `spark-4.1` profiles; under JDK 11 the Scala compile fails with `Class java.lang.Record not found`.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
