andygrove opened a new pull request, #5085: URL: https://github.com/apache/datafusion-comet/pull/5085
## Which issue does this PR close? Documentation-only follow-up to the open `correctness`+`bug` issues currently tracked. Not intended to close any of them; each remains open until fixed. ## Rationale for this change An audit of open `correctness`+`bug` issues found that several known divergences from Spark are not surfaced in the user-facing [Compatibility Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html). Users on Comet's current release can hit any of these silently. This PR does not fix the underlying bugs; it documents them so users can decide whether a specific query is affected. Where the divergence lives inside a per-expression serde, the note is added via `getCompatibleNotes` / `getIncompatibleReasons` so it renders on the auto-generated per-category compatibility pages (`math.md`, `datetime.md`, `map.md`, etc.). Cross-cutting or non-expression divergences go into the static compat pages (`compatibility/index.md`, `compatibility/scans.md`). ## What changes are included in this PR? ### Expression-level notes (auto-generated compat pages) | Expression | Issue | Nature | |-----------|-------|--------| | `Round` | [#5070](https://github.com/apache/datafusion-comet/issues/5070) | `LongType` returns `0` at `scale <= -19` instead of overflowing | | `IntegralDivide` | [#5065](https://github.com/apache/datafusion-comet/issues/5065) | `Long.MinValue div -1` silently returns wrapped value under ANSI | | `MakeDecimal` | [#5066](https://github.com/apache/datafusion-comet/issues/5066) | Native path drops `fail_on_error`, so ANSI overflow returns `NULL` | | `NextDay` | [#5073](https://github.com/apache/datafusion-comet/issues/5073) | ANSI error surfaces as `CometNativeException` not `SparkIllegalArgumentException` | | `MakeDate` | [#5073](https://github.com/apache/datafusion-comet/issues/5073) | ANSI error surfaces as `CometNativeException` not `SparkDateTimeException` | | `MapFromArrays`, `MapFromEntries` | [#4680](https://github.com/apache/datafusion-comet/issues/4680) | Per-element `NULL` key not rejected (Spark throws) | | `GetJsonObject` | [#4947](https://github.com/apache/datafusion-comet/issues/4947) | Duplicate keys: native returns last, Spark returns first | ### Static sections in `compatibility/index.md` - **ANSI-mode error classes and messages**: collects the wrong-exception-class / wrong-error-class issues into one section so users who key on `getCondition()` know what to expect. Covers [#5071](https://github.com/apache/datafusion-comet/issues/5071), [#5072](https://github.com/apache/datafusion-comet/issues/5072), [#5073](https://github.com/apache/datafusion-comet/issues/5073), [#4967](https://github.com/apache/datafusion-comet/issues/4967). - **Known result-value divergences**: collects the silent wrong-value bugs that don't map to a single expression's compat page. Covers boolean-to-decimal cast eval-mode ([#5068](https://github.com/apache/datafusion-comet/issues/5068)), `CAST(string AS boolean)` ISO controls ([#4959](https://github.com/apache/datafusion-comet/issues/4959)), `RANGE` window boundary overflow ([#5022](https://github.com/apache/datafusion-comet/issues/5022)), `null IN ()` legacy behavior ([#4786](https://github.com/apache/datafusion-comet/issues/4786)). - **Object store cache**: notes the ABFS container-in-userinfo collision documented in [#4993](https://github.com/apache/datafusion-comet/issues/4993). ### `compatibility/scans.md` Expands the existing datetime-rebase note ([#5010](https://github.com/apache/datafusion-comet/issues/5010)): - Widens the trigger from "written before Spark 3.0" to any file written with `datetimeRebaseModeInWrite=LEGACY`. - Adds that filters and aggregates also silently disagree with Spark, not just projections. - Notes that `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level `org.apache.spark.legacyDateTime` metadata are ignored. - Notes that `spark.comet.exceptionOnDatetimeRebase` is currently dead code. - Links the tracker issue. ### Not covered here - [#5067](https://github.com/apache/datafusion-comet/issues/5067) (Remainder on Float/Double NaN under ANSI): already fixed by 9bd5a6d05. - [#4764](https://github.com/apache/datafusion-comet/issues/4764) (EPIC: consistent invalid-UTF-8 handling): already documented in `compatibility/index.md`. ## How are these changes tested? Not applicable — this PR only touches documentation strings and static markdown. The per-category compat pages are regenerated in CI from the `getCompatibleNotes` / `getIncompatibleReasons` values, so the changes take effect on the next docs build. Verified locally: - `./mvnw package -Pspark-3.5 -DskipTests` compiles cleanly. - Regenerating docs under `-Pgenerate-docs` produces the expected new entries in each per-category page. - `npx prettier "docs/**/*.md" --write` reports all files unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
