This is an automated email from the ASF dual-hosted git repository.
kazuyukitanimura pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new f64553b6 chore: fix compatibility guide (#978)
f64553b6 is described below
commit f64553b6b788484e3ca03ab8d7b19e51670444ee
Author: KAZUYUKI TANIMURA <[email protected]>
AuthorDate: Fri Sep 27 13:56:45 2024 -0700
chore: fix compatibility guide (#978)
## Rationale for this change
This is a followup PR of #953
## What changes are included in this PR?
Updated the template, otherwise `make release` will wipe it out
---
docs/templates/compatibility-template.md | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/docs/templates/compatibility-template.md
b/docs/templates/compatibility-template.md
index 137870a7..f6a725ac 100644
--- a/docs/templates/compatibility-template.md
+++ b/docs/templates/compatibility-template.md
@@ -38,6 +38,14 @@ Comet uses the Rust regexp crate for evaluating regular
expressions, and this ha
regular expression engine. Comet will fall back to Spark for patterns that are
known to produce different results, but
this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`.
+## Floating number comparison
+
+Spark normalizes NaN and zero for floating point numbers for several cases.
See `NormalizeFloatingNumbers` optimization rule in Spark.
+However, one exception is comparison. Spark does not normalize NaN and zero
when comparing values
+because they are handled well in Spark (e.g.,
`SQLOrderingUtil.compareFloats`). But the comparison
+functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g.,
[arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
+So Comet will add additional normalization expression of NaN and zero for
comparison.
+
## Cast
Cast operations in Comet fall into three levels of support:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
