andygrove opened a new issue, #266:
URL: https://github.com/apache/datafusion-spark/issues/266
Part of #174 and apache/datafusion-comet#7300.
### What the function does
`log1p(expr)` returns `ln(1 + expr)`, computed accurately for `expr` near 0.
The argument is implicitly cast to DOUBLE (`UnaryMathExpression`, `inputTypes =
Seq(DoubleType)`) and the result is DOUBLE. Like the other Hive-derived log
functions (`UnaryLogExpression`), any input `<= -1.0` returns NULL instead of
`-Infinity` or NaN. The result is always nullable. Available since 1.4.0.
Spark evaluates `java.lang.StrictMath.log1p` (fdlibm) in both the
interpreted and code-generated paths.
### Examples
```sql
SELECT log1p(0);
-- 0.0
SELECT log1p(-1);
-- NULL
SELECT log1p(-2);
-- NULL
SELECT log1p(CAST('NaN' AS DOUBLE));
-- NaN
SELECT log1p(CAST('Infinity' AS DOUBLE));
-- Infinity
```
`MathExpressionsSuite` checks `log1p` against `StrictMath.log1p` for 0.0,
0.1, ..., 2.0 and expects NULL for -10.0, -9.0, ..., -1.0.
### Edge cases to test
- NULL input.
- Exactly -1.0 returns NULL (the boundary is `<= yAsymptote` with
`yAsymptote = -1.0`), and values just above it (e.g. `-0.9999999999999999`)
return a large negative finite value.
- Values below -1 (including `-Infinity`) return NULL, not NaN.
- NaN returns NaN (`NaN <= -1.0` is false). `+Infinity` returns `Infinity`.
- `-0.0` returns `-0.0` and `0.0` returns `0.0` (check the sign).
- Tiny values (`1e-300`, `1e-20`) return the input value, which is where
`ln(1 + x)` would lose precision.
- Bit-exact results: Spark uses fdlibm via `StrictMath`. Rust's `f64::ln_1p`
calls the platform `log1p`, which can differ in the last ulp on some inputs and
platforms. Compare against `StrictMath.log1p` values for a range of inputs
(e.g. 0.1 to 2.0 and large values like `1e300`), or use a port of fdlibm
`s_log1p.c` if they differ.
- Integer, DECIMAL, FLOAT and STRING arguments (cast to DOUBLE first; with
ANSI on an invalid string fails in the cast).
### Differences across Spark versions
None found (3.4 through 4.2; only `contextIndependentFoldable` was added).
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala`,
`Log1p`, base classes `UnaryLogExpression` and `UnaryMathExpression`.
- DataFusion core has no `log1p` (it has `ln`, `log`, `log2`, `log10`).
- Follow `crates/spark/src/function/math/expm1.rs`
(`Signature::exact(vec![Float64])`, `exp_m1`), using `ln_1p` and mapping `x <=
-1.0` to NULL.
- Tests: the placeholder `crates/sqllogictest/test_files/math/log1p.slt`
exists with a commented-out query; enable and extend it. Document with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]