ShayanGho opened a new issue, #24940: URL: https://github.com/apache/datafusion/issues/24940
### Describe the bug `datafusion-spark`'s `factorial` declares `Signature::exact(vec![DataType::Int32], ...)` (`datafusion/spark/src/function/math/factorial.rs`). Spark's `Factorial` declares `inputTypes = Seq(IntegerType)` but extends `ImplicitCastInputTypes`, so Spark casts every integer width to INT before evaluating. Because an untyped integer literal is `Int64` in DataFusion, even the simplest call fails in Spark mode. Identical in Spark 3.5.8, 4.0.4, 4.1.3 and 4.2.0 (the class only changed how it declares `nullIntolerant`). ### To Reproduce Spark SQL (verified with `pyspark==4.2.0`, `spark.sql.ansi.enabled` both `true` and `false`): ```sql SELECT factorial(5); -- 120 SELECT factorial(CAST(5 AS BIGINT)); -- 120 SELECT factorial(CAST(5 AS TINYINT)); -- 120 ``` DataFusion (`datafusion-cli --spark`, main): ```sql SELECT factorial(5); -- Error during planning: Failed to coerce arguments to satisfy a call to 'factorial' function: -- coercion from Int64 to the signature Exact(Int32) failed. SELECT factorial(5::BIGINT); -- same error ``` ### Expected behavior `factorial` accepts every integer width, as Spark does, and `SELECT factorial(5)` returns 120. ### Additional context The narrow signature was chosen deliberately in #16125 based on the Databricks reference, which documents the parameter as an INTEGER expression; a running Spark shows the accepted set is wider because of the implicit cast. The existing test asserts the coercion error for `factorial(5::BIGINT)` as expected behavior and needs to be replaced. Out-of-range values are a separate, smaller divergence. DataFusion always fails at the Int32 cast (this function does not consult `datafusion.execution.enable_ansi_mode`), while Spark 4.2.0 raises `CAST_OVERFLOW` under ANSI mode and returns NULL otherwise for `factorial(CAST(5000000000 AS BIGINT))`. Surfaced by the `audit-datafusion-spark-expression` skill. A PR for the integer-width part follows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
