Joel Robin created SPARK-59098:
----------------------------------
Summary: format_string/printf throws
IllegalFormatConversionException for decimal arguments with %f
Key: SPARK-59098
URL: https://issues.apache.org/jira/browse/SPARK-59098
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0, 5.0.0
Reporter: Joel Robin
{{printf}} and {{format_string}} throw if you pass a decimal to {{%f}} (or
{{%e}}, {{%g}}, {{%a}}):
{code:sql}
spark-sql> SELECT printf('%.2f', 1.5);
java.lang.IllegalArgumentException: f != org.apache.spark.sql.types.Decimal
{code}
No cast, nothing unusual. {{1.5}} is {{decimal(2,1)}}, so that query is just
"format a decimal with %f". Decimal columns fail the same way.
The error is hard to act on. {{IllegalFormatConversionException}} extends
{{IllegalArgumentException}}, so PySpark surfaces it as a plain
{{IllegalArgumentException}} and nothing in the message or the traceback
mentions {{format_string}}. And {{org.apache.spark.sql.types.Decimal}} is an
internal class - it isn't a type the user wrote.
{{%f}} on a NULL decimal returns {{"null"}} and doesn't throw. So this hides in
pipelines that format a sparse decimal column, and breaks the day real values
show up.
h3. Repro
{code:sql}
SELECT printf('%.2f', 1.5);
{code}
And via a column, which goes through whole-stage codegen rather than constant
folding:
{code:python}
spark.range(3).selectExpr("CAST(id + 0.5 AS DECIMAL(10,2)) AS
d").createOrReplaceTempView("t")
spark.sql("SELECT format_string('%.2f', d) FROM t").show()
{code}
Both fail on 4.0.0 (pyspark 4.0.0, local[1], OpenJDK 17.0.15). The codegen one
throws from
generated code:
{code:none}
java.util.IllegalFormatConversionException: f !=
org.apache.spark.sql.types.Decimal
at
java.base/java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4442)
at
java.base/java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2976)
at java.base/java.util.Formatter.format(Formatter.java:2689)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
