NestDream commented on PR #58797:
URL: https://github.com/apache/spark/pull/58797#issuecomment-5673012451
@zhengruifeng this is in the decimal rescaling from SPARK-53938. The Context
there uses HALF_EVEN while Decimal.set and CAST round HALF_UP, so 1.005 came
back as 1.00 from Arrow UDFs and from createDataFrame over Connect. I switched
it to HALF_UP and added tests for each path that goes through the converter.
Could you review?
Minimal repro on master (Arrow UDFs are the default since 4.2):
```python
from decimal import Decimal
from pyspark.sql.functions import col, udf
from pyspark.sql.types import DecimalType
df = spark.sql("SELECT * FROM VALUES ('1.005'), ('1.025'), ('0.125') AS
t(v)")
f = udf(lambda v: Decimal(v), DecimalType(20, 2))
df.select(col("v").cast(DecimalType(20, 2)).alias("cast"),
f("v").alias("udf")).show()
```
```
+----+----+
|cast| udf|
+----+----+
|1.01|1.00|
|1.03|1.02|
|0.13|0.12|
+----+----+
```
With this change the `udf` column matches `cast`.
cc @HyukjinKwon @Yicong-Huang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
