Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22979#discussion_r232487851
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
---
@@ -149,8 +156,8 @@ class UnivocityParser(
case dt: DecimalType => (d: String) =>
nullSafeDatum(d, name, nullable, options) { datum =>
- val value = new BigDecimal(datum.replaceAll(",", ""))
- Decimal(value, dt.precision, dt.scale)
+ val bigDecimal =
decimalParser.parse(datum).asInstanceOf[BigDecimal]
--- End diff --
For instance, there was a similar try to change the date parsing library. I
already know the different is quite breaking - so I suggested to add a
configuration or fallback for now. Probably we should similarily just document
the behaviour change in the migration guide but actually less sure yet even
about this. anyway will take another look shortly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
