peter-toth commented on code in PR #60003:
URL: https://github.com/apache/spark/pull/60003#discussion_r4444216548
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala:
##########
@@ -588,7 +588,9 @@ object PartitioningUtils extends SQLConfHelper {
case LongType => JLong.parseLong(value)
case FloatType => JDouble.parseDouble(value).toFloat
case DoubleType => JDouble.parseDouble(value)
- case _: DecimalType => Literal(new JBigDecimal(value)).value
+ // Cast in ANSI mode to fail on a value that the type cannot hold in any
session, so that
+ // `spark.sql.sources.validatePartitionColumns` alone decides between an
error and null.
+ case dt: DecimalType => Cast(Literal(value), dt, ansiEnabled = true).eval()
case DateType =>
Cast(Literal(value), DateType, Some(zoneId.getId)).eval()
Review Comment:
**Finding 1.** The gap your comment describes for decimals is still open for
`DATE`, `TIME` and the timestamp types below it. In a non-ANSI session, `Cast`
returns `NULL` for a value that doesn't parse, so the `NonFatal` catch in
`parsePartitions` never sees an error. I checked it on this head with
`InMemoryFileIndex`, a user-specified schema and
`spark.sql.sources.validatePartitionColumns=true`:
| `a=abc` read as | ANSI on | ANSI off |
|---|---|---|
| `DATE`, `TIME(6)`, `TIMESTAMP`, `TIMESTAMP_NTZ` |
`INVALID_PARTITION_VALUE` | `NULL` |
| `INTERVAL DAY TO SECOND`, `INTERVAL YEAR TO MONTH` |
`INVALID_PARTITION_VALUE` | `INVALID_PARTITION_VALUE` |
| `DECIMAL(5,2)`, with this PR | `INVALID_PARTITION_VALUE` |
`INVALID_PARTITION_VALUE` |
This goes against the 3.0 migration guide entry ("An exception is thrown if
the validation fails"). Like the decimal case, it dates back to SPARK-34314,
which dropped the `null` check after the cast. It's out of scope here. Could
you file a ticket for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
