Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23215#discussion_r239388954
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -272,9 +279,13 @@ object PartitioningUtils {
val literal = if (userSpecifiedDataTypes.contains(columnName)) {
// SPARK-26188: if user provides corresponding column schema, get
the column value without
// inference, and then cast it as user specified data
type.
- val columnValue = inferPartitionColumnValue(rawColumnValue, false,
timeZone)
- val castedValue =
- Cast(columnValue, userSpecifiedDataTypes(columnName),
Option(timeZone.getID)).eval()
+ val dataType = userSpecifiedDataTypes(columnName)
+ val columnValueLiteral = inferPartitionColumnValue(rawColumnValue,
false, timeZone)
+ val columnValue = columnValueLiteral.eval()
+ val castedValue = Cast(columnValueLiteral, dataType,
Option(timeZone.getID)).eval()
+ if (validatePartitionColumns && columnValue != null && castedValue
== null) {
+ throw new RuntimeException(s"Failed to cast partition value
`$columnValue` to $dataType")
--- End diff --
Can we also show `columnName` in this exception message? So it is easier to
know which partition column has such error.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
