dejankrak-db commented on code in PR #59305:
URL: https://github.com/apache/spark/pull/59305#discussion_r4277507788
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -344,6 +345,16 @@ object DataType {
TimestampNTZNanosType(p)
}
case "timestamp_ltz" => TimestampType
+ case FILE_TYPE(mode) =>
+ // Bare `"file"` resolves differently depending on context: inside a
StructField it is
+ // an external file, which is also what a schema written without any
metadata means; in a
+ // standalone parse (top-level `fromJson` or recursion outside any
StructField) it has no
+ // mode yet, as when a UDF's return type is serialized as a bare
`"file"`.
Review Comment:
Agreed, and you are right that a nested file in a UDF type should stay
unknown. The JSON form of a FileType now always carries its mode, as `"file
external"` or `"file unknown"`, in a field as well as standalone, so a nested
unknown file round-trips (9c76b11358a). `StructField` no longer special-cases
FileType. A bare `"file"` is still read for schemas written without a mode: as
external inside a struct field and as unknown elsewhere, and any other mode
name is rejected with `INVALID_JSON_DATA_TYPE`. `inStructField` stays only for
that no-mode case. Added a test for a nested unknown file in a struct, an array
and a nested struct.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala:
##########
@@ -77,7 +77,8 @@ case class StructField(
}
private[sql] def dataTypeJsonValue: JValue = {
- if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty) {
+ if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty &&
+ !dataType.existsRecursively(_.isInstanceOf[FileType])) {
Review Comment:
Follow-up: with 9c76b11358a a `FileType` writes its own mode in its JSON
form, so `StructField` has no FileType-specific handling any more (the file is
the same as on master) and `removeCollations` is untouched. A FILE-bearing
field takes the same path as any other field, and the `DataTypeSuite` test for
the `MAP<STRING COLLATE UTF8_BINARY, FILE EXTERNAL>` case still covers it.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -344,6 +345,16 @@ object DataType {
TimestampNTZNanosType(p)
}
case "timestamp_ltz" => TimestampType
+ case FILE_TYPE(mode) =>
+ // Bare `"file"` resolves differently depending on context: inside a
StructField it is
+ // an external file, which is also what a schema written without any
metadata means; in a
+ // standalone parse (top-level `fromJson` or recursion outside any
StructField) it has no
+ // mode yet, as when a UDF's return type is serialized as a bare
`"file"`.
Review Comment:
Edited: the behavior and the commit are described in my reply above.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala:
##########
@@ -77,7 +77,8 @@ case class StructField(
}
private[sql] def dataTypeJsonValue: JValue = {
- if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty) {
+ if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty &&
+ !dataType.existsRecursively(_.isInstanceOf[FileType])) {
Review Comment:
Edited: the behavior and the commit are described in my reply above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
