HyukjinKwon commented on code in PR #59305:
URL: https://github.com/apache/spark/pull/59305#discussion_r4287343879
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -344,6 +345,16 @@ object DataType {
TimestampNTZNanosType(p)
}
case "timestamp_ltz" => TimestampType
+ case FILE_TYPE(mode) =>
+ // A bare `"file"` has no mode: inside a StructField it is an external
file, which is also
+ // what a schema written without any mode means; elsewhere it is an
unknown one, as when a
+ // UDF's return type is serialized as a bare `"file"`.
Review Comment:
**Nit (P3):** This comment gives a UDF return type as the source of a bare
`"file"` outside a field. But `FileType.jsonValue` (`FileType.scala:75-76`)
always writes the mode, so `FileType()` is serialized as `"file unknown"`, as
`DataTypeSuite` asserts. The only Spark writer of a bare `"file"` is
`StructField.dataTypeJsonValue`, for an external file inside a field. A
maintainer reading this branch may keep or change it for a producer that no
longer exists. The real source of a bare `"file"` outside a field is a schema
written without a mode, for example by another engine, which is also how the PR
description puts it. Only the comment is wrong; decoding is unaffected.
```suggestion
// A bare `"file"` has no mode, as in a schema written without one,
for example by
// another engine: inside a StructField it is an external file,
elsewhere an unknown
// one. Spark itself writes a bare `"file"` only for an external
file in a field.
```
**Verification:**
- **Inspection:** The comment on the bare-file decoding branch describes
only producers that exist (schemas written without a mode) and names no Spark
serializer that writes a bare "file" outside a struct field.
- **Compile:** The sql/api module still compiles with the decoding logic
unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
