dejankrak-db commented on PR #59305: URL: https://github.com/apache/spark/pull/59305#issuecomment-6198093748
@HyukjinKwon, thanks for the review. Addressed in f30470017b8. - **Bare `"file"` coverage:** `DataTypeSuite` now decodes literal JSON with a bare `"file"` nested in an array and in a map, both standalone (unknown) and as the type of a struct field (external). - **Write form of an external file in a field:** you are right that a field read as `FILE EXTERNAL` should not be rewritten as `"file external"`. An external file in a struct field is written as the plain `"file"` again, also inside an array or map, so a schema that carries other engine-specific metadata for the field is written back with the same type and metadata. I checked one engine: DBR rejects a field whose metadata names a mode for it when the type is anything but `"file"`, so the rewrite would have made such a table unreadable there. I have not checked other engines. An unknown file is written as `"file unknown"` and a standalone type keeps its mode. - **Description:** the JSON bullet now says the field type and metadata are written back as they were read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
