Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18865#discussion_r132688380
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
---
@@ -114,7 +114,16 @@ class JsonFileFormat extends TextBasedFileFormat with
DataSourceRegister {
}
(file: PartitionedFile) => {
- val parser = new JacksonParser(actualSchema, parsedOptions)
+ // SPARK-21610: when the `requiredSchema` only contains
`_corrupt_record`,
--- End diff --
@cloud-fan @HyukjinKwon Because the results in those queries are pretty
weird if you don't know the details of it. If we are going to agree that
`_corrupted_record` is set to null if no columns are selected, we must clearly
document this behavior.
We need to let users know that `_corrupted_record` is a derived column from
other columns and cannot be selected alone in a query.
If you are all agreed that, I think we can revert this change and go to
update all documents related to `_corrupted_record` here.
What do you think? It makes sense to you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
