Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18865
Just to summarise my opinion:
I think I am unclear if we should call, selecting `_corrupt_record` alone,
a bug to disallow. If I understood correctly, I think we are basically saying:
- Results from selecting `_corrupt_record` alone are useless and weird:
Its output looks not completely useless and weird in a way as shown in
https://github.com/apache/spark/pull/18865#issuecomment-326832181
- Results from selecting `_corrupt_record` alone should produce like all
columns are parsed but it this looks violating the designed behavior. So, we
should rather disallow and inform the workaround in the exception.
I think we are unclear on this as the flipside looks also working - it
should work like no columns are parsed, apparently as designed. We should allow
but with informing the workaround in logging or documentation.
I am -0 for disallowing because I prefer a conservative way, and it does
not look challenging or demanding, but both sides look having some arguments
though.
If I understood correctly, we don't say the current behavior with selecting
`_corrupt_record` alone is incorrect as this might be as designed.
If we don't think this behavior is as designed and it should be dependent
on the selected json fields, and If we can't produce the expected results (not
only selecting `_corrupt_record` alone but also few columns together), I think
we should deprecate this option first with explaining this somewhere, regarding
the opinion here.
I have been thinking we disallow or change a behaviour when, for example,
it is clearly a bug, does not make sense, or is not as designed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
