[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-453653528 I am going to close this PR since it stuck. This is an

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-450823391 > then how about from_json always return null for corrupted record if mode is PERMISSIVE? Does `always` mean we

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-450558395 I still think the `PERMISSIVE` mode could be useful even for map/arrays if an user wants to bypass error and parse all

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-450385997 Let's imagine a situation when an user uses `from_json` to parse a column with JSON records and for some of the records

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-449728725 > Let me try to revert the commit locally. @cloud-fan Revert causes conflicts in the migration guide. Let me know

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-449722394 > It seems also reasonable to accept @HyukjinKwon 's proposal: just revert 38628dd ok. Let me try to revert the

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-449559886 @cloud-fan Can we move forward with the particular changes in the PR? -

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-448545158 > always return null if the token is empty, no matter it's row or array or map. In general, returning `null`s fo

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-448399353 > can you describe the behavior you proposed? @cloud-fan I updated PR's description. ---

[GitHub] MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource

MaxGekk commented on issue #23325: [SPARK-26376][SQL] Skip inputs without tokens by JSON datasource URL: https://github.com/apache/spark/pull/23325#issuecomment-447579404 @cloud-fan Please, review the PR. This is an automated