[
https://issues.apache.org/jira/browse/SPARK-58457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho resolved SPARK-58457.
-------------------------------
Fix Version/s: 4.3.0
Assignee: Jorge Machado
Resolution: Fixed
> CSV and JSON docs claim PERMISSIVE drops corrupt records when no
> columnNameOfCorruptRecord field is declared; it never does
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58457
> URL: https://issues.apache.org/jira/browse/SPARK-58457
> Project: Spark
> Issue Type: Documentation
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Jorge Molina
> Assignee: Jorge Machado
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> `docs/sql-data-sources-csv.md` and `docs/sql-data-sources-json.md` both
> describe
> `PERMISSIVE` mode with a character-identical pair of sentences:
> {quote}
> *To keep corrupt records*, an user can set a string type field named
> `columnNameOfCorruptRecord` in an user-defined schema. *If a schema does not
> have the
> field, it drops corrupt records during parsing.*
> {quote}
> Neither sentence matches the behaviour of either data source.
> `columnNameOfCorruptRecord` does not control whether a corrupt record is kept
> — it is
> always kept, with its malformed fields set to `null`. The field controls
> whether the
> *malformed string itself* is available.
> This is the sentence a reader consults when deciding whether they need the
> field, and
> it points them the wrong way — it implies a schema *without* the field is the
> safe
> choice because bad records disappear. In fact they are admitted,
> indistinguishable
> from records that were genuinely null at the source, with no error and no
> signal.
> h3. The test suite on master already asserts the correct behaviour, in both
> data sources
> *CSV* — `CSVSuite`'s test {{SPARK-18699 put malformed records in a
> `columnNameOfCorruptRecord` field}} reads `test-data/value-malformed.csv`,
> whose two
> records are `0,2013-111_11 12:13:14` (unparseable date) and `1,1983-08-04`.
> With
> schema `a INT, b DATE` and *no* corrupt-record field it asserts:
> {code:scala}
> checkAnswer(df1, Row(0, null) :: Row(1, java.sql.Date.valueOf("1983-08-04"))
> :: Nil)
> {code}
> Two rows — the corrupt record kept, its bad field nulled. Were the documented
> sentence accurate, this assertion would expect one row.
> *JSON* — `JsonSuite` is more direct still. The test is named {{Corrupt
> records:
> PERMISSIVE mode, without designated column for malformed records}}, reads the
> `corruptRecords` fixture with a three-column user-defined schema and no
> corrupt-record
> column, and asserts five rows — four entirely null — under Spark's own
> comment:
> {code:scala}
> // Corrupted records are replaced with null
> Row(null, null, null),
> {code}
> So the correction is not a claim about behaviour that needs establishing; it
> is
> alignment with what the project already asserts.
> h3. Corroborated by direct measurement
> `pyspark` 3.5.9, three source records per case, `PERMISSIVE`, user-defined
> schema
> `a INT, b STRING`:
> || format || bad record || rows without the field || rows with the field ||
> flagged corrupt ||
> | JSON | not parseable as JSON at all | 3 | 3 | 1 |
> | JSON | valid JSON, wrong type for schema | 3 | 3 | 1 |
> | CSV | wrong type for schema | 3 | 3 | 1 |
> | CSV | unterminated quote | 3 | 3 | 0 |
> Identical counts in every case. The JSON rows matter most: JSON is where the
> sentence
> would be most defensible, since a whole JSON record genuinely can be
> unparseable —
> and it is still retained.
> Measured on `pyspark` 3.5.9 rather than on `master`, so this is
> corroboration; the two
> `master` assertions above are the load-bearing evidence.
> h3. Proposed fix
> Replace *both* sentences, in both files, with:
> {quote}
> To capture the malformed string, an user can set a string type field named
> `columnNameOfCorruptRecord` in an user-defined schema. If a schema does not
> have the
> field, the corrupt record is still retained with its malformed fields set to
> `null`,
> but the malformed string is not available.
> {quote}
> Both sentences have to move together: correcting only the second would leave
> the
> paragraph contradicting itself, since the first assigns retention to the
> field while
> the second would deny retention without it. One line per file. PR to follow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
