[
https://issues.apache.org/jira/browse/SPARK-58458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jorge Molina updated SPARK-58458:
---------------------------------
Priority: Minor (was: Major)
> No test covers the CSV record split caused by a quoted line break when
> multiLine is disabled
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-58458
> URL: https://issues.apache.org/jira/browse/SPARK-58458
> Project: Spark
> Issue Type: Test
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Jorge Molina
> Priority: Minor
> Labels: pull-request-available
>
> A line break inside a quoted value is valid CSV (RFC 4180 §2.6). With the
> default
> `multiLine=false`, the CSV reader cuts such a record at the break: the head
> keeps its
> leading fields with the rest of the schema nulled, and the remainder of the
> quoted
> value starts a new record. The row count rises by one.
> `CSVSuite` does not cover this. The suite contains exactly one input literal
> with a
> newline inside quotes, in {{lineSep with 2 chars when multiLine set to ...}},
> and it is
> confined to the {{multiLine = true}} branch:
> {code:scala}
> val inputData = if (multiLine) {
> s"""name,"i am the${newLine} column1"${newLine}jack,30${newLine}tom,18"""
> } else {
> s"name,age${newLine}jack,30${newLine}tom,18"
> }
> {code}
> The {{multiLine = false}} branch drops the quoted break, so the default path
> is never
> exercised with the input that distinguishes it. `UnivocityParserSuite` does
> not cover
> it either.
> h3. Why it is worth a test
> The behaviour follows from the non-multiLine read path being line-oriented by
> design:
> `HadoopFileLinesReader` splits the input on the line separator before the
> tokenizer
> sees it, and `UnivocityParser.parseLine` then receives one physical line at a
> time.
> That is what makes the file splittable, and this ticket does *not* propose
> changing it.
> It proposes pinning what the current design produces, so a future change to
> that path
> cannot alter it silently.
> Two consequences are worth locking down in particular:
> * {{DROPMALFORMED}} discards *both* halves, so the whole source record is
> lost rather
> than the damaged half — the opposite of what the mode's name suggests for
> one bad
> line.
> * {{PERMISSIVE}} without {{columnNameOfCorruptRecord}} — the default —
> retains both
> halves with no signal. The split *adds* a row while rejecting the fragment
> *removes*
> one, so a row-count reconciliation can pass over a record that lost its
> trailing
> fields.
> h3. Measured behaviour to be asserted
> Measured on `pyspark` 3.5.9 — three records with a quoted line break in the
> second,
> explicit three-column schema. *These are the values to be asserted, not yet
> confirmed
> on `master`* (5.0.0-SNAPSHOT); the fork's CI run is what confirms them, and
> the PR will
> not be opened before it is green.
> || read || rows ||
> | default ({{multiLine=false}}) | 4: {{("1","ACME LTDA","SP")}},
> {{("2","EMPRESA COM",null)}}, {{("QUEBRA DE LINHA\"","RJ",null)}},
> {{("3","OUTRA EMPRESA","MG")}} |
> | {{multiLine=true}} | 3, break preserved inside the value |
> | {{mode=DROPMALFORMED}} | 2: records 1 and 3; record 2 lost entirely |
> | {{columnNameOfCorruptRecord}} declared | 4, both halves carrying their raw
> text |
> PR to follow, test-only, one test added to `CSVSuite`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
