JingsongLi commented on code in PR #10474:
URL: https://github.com/apache/paimon/pull/10474#discussion_r4230560084
##########
paimon-format/src/main/java/org/apache/paimon/format/csv/CsvParser.java:
##########
@@ -108,40 +108,41 @@ public GenericRow parse(String line) {
int position = 0;
while (position < line.length() && columnIndex < rowValues.length) {
char c = line.charAt(position);
- if (c == escapeChar) {
- // if the next character is special, process it here as to not
trigger the special
- // handling
- if (inQuotes || inField) {
- int nextCharacter = peekNextCharacter(line, position);
- if (nextCharacter == quoteChar || nextCharacter ==
escapeChar) {
- buffer.append(line.charAt(position + 1));
- position++;
- }
+ // Only a quote at the start of a field enables CSV quoting and
escaping.
+ // Quotes and backslashes in an unquoted field (for example JSON
in TSV)
+ // belong to the field value and must not be interpreted as CSV
syntax.
+ if (!inField && buffer.length() == 0 && quoteChar != '\0' && c ==
quoteChar) {
+ rowQuoted[columnIndex] = true;
+ }
+ if (rowQuoted[columnIndex] && escapeChar != '\0' && c ==
escapeChar) {
Review Comment:
[P2] Keep writer and parser behavior consistent for NUL quote/escape options
These guards disable NUL as CSV syntax, but `CsvFormatWriter.escapeField`
still uses the configured quote and escape characters to encode fields. Both
configurations where only one of these options is NUL are accepted.
I reproduced this with the actual writer and the base/head parsers: with
`quote = "\0"` and the default escape, writing `[1, "a,b", "tail"]` produces
`1,<NUL>a,b<NUL>,tail`. The base parser restores the original row, whereas this
parser returns `[1, "<NUL>a", "b<NUL>"]`, losing `tail`. With the default quote
and `escape = "\0"`, `say "hello"` also gains literal NUL characters on
readback.
Please synchronize the writer's encoding/validation with the
disabled-character semantics and account for files already produced by the
previous writer. Add writer-reader round-trip tests with quote and escape
disabled independently; the new test only disables both and parses hand-built
input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
