freemandealer opened a new pull request, #70144:
URL: https://github.com/apache/doris/pull/70144
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
CSV OUTFILE and EXPORT currently write cells without outer quotes. A value
such as `a,b` can be interpreted as two fields, and an embedded newline can be
interpreted as a new record.
Add the opt-in `"csv_quote" = "true"` property for CSV, CSV_WITH_NAMES, and
CSV_WITH_NAMES_AND_TYPES. Quote each complete serialized cell and header value,
and double embedded double quotes: `a,b` becomes `"a,b"`, and `a"b` becomes
`"a""b"`. Complex values are quoted as a whole; the existing SerDe text for
values, including NULL, is retained inside the quotes. The default is `false`.
Propagate the property through OUTFILE planning and persisted EXPORT jobs.
Older job metadata and missing optional sink fields default to `false`. Reject
invalid boolean values, non-CSV use, and separators containing double quotes
when quoting is enabled. The new optional Thrift field uses ID 24, preserving
the existing file-name padding field at ID 23. Hive text and native TVF writers
retain their existing behavior.
### Release note
OUTFILE and EXPORT support the optional `csv_quote` property for quoting
complete CSV fields and headers and doubling embedded double quotes. It is
disabled by default.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
Validation on upstream master `7866f5a36c02e15dc014aaef30e537a395f4059d`:
- Full BE ASAN and FE builds passed using the repository build entry point;
the UI was built from the committed lockfile with Node 22.14.0 and packaged
through `CUSTOM_UI_DIST`. FE Checkstyle remained enabled. The final BE ELF
passed the GLIBC 2.17 compatibility check.
- `run-be-ut.sh --run --filter=VCSVTransformerTest.*`: 8 tests passed.
- `run-fe-ut.sh --run
org.apache.doris.datasource.property.fileformat.CsvFileFormatPropertiesTest,org.apache.doris.planner.OutfileCsvQuoteTest`:
34 tests passed, no failures, errors, or skips.
- clang-format 16, build hygiene, diff whitespace checks, and the repository
clang-tidy check passed for all 7 changed C++ files. The compilation database
was expanded from actual Unity build commands; there were no analysis failures
or findings on changed lines.
- The S3 regression suite is included and its Groovy syntax/imports compile;
its S3 execution was not run locally.
Sanitizer runs retained leak detection with a narrow suppression for the
existing 352-byte `__kmp_affinity_create_x2apicid_map` OpenMP initialization
leak.
- Behavior changed:
- [ ] No.
- [x] Yes. Explicitly enabling `csv_quote` changes CSV cell/header
quoting; the default output remains unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. The OUTFILE/EXPORT SQL reference needs a follow-up for the
new property; its semantics and examples are documented above.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
