sollhui opened a new pull request, #69875:
URL: https://github.com/apache/doris/pull/69875
### What problem does this PR solve?
Issue Number: None
Related PR: #69716
Problem Summary:
After restoring sequential prefetch, large V2 CSV loads still spend
substantial CPU time in row parsing and string materialization. The reader
compares a general separator at every byte, updates atomic profile counters per
cell, starts three stage timers per row, and repeats checked string-column
casts per cell. It also ignores the row cap supplied by TableReader.
This change combines four optimizations on master:
- Compare the separator byte directly for plain single-character separators;
retain enclosed, multi-character, Hive/OpenCSV and PROTO parsing paths.
- Accumulate exact work counters locally and publish at
get_block/count/schema return, including errors. Sample stage timings every
1024 physical read_line calls, starting with the first, and expose unscaled
SampledReadLineTime, SampledSplitLineTime and SampledDeserializeTime with
sample counts. DelimitedTextReader remains full elapsed time; split-prefix
alignment has its own full timer.
- Validate nullable string column types and reserve offsets/null maps once
per mutable batch, then insert strings directly with the existing NULL, quoting
and escape semantics.
- Honor FileReader::set_batch_size, retaining the existing byte budget and
runtime default when no explicit cap is supplied. The caller still controls the
maximum batch size; this does not force 16352-row blocks.
Shared counters and batch sizing also apply to the Hive text reader. No
speedup is claimed yet; the combined patch is ready for a controlled benchmark
against its master parent.
### Release note
Reduce V2 CSV parsing overhead and honor caller-provided batch sizes.
Delimited-text per-stage profile timers are now explicitly named Sampled* and
contain unscaled sampled durations; use the full DelimitedTextReader timer for
elapsed-time comparisons.
### Check List (For Author)
- Test
- [x] Unit Test: added cases for dynamic/zero batch caps, byte limits,
filtered blocks, cross-batch sampling and exact counters,
single/multi-character separators, NULL and escape behavior, missing fields,
reordered projection and copy-on-write. Extended error, rejected-row and COUNT
counter coverage.
- Execution: run-be-ut.sh was attempted with CsvV2ReaderTest.*,
TextV2ReaderTest.* and LegacyAndV2/PlainCsvSplitTest.*. It exits before C++
compilation because the local JDK 17 fails to initialize (Failed setting boot
class path), including outside the sandbox. Full build/UT and performance
benchmark remain pending.
- Passed: repository clang-format 16 formatting/check scripts,
build-support/check-build-hygiene.sh and git diff --check.
- Behavior changed: Yes. Existing batch-size contract is honored, and
detailed stage timers are explicitly sampled. CSV value semantics are unchanged.
- Does this need documentation: No. The profile includes a timing
explanation; no new configuration or SQL syntax.
Benchmark: use identical hardware, input, compression, cache conditions and
load batch/byte-budget settings for the parent and this commit. Compare total
load time, Scanner CPU time, DelimitedTextReader time and RowsRead/BlockNumber.
Do not directly compare Sampled* timers with the previous full stage timers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
