This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new a7d11d47c5d [opt](load) Read JSON documents straight into the padded 
buffer (#70780)
a7d11d47c5d is described below

commit a7d11d47c5defd6710f488df414ce998d3c2a72d
Author: Daniel Lemire <[email protected]>
AuthorDate: Sat Nov 21 03:22:16 2026 -0500

    [opt](load) Read JSON documents straight into the padded buffer (#70780)
    
    ### Background
    
    The v2 JSON reader (`be/src/format_v2/json/json_reader.cpp`) parses each
    document with the simdjson On-Demand parser, which needs the input
    followed by `SIMDJSON_PADDING` (64) readable bytes. The reader keeps a
    reusable `_padding_buffer` for that. In routine load and stream load
    (`FILE_STREAM`), a document is one pipe message, or several chunks of a
    chunked HTTP transfer. When `read_json_by_line` is false, a document is
    the whole file split.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: None
    PR Type: Performance
    
    Problem Summary:
    
    Pipe messages and whole-file documents were first copied into
    `_document_buffer`, then copied again into `_padding_buffer` before
    parsing. Whole-file reads also zero-filled a freshly allocated
    `_document_buffer` the size of the file. In addition, `open()`
    zero-filled an 8 MB `_padding_buffer` in every reader, even for small
    documents. This PR reads and copies documents straight into
    `_padding_buffer`, and grows that buffer to the largest document instead
    of allocating 8 MB up front. Results and error handling are unchanged.
    
    ### Details
    
    - **Why it was slow:** The cost follows from the code. For each pipe
    message, `_read_one_document_from_pipe` does
    `_document_buffer.assign(...)`, then `_parse_next_json` does
    `memcpy(_padding_buffer, _document_view, size)`. For a whole-file
    document, `_document_buffer.resize(n)` allocates and zeroes `n` bytes,
    `read_at` fills them, and the same `memcpy` follows. `open()` runs
    `_padding_buffer.resize(8 MB + 64)` every time.
    - **Optimization:** `_read_one_document_from_pipe` copies messages, and
    appends chunks, directly into `_padding_buffer`. Whole-file mode calls
    `read_at` directly into `_padding_buffer`. `_parse_next_json` copies
    only when the document is not already at the start of the buffer. That
    is line mode, where the line reader's span is still borrowed, and a
    skipped UTF-8 BOM, where it uses `memmove` because the ranges overlap.
    `_document_buffer` and `_padded_size` are removed. The buffer grows with
    `std::string::resize`, which keeps the contents and grows capacity
    geometrically.
    - **Results:** A standalone program reproduces the old and new buffer
    handling with simdjson On-Demand and a full traversal of every value.
    These are not Doris end-to-end numbers. Apple M4, clang, `-O3`, best of
    30 iterations, two runs (range shown):
    
      | Workload | Before | After |
      |---|---:|---:|
      | Pipe, 20,000 messages of 89 B | 1.05–1.07 GB/s | 1.15–1.16 GB/s |
    | Pipe, 2,000 messages of 6.3 KB (twitter statuses) | 3.42 GB/s |
    3.56–3.58 GB/s |
    | Whole file, twitter.json (632 KB) | 2.93–3.04 GB/s | 3.28–3.30 GB/s |
    | Whole file, citm_catalog.json (1.7 MB) | 3.28–3.36 GB/s | 3.75–3.96
    GB/s |
      | Whole file, 40 MB array | 1.93–1.95 GB/s | 2.16–2.19 GB/s |
    
    A new reader's 8 MB zero-fill costs about 49 µs on the same machine,
    while a 1 KB document parses in about 1 µs.
    - **Applicability and cost:** Applies to `FILE_STREAM` (routine load,
    stream load) and to whole-file JSON. Line mode (`read_json_by_line`)
    keeps its single copy and only benefits from the smaller initial
    allocation. The padding buffer is now sized to the largest document seen
    by the reader, plus 64 bytes. Before, it was at least 8 MB. A reader
    that sees a large document grows the buffer once, as before. No new
    config.
    
    ### Testing
    
    - `be/test/format_v2/json/json_reader_test.cpp`: the line-mode test now
    checks that the document is still borrowed from the line reader and that
    the padding buffer fits the longest line plus padding (not 8 MB). The
    whole-file outer-array test checks that the document is read in place
    and that the buffer is the file size plus padding.
    - `./run-be-ut.sh --run --filter='JsonReader*'` in the
    `apache/doris:build-env-ldb-toolchain-latest` image (ASAN build): 35
    tests from `JsonReaderTest`, `JsonReaderSplitTest` and
    `JsonReaderPrefetchTest` pass.
    - The pipe path has no unit test in this file. Its buffer logic (chunked
    appends that grow the buffer, a BOM in whole-file and line mode, a long
    line after short ones) was checked in a standalone program under ASan
    and UBSan.
    
    ### Release note
    
    Faster JSON routine load, stream load, and whole-file JSON load, with
    less memory per JSON reader.
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No
    - Pre-submission status: Partially verified: the performance numbers
    come from a standalone program that reproduces the reader's buffer
    handling, not from a Doris benchmark.
    
    ### AI Disclosure
    
    - Tool: Claude Code (Claude Opus 5.5)
    - Generated: code, tests, and this description
    - Human review: all code reviewed by Daniel
    - Prompts: The author asked to apply the single-copy fix of
    nodejs/node#67623 to Doris JSON loading, then to also look at the 8 MB
    buffer allocated in `open()` and at whether a newer simdjson would help
    (left for a separate proposal).
---
 be/src/format_v2/json/json_reader.cpp       | 43 +++++++++++++++++------------
 be/src/format_v2/json/json_reader.h         |  9 ++++--
 be/test/format_v2/json/json_reader_test.cpp | 16 +++++++++--
 3 files changed, 44 insertions(+), 24 deletions(-)

diff --git a/be/src/format_v2/json/json_reader.cpp 
b/be/src/format_v2/json/json_reader.cpp
index 3a3e2fe3615..4c226f6e5b3 100644
--- a/be/src/format_v2/json/json_reader.cpp
+++ b/be/src/format_v2/json/json_reader.cpp
@@ -359,7 +359,6 @@ Status JsonReader::open(std::shared_ptr<FileScanRequest> 
request) {
     }
     RETURN_IF_ERROR(_parse_jsonpath_and_json_root());
     _json_parser = std::make_unique<simdjson::ondemand::parser>();
-    _padding_buffer.resize(_padded_size);
     _reader_eof = false;
     _single_document_read = false;
     _eof = false;
@@ -653,11 +652,11 @@ Status JsonReader::_read_one_document(size_t* size, bool* 
eof) {
         *eof = true;
         return Status::OK();
     }
-    _document_buffer.resize(cast_set<size_t>(read_size));
-    Slice result(_document_buffer.data(), _document_buffer.size());
+    const auto document_size = cast_set<size_t>(read_size);
+    _grow_padding_buffer(document_size);
+    Slice result(_padding_buffer.data(), document_size);
     RETURN_IF_ERROR(_physical_file_reader->read_at(_current_offset, result, 
size, _io_ctx.get()));
-    _document_buffer.resize(*size);
-    _document_view = _document_buffer;
+    _document_view = std::string_view(_padding_buffer.data(), *size);
     if (*size == 0) {
         *eof = true;
     }
@@ -672,12 +671,12 @@ Status JsonReader::_read_one_document_from_pipe(size_t* 
read_size) {
     DorisUniqueBufferPtr<uint8_t> file_buf;
     RETURN_IF_ERROR(stream_load_pipe->read_one_message(&file_buf, read_size));
     if (*read_size == 0) {
-        _document_buffer.clear();
         _document_view = {};
         return Status::OK();
     }
-    _document_buffer.assign(reinterpret_cast<const char*>(file_buf.get()), 
*read_size);
-    _document_view = _document_buffer;
+    _grow_padding_buffer(*read_size);
+    std::memcpy(_padding_buffer.data(), file_buf.get(), *read_size);
+    _document_view = std::string_view(_padding_buffer.data(), *read_size);
     if (!stream_load_pipe->is_chunked_transfer()) {
         return Status::OK();
     }
@@ -689,13 +688,21 @@ Status JsonReader::_read_one_document_from_pipe(size_t* 
read_size) {
         if (next_size == 0) {
             break;
         }
-        _document_buffer.append(reinterpret_cast<const char*>(next_buf.get()), 
next_size);
+        _grow_padding_buffer(*read_size + next_size);
+        std::memcpy(_padding_buffer.data() + *read_size, next_buf.get(), 
next_size);
         *read_size += next_size;
     }
-    _document_view = _document_buffer;
+    _document_view = std::string_view(_padding_buffer.data(), *read_size);
     return Status::OK();
 }
 
+void JsonReader::_grow_padding_buffer(size_t size) {
+    const size_t needed = size + simdjson::SIMDJSON_PADDING;
+    if (_padding_buffer.size() < needed) {
+        _padding_buffer.resize(needed);
+    }
+}
+
 Status JsonReader::_parse_next_json(size_t* size, bool* eof) {
     RETURN_IF_ERROR(_read_one_document(size, eof));
     if (*eof || *size == 0) {
@@ -707,17 +714,17 @@ Status JsonReader::_parse_next_json(size_t* size, bool* 
eof) {
         _document_view.remove_prefix(3);
         *size -= 3;
     }
-    if (*size + simdjson::SIMDJSON_PADDING > _padded_size) {
-        _padded_size = *size + simdjson::SIMDJSON_PADDING;
-        _padding_buffer.resize(_padded_size);
-    }
     // Ondemand values reference the input buffer. Keep the padded bytes in a 
member buffer until the
-    // current document is fully materialized.
-    std::memcpy(_padding_buffer.data(), _document_view.data(), *size);
+    // current document is fully materialized. File and pipe documents are 
already there, at offset
+    // 3 when a BOM was skipped, so the ranges may overlap.
+    if (_document_view.data() != _padding_buffer.data()) {
+        _grow_padding_buffer(*size);
+        std::memmove(_padding_buffer.data(), _document_view.data(), *size);
+    }
     _original_doc_size = *size;
+    const std::string_view document(_padding_buffer.data(), *size);
     const auto error =
-            _json_parser->iterate(std::string_view(_padding_buffer.data(), 
*size), _padded_size)
-                    .get(_original_json_doc);
+            _json_parser->iterate(document, 
_padding_buffer.size()).get(_original_json_doc);
     if (error != simdjson::error_code::SUCCESS) {
         return Status::DataQualityError(
                 "Parse json data for JsonDoc failed. code: {}, error info: 
{}", error,
diff --git a/be/src/format_v2/json/json_reader.h 
b/be/src/format_v2/json/json_reader.h
index 2ac77c2242c..13baa533c9d 100644
--- a/be/src/format_v2/json/json_reader.h
+++ b/be/src/format_v2/json/json_reader.h
@@ -111,7 +111,10 @@ public:
     Status close() override;
 
 #ifdef BE_TEST
-    size_t TEST_document_buffer_size() const { return _document_buffer.size(); 
}
+    bool TEST_document_in_padding_buffer() const {
+        return !_document_view.empty() && _document_view.data() == 
_padding_buffer.data();
+    }
+    size_t TEST_padding_buffer_size() const { return _padding_buffer.size(); }
 #endif
 
 private:
@@ -139,6 +142,8 @@ private:
     // Chunked HTTP transfers concatenate their pipe buffers into one document.
     Status _read_one_document(size_t* size, bool* eof);
     Status _read_one_document_from_pipe(size_t* read_size);
+    // Grows `_padding_buffer` to hold `size` bytes plus simdjson padding, 
keeping its contents.
+    void _grow_padding_buffer(size_t size);
     // Moves the logical document into a simdjson-padded buffer and creates an 
ondemand document.
     Status _parse_next_json(size_t* size, bool* eof);
     // Applies json_root and validates the object/array shape required by 
strip_outer_array.
@@ -223,11 +228,9 @@ private:
     simdjson::ondemand::value _json_value;
     simdjson::ondemand::array _array;
     simdjson::ondemand::array_iterator _array_iter;
-    std::string _document_buffer;
     std::string_view _document_view;
     std::string _padding_buffer;
     size_t _original_doc_size = 0;
-    size_t _padded_size = 1024 * 1024 * 8 + simdjson::SIMDJSON_PADDING;
     SimdJsonReadContext _simdjson_context;
 };
 
diff --git a/be/test/format_v2/json/json_reader_test.cpp 
b/be/test/format_v2/json/json_reader_test.cpp
index 8d33c0b13ca..e072f690118 100644
--- a/be/test/format_v2/json/json_reader_test.cpp
+++ b/be/test/format_v2/json/json_reader_test.cpp
@@ -26,6 +26,7 @@
 #include <fstream>
 #include <memory>
 #include <string>
+#include <string_view>
 #include <thread>
 #include <vector>
 
@@ -262,7 +263,8 @@ struct ReadResult {
     bool eof = false;
     size_t second_rows = 0;
     bool second_eof = false;
-    size_t document_buffer_size = 0;
+    bool document_in_padding_buffer = false;
+    size_t padding_buffer_size = 0;
     int64_t rejected_rows = 0;
     std::map<LocalColumnId, std::vector<uint8_t>> missing_fields;
     std::vector<ColumnDefinition> schema;
@@ -310,7 +312,8 @@ ReadResult read_once(const std::string& file_name, const 
std::string& content,
 
     result.block = make_block(result.schema, requested_local_ids);
     result.status = reader.get_block(&result.block, &result.rows, &result.eof);
-    result.document_buffer_size = reader.TEST_document_buffer_size();
+    result.document_in_padding_buffer = 
reader.TEST_document_in_padding_buffer();
+    result.padding_buffer_size = reader.TEST_padding_buffer_size();
     result.rejected_rows = io_ctx->load_filtered_rows;
     result.missing_fields = reader.missing_fields();
     if (result.status.ok() && read_twice) {
@@ -687,7 +690,10 @@ TEST(JsonReaderTest, 
ReadsRequestedColumnsInFileScanRequestOrder) {
     ASSERT_TRUE(result.second_status.ok()) << result.second_status.to_string();
     EXPECT_EQ(result.second_rows, 0);
     EXPECT_TRUE(result.second_eof);
-    EXPECT_EQ(result.document_buffer_size, 0);
+    EXPECT_FALSE(result.document_in_padding_buffer);
+    // The buffer fits the longest line instead of a fixed 8 MB.
+    EXPECT_EQ(result.padding_buffer_size,
+              std::string_view(R"({"id":1,"name":"alice"})").size() + 
simdjson::SIMDJSON_PADDING);
 }
 
 TEST(JsonReaderTest, HiveColumnLookupIsCaseInsensitiveWithoutNormalizedKeys) {
@@ -716,6 +722,10 @@ TEST(JsonReaderTest, ReadsSingleDocumentOuterArray) {
     EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column, 0), 
"carol");
     EXPECT_EQ(nullable_int_at(*result.block.get_by_position(0).column, 1), 4);
     EXPECT_EQ(nullable_string_at(*result.block.get_by_position(1).column, 1), 
"dave");
+    EXPECT_TRUE(result.document_in_padding_buffer);
+    EXPECT_EQ(result.padding_buffer_size,
+              
std::string_view(R"([{"id":3,"name":"carol"},{"id":4,"name":"dave"}])").size() +
+                      simdjson::SIMDJSON_PADDING);
 }
 
 TEST(JsonReaderTest, ReadsJsonRootByLine) {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to