AmanBoronbaev commented on PR #55641:
URL: https://github.com/apache/arrow/pull/55641#issuecomment-6961313451
Follow-up on initialization overhead: updated in
ddb215160c40192deb14a6a6cab25ae1a9b00d2e to reuse the lookup already performed
by `AddField`. A unique field gets the next index; a duplicate returns an
existing index. Comparing that return value with the previous field count
detects duplicates without a second hash-table lookup. The object-builder
pointer is obtained after recursive builder construction.
I also measured parser construction/destruction with fixed-length unique
field names and schema construction outside the timed loop. Local results below
are medians of seven batches after one warm-up batch, with `100000 /
field_count` initializations per batch:
| Fields | Baseline (us) | Original PR (us) | Updated PR (us) |
| --- | --- | --- | --- |
| 10 | 93.56 | 100.59 | 94.48 |
| 100 | 724.47 | 814.91 | 716.59 |
| 1000 | 6221.55 | 7041.70 | 6422.56 |
| 10000 | 64663.62 | 66689.00 | 63723.14 |
Environment: Intel i5-12450H, Clang 22.1.8, Arrow Debug library (`-O0 -g0`),
C++20; the standalone driver was built with `-O2`. Baseline is the parent of
the original PR commit; “Original PR” is
7e7a876d27b1d02601ae42d9b488284bca70f959. Variants ran sequentially. These are
local microbenchmark results, not release-build throughput estimates or a
statistically controlled speedup claim. They include parser destruction and
allocations and show approximately linear scaling over the tested widths; they
do not establish a worst-case complexity bound.
Validation after the update: all 254 C++ JSON tests passed, including
duplicate names at root/nested/list levels and valid schemas; clang-format,
cpplint and the `-Werror` build passed.
<details>
<summary>Standalone reproduction driver</summary>
```cpp
#include <chrono>
#include <iomanip>
#include <iostream>
#include <memory>
#include <string>
#include "arrow/json/options.h"
#include "arrow/json/parser.h"
#include "arrow/type.h"
int main() {
std::cout << "fields,repeat,iterations,ns_per_init,ns_per_field\n";
for (int count : {10, 100, 1000, 10000}) {
arrow::FieldVector fields;
for (int i = 0; i < count; ++i) {
// Fixed-length names keep hashing cost comparable across schema
widths.
auto suffix = std::to_string(i);
fields.push_back(arrow::field("field_" + std::string(8 -
suffix.size(), '0') + suffix,
arrow::int64()));
}
auto options = arrow::json::ParseOptions::Defaults();
options.explicit_schema = arrow::schema(fields);
options.unexpected_field_behavior =
arrow::json::UnexpectedFieldBehavior::Error;
int iterations = 100000 / count;
for (int repeat = -1; repeat < 7; ++repeat) {
auto start = std::chrono::steady_clock::now();
for (int i = 0; i < iterations; ++i) {
std::unique_ptr<arrow::json::BlockParser> parser;
auto status = arrow::json::BlockParser::Make(options, &parser);
if (!status.ok()) { std::cerr << status << '\n'; return 1; }
}
double ns = std::chrono::duration<double, std::nano>(
std::chrono::steady_clock::now() - start).count() / iterations;
if (repeat >= 0) std::cout << count << ',' << repeat << ',' <<
iterations << ','
<< std::fixed << std::setprecision(1) << ns
<< ',' << ns/count << '\n';
}
}
}
```
Build the Arrow JSON shared library at each revision with the same
configuration. Compile this driver against that checkout's `cpp/src` and
build's `src` include paths, link `-larrow`, and set the library path to that
build. Rebuild only the parser/library between variants; construct the schema
outside the timed loop as above.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]