sunchao commented on code in PR #4971:
URL: https://github.com/apache/datafusion-comet/pull/4971#discussion_r4100699856
##########
native/spark-expr/src/string_funcs/get_json_object.rs:
##########
@@ -372,25 +644,107 @@ impl<'de> Visitor<'de> for SegmentVisitor<'_> {
}
fn visit_seq<A: SeqAccess<'de>>(self, mut seq: A) -> Result<Self::Value,
A::Error> {
- let PathSegment::Index(idx) = &self.segments[0] else {
- IgnoredAny.visit_seq(seq)?;
- return Ok(None);
- };
-
- for _ in 0..*idx {
- if seq.next_element::<IgnoredAny>()?.is_none() {
- return Ok(None);
+ match &self.segments[0] {
+ PathSegment::Index(idx) => {
+ // Spark switches to Quoted style for the "one or more results"
+ // case: an index immediately followed by a subscript wildcard.
+ let child_style = match self.segments.get(1) {
+ Some(PathSegment::SubscriptWildcard) |
Some(PathSegment::DoubleWildcard) => {
+ Style::Quoted
+ }
+ _ => self.style,
+ };
+ for _ in 0..*idx {
+ if seq.next_element::<IgnoredAny>()?.is_none() {
+ return Ok(PathResult::default());
+ }
+ }
+ let found = seq
+ .next_element_seed(PathSeed {
+ segments: &self.segments[1..],
+ style: child_style,
+ reject_direct_null: false,
+ })?
+ .unwrap_or_default();
+ // The remaining elements are still visited, so that a
malformed element
+ // after the match yields no match, as a full parse would.
+ IgnoredAny.visit_seq(seq)?;
+ Ok(found)
+ }
+ PathSegment::DoubleWildcard => {
+ // Spark consumes both wildcards of `[*][*]` at once: the
+ // remaining path applies to the outer elements in flatten
+ // style, and the collected writes always form a single array,
+ // even when there is only one element or none matched.
+ let mut writes = SmallVec::new();
+ let mut matched = false;
+ while let Some(mut result) = seq.next_element_seed(PathSeed {
+ segments: &self.segments[1..],
+ style: Style::Flatten,
+ reject_direct_null: false,
+ })? {
+ matched |= result.matched;
+ writes.append(&mut result.writes);
+ }
+ Ok(PathResult::wrap(writes, matched))
+ }
+ PathSegment::SubscriptWildcard => match self.style {
+ // Quoted style: the array wrapper is always kept, even for a
+ // single match.
+ Style::Quoted => {
+ let mut writes = SmallVec::new();
+ let mut matched = false;
+ while let Some(mut result) =
seq.next_element_seed(PathSeed {
+ segments: &self.segments[1..],
+ style: Style::Quoted,
+ reject_direct_null: false,
+ })? {
+ matched |= result.matched;
+ writes.append(&mut result.writes);
+ }
+ Ok(PathResult::wrap(writes, matched))
+ }
+ // Raw or Flatten style: Spark buffers the element writes into
+ // a temporary array and only emits it when more than one
+ // element wrote; a lone writer's brackets are stripped, and
+ // nothing at all is written when no element matched.
+ Style::Raw | Style::Flatten => {
+ let child_style = if self.style == Style::Raw {
+ Style::Quoted
+ } else {
+ Style::Flatten
+ };
+ let mut writers = 0;
+ let mut writes = SmallVec::new();
+ while let Some(mut result) =
seq.next_element_seed(PathSeed {
Review Comment:
[P2] Preserve Spark's nesting limit when streaming wildcard paths
For `$[*].a`, construct the valid JSON as `'[{"a":1,"skip":' + '[' * 999 +
'0' + ']' * 999 + '}]'`. Its total nesting depth is 1,001. Spark 3.5.9, 4.0.4,
4.1.3, and the PR base return SQL NULL, while this head returns `1`. I
reproduced the head result through the compiled scalar and both column entry
points. Reducing the inner arrays to 998 makes Spark and head return `1`;
either field order reproduces the boundary.
The new wildcard traversal skips unrelated subtrees through `IgnoredAny`,
which does not enforce Jackson's depth constraint. This extends a pre-existing
nonwildcard limitation to wildcard paths that the base rejected.
Could we enforce the applicable Spark version's nesting limit while skipping
subtrees and add both sides of this boundary? Spark 3.4.3 accepts these
documents, so the check needs to be version-aware. Restoring serde's older
128-level limit would be too restrictive.
##########
native/spark-expr/src/string_funcs/get_json_object.rs:
##########
@@ -246,68 +306,269 @@ fn parse_json_path(path: &str) -> Option<ParsedPath> {
}
}
- Some(ParsedPath {
- segments,
- has_wildcard,
- })
+ Some(ParsedPath { segments })
+}
+
+/// Find the end of a string body starting at `i` (just past the opening
+/// quote). Short bodies are scanned inline; long bodies use memchr2, the same
+/// approach as serde_json's `ignore_str`. Returns the index just past the
+/// closing quote, or None for an unterminated string (the parser rejects the
+/// document anyway).
+#[inline]
+fn skip_string_body(bytes: &[u8], mut i: usize) -> Option<usize> {
+ const SHORT_STRING: usize = 32;
+ if bytes.len().checked_sub(i)? <= SHORT_STRING {
+ while i < bytes.len() {
+ match bytes[i] {
+ b'"' => return Some(i + 1),
+ b'\\' => i = i.checked_add(2)?,
+ _ => i += 1,
+ }
+ }
+ return None;
+ }
+ loop {
+ match memchr::memchr2(b'"', b'\\', bytes.get(i..)?) {
+ Some(off) if bytes[i + off] == b'"' => return Some(i + off + 1),
+ Some(off) => i = i.checked_add(off)?.checked_add(2)?, // escaped
byte
+ None => return None,
+ }
+ }
+}
+
+/// Spark 3.5+ bundles Jackson versions that reject numbers beyond the default
+/// 1000-digit limit, including values this evaluation skips. serde_json's
+/// `IgnoredAny` enforces no such limit, so inspect number tokens before
parsing.
+/// Spark 3.4's Jackson has no default limit and bypasses this scan.
+fn has_oversized_number(json: &str) -> bool {
+ const MAX_NUMBER_DIGITS: usize = 1000;
+ const JACKSON_READER_BUFFER_UNITS: usize = 4000;
+ let bytes = json.as_bytes();
+ let mut i = 0;
+ // Only near-limit floats need the Reader's UTF-16 position. Keep the
+ // prefix count between candidates so many long numbers stay linear.
+ let mut counted_through = 0;
+ let mut utf16_units = 0;
+ while i < bytes.len() {
+ match bytes[i] {
+ // Skip string bodies: Jackson applies no numeric constraint to
+ // string content.
+ b'"' => match skip_string_body(bytes, i + 1) {
+ Some(end) => i = end,
+ None => return false,
+ },
+ b'-' | b'0'..=b'9' => {
+ let mut j = i + usize::from(bytes[i] == b'-');
+ let int_start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ let int_len = j - int_start;
+ let mut fract_len = 0;
+ if j < bytes.len() && bytes[j] == b'.' {
+ j += 1;
+ let start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ fract_len = j - start;
+ }
+ let mut exp_len = 0;
+ if j < bytes.len() && (bytes[j] | 0x20) == b'e' {
+ j += 1;
+ if j < bytes.len() && (bytes[j] == b'+' || bytes[j] ==
b'-') {
+ j += 1;
+ }
+ let start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ exp_len = j - start;
+ }
+ let is_float = fract_len > 0 || exp_len > 0;
+ let digit_count = if is_float {
+ // Spark parses UTF8String via InputStreamReader, then
+ // ReaderBasedJsonParser with a 4000-UTF-16-unit buffer.
+ // Its slow path uses -1 for an absent fraction or
exponent,
+ // forgiving one digit when a number reaches a buffer edge
+ // or EOF. Numbers starting with zero always take that
path.
+ let starts_with_zero = int_len == 1 && bytes[int_start] ==
b'0';
+ let reaches_buffer_edge = if int_len + fract_len + exp_len
> MAX_NUMBER_DIGITS {
+ utf16_units +=
json[counted_through..i].encode_utf16().count();
+ counted_through = i;
+ utf16_units % JACKSON_READER_BUFFER_UNITS + (j - i)
Review Comment:
[P2] Account for recycled Jackson buffers in numeric validation
This assumes every reader buffer has 4,000 UTF-16 units, but Jackson can
reuse a larger buffer allocated by an earlier String parser. For
`{"a":1,"pad":"<5000 x characters>","n":1.<1000 zeros>}`, `$.a` and `$.n`
return `1` and `1.0` on the base and actual Spark with a recycled
6,000-character buffer. This head returns NULL for both, including through the
compiled scalar and both column entry points.
I reproduced the larger-buffer behavior on Spark 3.5.9, 4.0.4, and 4.1.3
evaluators. It also occurs in ordinary Spark 4.1.3 SQL: read a 6,000-character
JSON Dataset row, build this JSON from a nonconstant input column, then apply
`get_json_object`. It reproduces with whole-stage codegen disabled and enabled.
[JsonFactory allocates the String parser buffer by input
length](https://github.com/FasterXML/jackson-core/blob/jackson-core-2.21.2/src/main/java/com/fasterxml/jackson/core/JsonFactory.java#L1362-L1373),
and [BufferRecycler retains larger
buffers](https://github.com/FasterXML/jackson-core/blob/jackson-core-2.21.2/src/main/java/com/fasterxml/jackson/core/util/BufferRecycler.java#L190-L207).
Could we revisit this fixed-buffer compatibility model and cover the JSON
Dataset pipeline? A fixed modulo cannot recover the JVM recycler state from the
JSON text. An explicit compatibility policy for these rare long-number inputs
would also be clearer than claiming reader-boundary parity.
##########
native/spark-expr/src/string_funcs/get_json_object.rs:
##########
@@ -246,68 +306,269 @@ fn parse_json_path(path: &str) -> Option<ParsedPath> {
}
}
- Some(ParsedPath {
- segments,
- has_wildcard,
- })
+ Some(ParsedPath { segments })
+}
+
+/// Find the end of a string body starting at `i` (just past the opening
+/// quote). Short bodies are scanned inline; long bodies use memchr2, the same
+/// approach as serde_json's `ignore_str`. Returns the index just past the
+/// closing quote, or None for an unterminated string (the parser rejects the
+/// document anyway).
+#[inline]
+fn skip_string_body(bytes: &[u8], mut i: usize) -> Option<usize> {
+ const SHORT_STRING: usize = 32;
+ if bytes.len().checked_sub(i)? <= SHORT_STRING {
+ while i < bytes.len() {
+ match bytes[i] {
+ b'"' => return Some(i + 1),
+ b'\\' => i = i.checked_add(2)?,
+ _ => i += 1,
+ }
+ }
+ return None;
+ }
+ loop {
+ match memchr::memchr2(b'"', b'\\', bytes.get(i..)?) {
+ Some(off) if bytes[i + off] == b'"' => return Some(i + off + 1),
+ Some(off) => i = i.checked_add(off)?.checked_add(2)?, // escaped
byte
+ None => return None,
+ }
+ }
+}
+
+/// Spark 3.5+ bundles Jackson versions that reject numbers beyond the default
+/// 1000-digit limit, including values this evaluation skips. serde_json's
+/// `IgnoredAny` enforces no such limit, so inspect number tokens before
parsing.
+/// Spark 3.4's Jackson has no default limit and bypasses this scan.
+fn has_oversized_number(json: &str) -> bool {
+ const MAX_NUMBER_DIGITS: usize = 1000;
+ const JACKSON_READER_BUFFER_UNITS: usize = 4000;
+ let bytes = json.as_bytes();
+ let mut i = 0;
+ // Only near-limit floats need the Reader's UTF-16 position. Keep the
+ // prefix count between candidates so many long numbers stay linear.
+ let mut counted_through = 0;
+ let mut utf16_units = 0;
+ while i < bytes.len() {
+ match bytes[i] {
+ // Skip string bodies: Jackson applies no numeric constraint to
+ // string content.
+ b'"' => match skip_string_body(bytes, i + 1) {
+ Some(end) => i = end,
+ None => return false,
+ },
+ b'-' | b'0'..=b'9' => {
+ let mut j = i + usize::from(bytes[i] == b'-');
+ let int_start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ let int_len = j - int_start;
+ let mut fract_len = 0;
+ if j < bytes.len() && bytes[j] == b'.' {
+ j += 1;
+ let start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ fract_len = j - start;
+ }
+ let mut exp_len = 0;
+ if j < bytes.len() && (bytes[j] | 0x20) == b'e' {
+ j += 1;
+ if j < bytes.len() && (bytes[j] == b'+' || bytes[j] ==
b'-') {
+ j += 1;
+ }
+ let start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ exp_len = j - start;
+ }
+ let is_float = fract_len > 0 || exp_len > 0;
+ let digit_count = if is_float {
+ // Spark parses UTF8String via InputStreamReader, then
+ // ReaderBasedJsonParser with a 4000-UTF-16-unit buffer.
+ // Its slow path uses -1 for an absent fraction or
exponent,
+ // forgiving one digit when a number reaches a buffer edge
+ // or EOF. Numbers starting with zero always take that
path.
+ let starts_with_zero = int_len == 1 && bytes[int_start] ==
b'0';
+ let reaches_buffer_edge = if int_len + fract_len + exp_len
> MAX_NUMBER_DIGITS {
+ utf16_units +=
json[counted_through..i].encode_utf16().count();
+ counted_through = i;
+ utf16_units % JACKSON_READER_BUFFER_UNITS + (j - i)
+ >= JACKSON_READER_BUFFER_UNITS
+ } else {
+ false
+ };
+ let slow_path = starts_with_zero || reaches_buffer_edge ||
j == bytes.len();
+ let absent_component = fract_len == 0 || exp_len == 0;
+ int_len + fract_len + exp_len - usize::from(slow_path &&
absent_component)
+ } else {
+ int_len
+ };
+ if digit_count > MAX_NUMBER_DIGITS {
+ return true;
+ }
+ i = j;
+ }
+ _ => i += 1,
+ }
+ }
+ false
}
/// Evaluate a parsed JSONPath against a JSON string.
/// Returns the result as a string, or None if no match.
+#[cfg(test)]
fn evaluate_path(json_str: &str, path: &ParsedPath) -> Option<String> {
- if !path.has_wildcard {
- return value_into_string(extract_no_wildcard(json_str,
&path.segments)?);
- }
-
- let value: Value = serde_json::from_str(json_str).ok()?;
+ evaluate_path_with_number_limit(json_str, path, true)
+}
- // Wildcard path: may return multiple results
- let results = evaluate_with_wildcard(&value, &path.segments);
+fn evaluate_path_with_number_limit(
+ json_str: &str,
+ path: &ParsedPath,
+ check_number_length: bool,
+) -> Option<String> {
+ if check_number_length && has_oversized_number(json_str) {
Review Comment:
[P2] Skip numeric validation for inputs too short to exceed the limit
This scans every document before extraction on Spark 3.5+, even when the
entire input is at most 1,000 bytes and therefore cannot contain more than
1,000 ASCII numeric digits.
For a representative 198-byte record with path `$.name`, an independent
five-round alternating exact-source benchmark measured 338.54 ns on base versus
547.29 ns on head. Adding only `json_str.len() > 1000` before this scan reduced
the result to 356.60 ns, with identical outputs. Two separate seven-round runs
corroborated the slowdown. Paths were parsed once and inputs/results were
black-boxed; these are evaluator timings, not whole-query timings.
Could we add this length guard? Serde still validates the document's syntax.
The earlier bytewise 64 KiB string-scan issue is fixed; this is remaining
avoidable work on ordinary short records.
##########
native/spark-expr/src/string_funcs/get_json_object.rs:
##########
@@ -246,68 +306,269 @@ fn parse_json_path(path: &str) -> Option<ParsedPath> {
}
}
- Some(ParsedPath {
- segments,
- has_wildcard,
- })
+ Some(ParsedPath { segments })
+}
+
+/// Find the end of a string body starting at `i` (just past the opening
+/// quote). Short bodies are scanned inline; long bodies use memchr2, the same
+/// approach as serde_json's `ignore_str`. Returns the index just past the
+/// closing quote, or None for an unterminated string (the parser rejects the
+/// document anyway).
+#[inline]
+fn skip_string_body(bytes: &[u8], mut i: usize) -> Option<usize> {
+ const SHORT_STRING: usize = 32;
+ if bytes.len().checked_sub(i)? <= SHORT_STRING {
+ while i < bytes.len() {
+ match bytes[i] {
+ b'"' => return Some(i + 1),
+ b'\\' => i = i.checked_add(2)?,
+ _ => i += 1,
+ }
+ }
+ return None;
+ }
+ loop {
+ match memchr::memchr2(b'"', b'\\', bytes.get(i..)?) {
+ Some(off) if bytes[i + off] == b'"' => return Some(i + off + 1),
+ Some(off) => i = i.checked_add(off)?.checked_add(2)?, // escaped
byte
+ None => return None,
+ }
+ }
+}
+
+/// Spark 3.5+ bundles Jackson versions that reject numbers beyond the default
+/// 1000-digit limit, including values this evaluation skips. serde_json's
+/// `IgnoredAny` enforces no such limit, so inspect number tokens before
parsing.
+/// Spark 3.4's Jackson has no default limit and bypasses this scan.
+fn has_oversized_number(json: &str) -> bool {
+ const MAX_NUMBER_DIGITS: usize = 1000;
+ const JACKSON_READER_BUFFER_UNITS: usize = 4000;
+ let bytes = json.as_bytes();
+ let mut i = 0;
+ // Only near-limit floats need the Reader's UTF-16 position. Keep the
+ // prefix count between candidates so many long numbers stay linear.
+ let mut counted_through = 0;
+ let mut utf16_units = 0;
+ while i < bytes.len() {
+ match bytes[i] {
+ // Skip string bodies: Jackson applies no numeric constraint to
+ // string content.
+ b'"' => match skip_string_body(bytes, i + 1) {
+ Some(end) => i = end,
+ None => return false,
+ },
+ b'-' | b'0'..=b'9' => {
+ let mut j = i + usize::from(bytes[i] == b'-');
+ let int_start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ let int_len = j - int_start;
+ let mut fract_len = 0;
+ if j < bytes.len() && bytes[j] == b'.' {
+ j += 1;
+ let start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ fract_len = j - start;
+ }
+ let mut exp_len = 0;
+ if j < bytes.len() && (bytes[j] | 0x20) == b'e' {
+ j += 1;
+ if j < bytes.len() && (bytes[j] == b'+' || bytes[j] ==
b'-') {
+ j += 1;
+ }
+ let start = j;
+ while j < bytes.len() && bytes[j].is_ascii_digit() {
+ j += 1;
+ }
+ exp_len = j - start;
+ }
+ let is_float = fract_len > 0 || exp_len > 0;
+ let digit_count = if is_float {
+ // Spark parses UTF8String via InputStreamReader, then
+ // ReaderBasedJsonParser with a 4000-UTF-16-unit buffer.
+ // Its slow path uses -1 for an absent fraction or
exponent,
+ // forgiving one digit when a number reaches a buffer edge
+ // or EOF. Numbers starting with zero always take that
path.
+ let starts_with_zero = int_len == 1 && bytes[int_start] ==
b'0';
+ let reaches_buffer_edge = if int_len + fract_len + exp_len
> MAX_NUMBER_DIGITS {
+ utf16_units +=
json[counted_through..i].encode_utf16().count();
+ counted_through = i;
+ utf16_units % JACKSON_READER_BUFFER_UNITS + (j - i)
+ >= JACKSON_READER_BUFFER_UNITS
+ } else {
+ false
+ };
+ let slow_path = starts_with_zero || reaches_buffer_edge ||
j == bytes.len();
+ let absent_component = fract_len == 0 || exp_len == 0;
+ int_len + fract_len + exp_len - usize::from(slow_path &&
absent_component)
+ } else {
+ int_len
+ };
+ if digit_count > MAX_NUMBER_DIGITS {
+ return true;
+ }
+ i = j;
+ }
+ _ => i += 1,
+ }
+ }
+ false
}
/// Evaluate a parsed JSONPath against a JSON string.
/// Returns the result as a string, or None if no match.
+#[cfg(test)]
fn evaluate_path(json_str: &str, path: &ParsedPath) -> Option<String> {
- if !path.has_wildcard {
- return value_into_string(extract_no_wildcard(json_str,
&path.segments)?);
- }
-
- let value: Value = serde_json::from_str(json_str).ok()?;
+ evaluate_path_with_number_limit(json_str, path, true)
+}
- // Wildcard path: may return multiple results
- let results = evaluate_with_wildcard(&value, &path.segments);
+fn evaluate_path_with_number_limit(
+ json_str: &str,
+ path: &ParsedPath,
+ check_number_length: bool,
+) -> Option<String> {
+ if check_number_length && has_oversized_number(json_str) {
+ return None;
+ }
- match results.len() {
- 0 => None,
- 1 => {
- // Single wildcard match: Spark preserves JSON serialization format
- // (strings keep their quotes, numbers don't)
- if results[0].is_null() {
- None
- } else {
- serde_json::to_string(results[0]).ok()
- }
- }
- // Multiple results: wrap in JSON array. A slice of `&Value` serializes
- // as a JSON array, so no clone into an owned `Value::Array` is needed.
- _ => serde_json::to_string(&results).ok(),
+ let result = extract_path(json_str, &path.segments)?;
+ if !result.matched {
+ return None;
}
+ // The top level is not an array context. Jackson's generator separates
+ // consecutive root-level writes with a single space, so join with one.
+ Some(PathResult::join(result.writes, " "))
}
-/// Evaluation for paths without wildcards.
-///
/// Descends into the document while it is being parsed, so only the matched
-/// subtree is materialized as a `Value`; everything else is skipped by the
-/// parser without allocating. The whole document is still consumed, so
-/// malformed JSON anywhere in the input yields no match, as a full parse
would.
-fn extract_no_wildcard(json_str: &str, segments: &[PathSegment]) ->
Option<Value> {
+/// subtrees are materialized as `Value`s; everything else is skipped by the
+/// parser without allocating. The whole document is still consumed, so
malformed
+/// JSON anywhere in the input yields no match, as a full parse would.
+fn extract_path(json_str: &str, segments: &[PathSegment]) ->
Option<PathResult> {
let mut de = serde_json::Deserializer::from_str(json_str);
- let found = PathSeed { segments }.deserialize(&mut de).ok()?;
+ let found = PathSeed {
+ segments,
+ style: Style::Raw,
+ reject_direct_null: false,
+ }
+ .deserialize(&mut de)
+ .ok()?;
de.end().ok()?;
- found
+ Some(found)
}
/// Deserializes the value at `segments`, discarding everything else.
struct PathSeed<'a> {
segments: &'a [PathSegment],
+ /// The output style in effect, mirroring the `style` parameter Spark
+ /// threads through `evaluatePath`.
+ style: Style,
+ /// A JSON null directly below a named field is not a match in Spark. Nulls
+ /// reached through array traversal are matches and serialize as `null`.
+ reject_direct_null: bool,
+}
+
+/// The outcome of applying (part of) a path, modeled on Spark's generator
+/// protocol: `writes` holds one rendered fragment per generator write and
+/// `matched` is Spark's dirty flag.
+///
+/// The two can diverge: the wildcard arms that write directly to the generator
+/// emit their array wrapper even when nothing inside matched, so an unmatched
+/// result can still carry writes. Spark's generator keeps those bytes — a
later
+/// occurrence of a duplicated field can build on them — so they are preserved
+/// here rather than discarded.
+#[derive(Default)]
+struct PathResult {
+ // A simple field or index lookup produces one write; keep it inline while
+ // descending through nested objects and arrays.
+ writes: SmallVec<[String; 1]>,
+ matched: bool,
+}
+
+impl PathResult {
+ fn join(mut writes: SmallVec<[String; 1]>, separator: &str) -> String {
+ if writes.len() == 1 {
+ writes.pop().unwrap()
+ } else {
+ writes.join(separator)
+ }
+ }
+
+ /// A single verbatim write of a matched value, honoring the output style:
+ /// a string in Raw style is written unquoted (Spark's scalar-unwrap arm),
+ /// everything else keeps JSON serialization.
+ fn write(value: Value, style: Style) -> Self {
+ match value {
+ Value::String(s) if style == Style::Raw => Self {
+ writes: smallvec![s],
+ matched: true,
+ },
+ value => Self {
+ writes: smallvec![value.to_string()],
Review Comment:
[P2] Avoid per-leaf temporary serialization for dense wildcard results
Each selected wildcard leaf is serialized into its own String here, then
accumulated and copied into the wrapper at line 528. Repeated alternating
exact-source benchmarks against the PR base show 1,000-number wildcard
extraction at about 1.8-2.1x base runtime, and 1,000-string extraction at
1.65-1.69x. An independent five-round check measured 54.02 us to 99.57 us for
numbers and 94.23 us to 157.10 us for strings. The regression persists with
numeric validation disabled, so it is separate from the pre-scan issue.
The singleton ownership fix works, and numeric-wildcard allocation counts
have improved. This finding concerns the remaining serialization/copy CPU cost.
Sparse wildcard objects improve through selective traversal, so keeping that
behavior matters.
Could we add a common terminal-wildcard path that writes directly into an
output buffer while preserving the existing per-level styles, match counts, and
duplicate-field side effects? These measurements use cached parsed paths and
black-boxed inputs/results and establish evaluator regressions, not whole-query
timings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
