david-mollitor-db opened a new pull request, #58889:
URL: https://github.com/apache/spark/pull/58889
### What changes were proposed in this pull request?
`JsonExpressionEvalUtils` has several JSON-extraction evaluators. Two of
them --
`GetJsonObjectEvaluator` and `MultiGetJsonObjectEvaluator` -- already reuse
a single
`@transient private lazy val outputBuffer` (a `ByteArrayOutputStream`)
across rows, calling
`reset()` before each use. The other two allocate a fresh
`ByteArrayOutputStream` on every
field/element:
- `JsonTupleEvaluator.parseRow` -- a new buffer per extracted field
(unbounded per row for a wide
`json_tuple(...)`).
- `JsonTableEvaluator.serializeCurrentValue` -- a new buffer per call; the
hot caller expands a
JSON array one element at a time.
This extends the existing `outputBuffer` reuse pattern to those two
evaluators: each gets an
instance `outputBuffer` field and calls `reset()` before writing to /
reading from it.
### Why are the changes needed?
A no-arg `ByteArrayOutputStream` starts at the JDK default of 32 bytes and
grows by doubling;
allocating (and then GC-ing) a fresh one per field/element is steady
allocation churn on the
`json_tuple` / JSON_TABLE eval path. Reusing one buffer per evaluator --
already the pattern for
the two get-json-object evaluators in the same file -- removes it.
### Does this PR introduce _any_ user-facing change?
No. Each value's bytes are fully materialized into a `UTF8String`
(`UTF8String.fromBytes(outputBuffer.toByteArray)`) before the buffer is
reused, so the output is
byte-for-byte identical. The evaluators are per-expression instances
evaluated one row at a time
(`GenerateExec` drains each row's iterator sequentially), so the reused
buffer is never shared
concurrently or used re-entrantly.
### How was this patch tested?
Existing `JsonExpressionsSuite` (`json_tuple` and the get-json-object
evaluators) and
`JsonTableSuite` (JSON_TABLE row expansion, including multi-element arrays
-- the reused-buffer hot
path) pass. This is a behavior-preserving change, so no new tests were added.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
