david-mollitor-db opened a new pull request, #58832:
URL: https://github.com/apache/spark/pull/58832
### What changes were proposed in this pull request?
When reading JSON with an explicit `encoding` option,
`CreateJacksonParser.getStreamDecoder` builds
a `CharsetDecoder` per input record via `CharsetProvider.newDecoder(enc,
caller = "Jackson Parser")`,
and `newDecoder`'s two default arguments each call `SQLConf.get`:
```scala
def newDecoder(charset: String,
legacyCharsets: Boolean = SQLConf.get.legacyJavaCharsets,
legacyErrorAction: Boolean = SQLConf.get.legacyCodingErrorAction,
caller: String = "decode"): CharsetDecoder
```
So `SQLConf.get` runs twice per record. This PR resolves the two flags once
per reader in
`JSONOptions` and passes them explicitly to `newDecoder`:
- `JSONOptions`: add `legacyJavaCharsets` and `legacyCodingErrorAction` vals
resolved from
`SQLConf.get` at construction (matching the existing SQLConf-derived
fields such as
`writeNullIfWithDefaultValue`).
- `CreateJacksonParser`: thread the two flags through `getStreamDecoder` and
the `text`,
`internalRow`, and `bytes` encoding variants into
`CharsetProvider.newDecoder`.
- `TextInputJsonDataSource`: pass the resolved flags from the options at the
three closure sites
(`inferFromDataset`, `readFile`, `readStream`).
Only the single-line/text read path is affected (it calls
`getStreamDecoder`). The multi-line path
uses `new InputStreamReader(is, enc)` and never called `newDecoder`, so it
is untouched.
`CharsetProvider.newDecoder`'s signature/defaults are unchanged (its other
caller, `decode()`,
already passes explicit flags).
### Why are the changes needed?
On an executor task thread `SQLConf.get` constructs a fresh
`ReadOnlySQLConf` + `ConfigReader` +
`HashMap` each call. The two flags are constant for the whole task, so
reading them per record is
pure allocation churn. This mirrors what CSV already does — `CSVOptions`
resolves
`legacyJavaCharsets` once at construction via
`CharsetProvider.forName(_, SQLConf.get.legacyJavaCharsets, ...)`.
Profiling `JsonBenchmark`'s encoding-specified read cases under JFR
(`settings=profile`):
- Allocation samples flowing through `getStreamDecoder` together with
`SQLConf.get`/`ConfigReader`/`ReadOnlySQLConf` drop from ~7,600 to 0.
- `ConfigReader.<init>` drops from ~2% of sampled allocation pressure to
absent.
- Wall-clock time is unchanged within run-to-run noise (this is an
allocation / GC-pressure
reduction, not a CPU one).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests, which exercise this path (including the SPARK-23723 /
SPARK-23724 encoding tests
with `spark.sql.legacy.javaCharsets` enabled):
```
build/sbt 'sql/testOnly *JsonSuite *JsonV1Suite *JsonV2Suite
*JsonLegacyTimeParserSuite'
build/sbt 'catalyst/testOnly *JacksonParserSuite'
```
All 502 tests pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
