david-mollitor-db opened a new pull request, #58838:
URL: https://github.com/apache/spark/pull/58838
### What changes were proposed in this pull request?
Row-based readers apply column "existence default values" (the value used
for a column missing from
the data but with a DEFAULT in the schema) per record via
`ResolveDefaultColumns.applyExistenceDefaultValuesToRow(schema, row,
bitmask)` /
`resetExistenceDefaultsBitmask(schema, bitmask)`, which internally call:
```scala
def existenceDefaultValues(schema: StructType): Array[Any] =
schema.fields.map(getExistenceDefaultValue) // new Array[Any];
parses+evaluates each default expr
```
This is a pure function of the schema (field metadata only, never row data)
and is not cached, so it
is recomputed every record. `applyExistenceDefaultValuesToRow` recomputes it
and also calls
`hasExistenceDefaultValues(schema)` (recomputes again); the recompute
happens whether or not the
schema has any defaults. The utility already documents this: "The caller
should avoid using such
methods in a loop for efficiency."
This PR:
- adds overloads that take the precomputed values, keeping the schema-based
methods (still used
one-time by the analyzer): `resetExistenceDefaultsBitmask(defaultValues:
Array[Any], bitmask)` and
`applyExistenceDefaultValuesToRow(defaultValues: Array[Any], row,
bitmask)`;
- caches the values once per reader and uses the overloads:
- `ParquetRowConverter` and `OrcDeserializer`: add a per-instance
`existenceDefaults` and a
`hasExistenceDefaults` boolean; the bitmask starts all-true and is only
corrected when defaults
exist, so the guard is kept, then the cached array is passed to the
overload;
- `UnivocityParser` (CSV): caches `existenceDefaults` and indexes it in
the parse-error fallback.
`JacksonParser` (JSON) uses the same pattern but its `convertObject` is
recursive over a different
struct schema per call; that hoist is handled in a separate PR.
### Why are the changes needed?
JFR profiling of `DataSourceReadBenchmark` showed this recompute as ~4.5% of
sampled allocation
(`ManifestFactory.newArray` under `ParquetRowConverter.currentRecord` and
the ORC reader). It is the
same "schema-derived constant recomputed per row" pattern as SPARK-59506 and
SPARK-59543.
Measuring a trimmed `DataSourceReadBenchmark` (single INT column scan) under
JFR
(`settings=profile`), allocation samples through
`ResolveDefaultColumns.getExistenceDefaultValue`:
- `ParquetRowConverter.currentRecord`: ~1,050 → 0.
- `OrcDeserializer`: ~520 → 0.
No new allocation is introduced (the values are computed once); the CSV path
only touches them on a
parse error.
### Does this PR introduce _any_ user-facing change?
No. The values are a schema constant, so the applied defaults are identical.
### How was this patch tested?
Existing tests, which write data, `ALTER TABLE ADD COLUMN` with a DEFAULT,
then read the old data
expecting the default — across Parquet/ORC/CSV, vectorized and row:
```
build/sbt 'sql/testOnly *InsertSuite *ResolveDefaultColumnsSuite'
build/sbt 'catalyst/testOnly *StructTypeSuite'
```
All pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
