viirya opened a new pull request, #57583: URL: https://github.com/apache/spark/pull/57583
### What changes were proposed in this pull request? Short-circuit `ArrowCachedBatchSerializer.convertCachedBatchToInternalRow` when the projection is empty: emit `numRows` reused 0-field `UnsafeRow`s per cached batch, without deserializing or decompressing the batch's Arrow payload. The row count is already recorded on `ArrowCachedBatch`. Also fixes the `ArrowCachedBatch` scaladoc, which listed the per-column statistics as `(upperBound, lowerBound, ...)` while both write paths produce the `ColumnStats.collectedStatistics` order `(lowerBound, upperBound, nullCount, count, sizeInBytes)`; the code was consistently lower-first everywhere, only the doc was wrong. ### Why are the changes needed? An empty projection (e.g. a count aggregate through the row-based reader, `spark.sql.inMemoryColumnarStorage.enableVectorizedReader=false`) selects no columns, yet the reader still paid full IPC deserialization and decompression for every cached batch just to iterate its rows. That cost is pure waste: the answer is a stored integer. ### Does this PR introduce _any_ user-facing change? No. The Arrow cache serializer (SPARK-57268) is unreleased, and the change is performance-only; results are identical. ### How was this patch tested? Two new tests in `ArrowCachedBatchSerializerSuite`: - `empty projection emits row counts without deserializing the Arrow payload`: hands the reader a cached batch whose Arrow payload is garbage bytes. The empty projection returns the correct number of empty rows purely from `numRows` (fails before this change with `IllegalArgumentException: capacity < 0` from the IPC reader, proving the payload used to be deserialized), while a projection that actually needs the payload still fails on the same batch, pinning that only the empty-projection case skips the read. - `count aggregate over the cached relation with the row-based reader`: end-to-end `count(*)` over a cached relation spanning many small Arrow batches with the vectorized reader disabled, plus a `sum` over the same cached data verifying projecting reads still decode the payload correctly. Full `ArrowCachedBatchSerializerSuite` (73 tests) and `ArrowCachedBatchKryoRegistrationSuite` pass. ### Was this patch authored or co-authored using generative AI tooling? Yes, this pull request and its description were written by Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
