ganeshashree commented on code in PR #57395:
URL: https://github.com/apache/spark/pull/57395#discussion_r3718247693
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:
##########
@@ -170,9 +174,12 @@ class ArrowCachedBatchSerializer extends
SimpleMetricsCachedBatchSerializer {
val selectedSchema = DataTypeUtils.fromAttributes(selectedAttributes)
val timeZoneId = conf.sessionLocalTimeZone
- // Calculate column indices for projection
+ // Calculate column indices for projection. Use AttributeSeq's cached
exprId -> ordinal
+ // map so each selected attribute is a constant-time lookup, instead of
linear-scanning
+ // the cache attributes per selected attribute.
+ val cacheAttributeSeq = AttributeSeq(cacheAttributes)
val selectedIndices = selectedAttributes.map { attr =>
Review Comment:
Confirmed and moved. The fallback branch delegates to
`convertCachedBatchToColumnarBatch`, which resolves the indices itself, so
`selectedIndices` was only ever read by the fast path. It now lives in the else
branch next to prefetchEnabled (`ArrowCachedBatchSerializer.scala:213`), with a
short comment recording why it sits there so it doesn't drift back up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
