Ganesha S created SPARK-58232:
---------------------------------
Summary: [SQL] Resolve cached-batch column indices via a map
instead of O(n*m) linear scan
Key: SPARK-58232
URL: https://issues.apache.org/jira/browse/SPARK-58232
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Ganesha S
*Summary*
When reading from an in-memory (cached) relation,
`DefaultCachedBatchSerializer` maps
each selected output attribute to its ordinal in the cached schema. It
currently does
this with an `O(n*m)` algorithm that, for every selected attribute, rebuilds
the full
list of cached-schema ExprIds and linear-scans it with indexOf. This is
wasteful time
and transient allocation on wide cached scans and should be a single map lookup.
*Affected code*
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala
Both conversion methods share the pattern:
* convertCachedBatchToColumnarBatch (vectorized path)
* convertCachedBatchToInternalRow (row path)
{code:scala}
val columnIndices =
selectedAttributes.map(a => cacheAttributes.map(o =>
o.exprId).indexOf(a.exprId)).toArray
{code}
For each of m selected attributes this rebuilds a fresh Seq[ExprId] from all n
cached
attributes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
