[
https://issues.apache.org/jira/browse/SPARK-58232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58232:
-----------------------------------
Labels: pull-request-available (was: )
> [SQL] Simplify cached-batch column-index resolution with a map lookup
> ----------------------------------------------------------------------
>
> Key: SPARK-58232
> URL: https://issues.apache.org/jira/browse/SPARK-58232
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Ganesha S
> Priority: Major
> Labels: pull-request-available
>
> *Summary*
> In DefaultCachedBatchSerializer
> (sql/core/.../execution/columnar/InMemoryRelation.scala), the mapping from
> selected
> output attributes to their ordinals in the cached schema is computed with an
> O(n*m)
> algorithm: for each selected attribute it rebuilds the full list of
> cached-schema
> ExprIds and linear-scans it with indexOf. This replaces that with a single
> ExprId -> ordinal map and O(1) lookups (overall O(n+m)).
> This is a code-quality cleanup, not a performance fix. The computation runs
> once per
> scan execution on the driver, so the runtime impact is negligible in
> practice; the value
> is removing a quadratic algorithm and a per-attribute list re-allocation in
> favor of the
> straightforward map lookup.
>
> *Affected code*
> sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala
> Both conversion methods share the pattern:
> * convertCachedBatchToColumnarBatch (vectorized path)
> * convertCachedBatchToInternalRow (row path)
> {code:scala}
> val columnIndices =
> selectedAttributes.map(a => cacheAttributes.map(o =>
> o.exprId).indexOf(a.exprId)).toArray
> {code}
> For each of m selected attributes this rebuilds a fresh Seq[ExprId] from all
> n cached
> attributes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
