[
https://issues.apache.org/jira/browse/SPARK-60832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60832:
-----------------------------------
Labels: pull-request-available (was: )
> Shrink the sparse index array of LongToUnsafeRowMap at optimize time
> --------------------------------------------------------------------
>
> Key: SPARK-60832
> URL: https://issues.apache.org/jira/browse/SPARK-60832
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
> Reporter: Yuming Wang
> Priority: Major
> Labels: pull-request-available
>
> LongToUnsafeRowMap (the map behind broadcast LongHashedRelation for single
> long keys)
> sizes its sparse index array by the estimated ROW count and keeps a load
> factor <= 0.5
> headroom while being built. When many rows share few distinct keys and the
> key range is
> too wide for optimize()'s dense conversion, the finished map keeps a
> mostly-empty array,
> and write() serializes it verbatim -- so the headroom is paid for in driver
> memory, in
> the broadcast payload, and again in every executor's deserialized copy.
> Example: 100M rows sharing 10K wide-range keys keep a ~2 GiB index that only
> needs
> 512 KiB (~4096x waste).
> Proposal: shrink the sparse array once at optimize() time (when dense mode
> does not
> apply) by rehashing the occupied (key, address) pairs into the smallest
> power-of-two
> array that keeps updateIndex's load factor invariant (numKeys * 4 <=
> array.size()).
> write()/read() stay unchanged, so the wire format is untouched and executors
> deserialize
> a right-sized index. Right-sized builds (unique keys, or any map that grew
> during the
> build) are no-ops; at the 1<<30 array-size ceiling the shrink never grows.
> ShuffledHashJoinExec passes no size estimate and its growth ladder already
> lands on the
> minimal size, so this is effectively broadcast-only.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
