Peter Toth created SPARK-59249:
----------------------------------
Summary: Take the grouped key-row ordering from the shared
InternalRowComparableWrapper cache
Key: SPARK-59249
URL: https://issues.apache.org/jira/browse/SPARK-59249
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
`KeyedPartitioning.groupedKeyRowOrdering` builds the ordering that grouped
partition keys are laid out by:
{code:scala}
def groupedKeyRowOrdering(dataTypes: Seq[DataType]): BaseOrdering =
RowOrdering.createNaturalAscendingOrdering(dataTypes)
{code}
That call is byte-for-byte the {{loadFunc}} of
{{InternalRowComparableWrapper}}'s private {{orderingCache}}, a 1024-entry
{{NonFateSharingCache}}:
{code:scala}
private val orderingCache = {
val loadFunc = (dataTypes: Seq[DataType]) => {
RowOrdering.createNaturalAscendingOrdering(dataTypes)
}
NonFateSharingCache(loadFunc, MAX_CACHE_ENTRIES)
}
{code}
So every call regenerates an ordering the cache already holds, and that every
partition key wrapper already exposes as {{ordering}}. {{GenerateOrdering}} has
its own code cache, so nothing is recompiled, but the {{SortOrder}} list and
the generated instance are rebuilt each time. The callers are
{{KeyedPartitioning.keyRowOrdering}},
{{GroupPartitionsExec.groupAndSortByKeys}} and the reduced-key ordering in
{{EnsureRequirements}}.
The more useful half is not the cost. {{InternalRowComparableWrapper.equals}} is
{code:scala}
ordering.compare(row, otherWrapper.row) == 0
{code}
over that same ordering, so "two partition keys are equal" and "two partition
keys sort together" already come from one definition, written down in two
places. Making them share it explicitly is what this ticket is about.
The proposal is a public
{{InternalRowComparableWrapper.orderingFor(dataTypes)}} that returns the cached
ordering, with {{groupedKeyRowOrdering}} delegating to it.
{{DataSourceV2ScanExecBase.outputPartitioning}} is a second site. It calls
{{RowOrdering.createNaturalAscendingOrdering}} directly to sort the partition
keys it is about to hand to {{KeyedPartitioning}}, which is exactly the grouped
key layout order, so it should name that contract rather than rebuild it.
No behaviour change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
