[
https://issues.apache.org/jira/browse/SPARK-59187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-59187:
-------------------------------
Affects Version/s: 4.4.0
> Carry the partition key data types on KeyedPartitioning
> -------------------------------------------------------
>
> Key: SPARK-59187
> URL: https://issues.apache.org/jira/browse/SPARK-59187
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Peter Toth
> Priority: Major
>
> {{KeyedPartitioning}} carries its partition keys as
> {{InternalRowComparableWrapper}} rows and no types. Every reader that needs
> the types samples them from the first key row, via {{keyDataTypes}}:
> {code:scala}
> @transient lazy val keyDataTypes: Seq[DataType] =
> partitionKeys.headOption.map(_.dataTypes).getOrElse(expressionDataTypes)
> {code}
> Two problems follow from that.
> * With no key row there is nothing to sample, so it falls back to the
> partition expressions' own types. Those describe the keys only while the
> expressions do, and after a join reduced both sides' keys they no longer do
> (SPARK-59121). SPARK-59176 was a query that failed because one caller held
> that fallback against a real answer.
> * Nothing checks that the remaining key rows agree with the first one.
> The types are known where the keys are built, and are already computed and
> dropped there:
> {code:scala}
> def apply(expressions: Seq[Expression], partitionKeys: Seq[InternalRow]):
> KeyedPartitioning = {
> val dataTypes = expressions.map(_.dataType)
> val comparableKeyWrapperFactory =
>
> InternalRowComparableWrapper.getInternalRowComparableWrapperFactory(dataTypes)
> ...
> {code}
> The proposal is to carry them on the partitioning beside {{expressions}}, and
> to have {{projectKeys}}, {{reduceKeys}},
> {{KeyedShuffleSpec.createPartitioning}} and {{GroupPartitionsExec}} carry
> them through the transformations they apply to the keys. Then no reader
> samples a key row, and a partitioning with no key still reports what its keys
> would hold.
> This subsumes SPARK-59176's fix, which leaves a keyless side out of one
> comparison rather than giving it an answer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
