LuciferYang opened a new pull request, #13006:
URL: https://github.com/apache/gluten/pull/13006
## What changes are proposed in this pull request?
#12954 removed 19 `SparkShims` methods whose four implementations had become
identical once Spark 3.3 left. Two more belong in that set, and they were easy
to miss then because the four bodies are not textually equal.
`structFromAttributes` and `attributesFromStruct` are declared abstract on
the trait. spark34 spells the bodies out; spark35, spark40 and spark41 delegate
to `DataTypeUtils.fromAttributes` and `DataTypeUtils.toAttributes`, whose
implementations are spark34's bodies verbatim:
```scala
def fromAttributes(attributes: Seq[Attribute]): StructType =
StructType(attributes.map(a => StructField(a.name, a.dataType, a.nullable,
a.metadata)))
def toAttribute(field: StructField): AttributeReference =
AttributeReference(field.name, field.dataType, field.nullable,
field.metadata)()
def toAttributes(schema: StructType): Seq[AttributeReference] =
schema.map(toAttribute)
```
So the two become concrete trait methods and the eight overrides go. The
trait body has to be spark34's form rather than the delegation, because
`DataTypeUtils` does not exist in the 3.4 catalyst jar. That asymmetry is the
whole reason these two sat behind the shim.
One difference is worth naming even though nothing depends on it: on 3.5,
4.0 and 4.1 `attributesFromStruct` used to come back list-backed via
`schema.map`, and now comes back array-backed via `structType.fields.map`. Same
elements in the same order, and every caller takes it as a `Seq[Attribute]`;
the declared return type was already `Seq[Attribute]` rather than
`DataTypeUtils`' narrower `Seq[AttributeReference]`, so no signature moves
either.
`invalidBucketFile` comes along as a `protected def`. It was a
fourfold-duplicated `private def` that needs only `SparkException`, called from
`filesGroupedToBuckets`, which has to stay per-module because it calls the
per-module `PartitionedFileUtilShim`. No signature changes, so no caller moves.
Two others look collapsible and are not, recorded here so nobody retries.
`createParquetFilters` has four byte-identical bodies, but
`LegacyBehaviorPolicy` sits under `SQLConf` on 3.4 and stands alone from 3.5
on, so no single spelling compiles everywhere. `getShuffleBlockFetcherIterator`
constructs `GlutenShuffleBlockFetcherIterator`, which exists once per shim
module and is invisible to `shims/common`, since the dependency runs spark3x to
common rather than the other way.
Worth knowing for whoever picks up the next round: of the 27 abstract
members left, 14 are identical across 3.5, 4.0 and 4.1 with 3.4 as the only
outlier. Dropping 3.4 would collapse most of the trait; continuing to mine 3.3
residue will not.
## How was this patch tested?
`clean test-compile` on Spark 3.4, 3.5, 4.0 and 4.1 with `-Pbackends-velox
-Pspark-ut -Piceberg -Pdelta`, Scala 2.13, and the 3.5 run adds
`-Pbackends-clickhouse`, that being the only profile clickhouse builds on.
That is the check that matters here. The change is a compile-time move, and
the `DataTypeUtils` calls it replaces exist on three of the four versions but
not the fourth, so picking the wrong body fails to build rather than
misbehaving at runtime.
`spotless:apply` produced no changes beyond the edits. No suite was run:
same expressions, same signatures, callers untouched.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5
Related issue: #13003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
