Zouxxyy commented on PR #10838:
URL:
https://github.com/apache/incubator-gluten/pull/10838#issuecomment-3381767298
> Should we also refactor GlutenWholeStageColumnarRDD?
For GlutenWholeStageColumnarRDD, I think these modifications below are
sufficient. Do you think it needs to be modified in this PR as well? (Besieds.
NativeFileScanColumnarRDD doesn't seem to be used. Should I delete it?)
```scala
case class FirstZippedPartitionsPartition(
index: Int,
inputPartition: Partition,
inputColumnarRDDPartitions: Seq[Partition] = Seq.empty)
extends Partition
class GlutenWholeStageColumnarRDD(
@transient sc: SparkContext,
@transient private val inputPartitions: Seq[Partition],
```
> Does soft-affinity still work correctly after the change?
do you mean getPreferredLocations? I haven't modified its logic, so it still
applies
e.g. in GlutenWholeStageColumnarRDD, It will do the correct cast.
```
override def getPreferredLocations(split: Partition): Seq[String] = {
castNativePartition(split)._1.preferredLocations()
}
```
Or just like `FileScanRDD` in apache spark
```scala
override protected def getPreferredLocations(split: RDDPartition):
Seq[String] = {
split.asInstanceOf[FilePartition].preferredLocations()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
