aokolnychyi commented on a change in pull request #26751: [SPARK-30107][SQL] Expose nested schema pruning to all V2 sources URL: https://github.com/apache/spark/pull/26751#discussion_r357109500
########## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala ########## @@ -27,15 +27,20 @@ abstract class FileScanBuilder( dataSchema: StructType) extends ScanBuilder with SupportsPushDownRequiredColumns { private val partitionSchema = fileIndex.partitionSchema private val isCaseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis + protected val supportsNestedSchemaPruning: Boolean = false protected var requiredSchema = StructType(dataSchema.fields ++ partitionSchema.fields) override def pruneColumns(requiredSchema: StructType): Unit = { + // [SPARK-30107] While the passed `requiredSchema` always have pruned nested columns, the actual + // data schema of this scan is determined in `readDataSchema`. File formats that don't support + // nested schema pruning, use `requiredSchema` as a reference and perform the pruning partially. this.requiredSchema = requiredSchema } protected def readDataSchema(): StructType = { val requiredNameSet = createRequiredNameSet() - val fields = dataSchema.fields.filter { field => + val schema = if (supportsNestedSchemaPruning) requiredSchema else dataSchema Review comment: The difference between those two is that `requiredSchema` can have pruned nested columns if nested schema pruning is enabled. Therefore, ORC and Parquet must always use data types from `requiredSchema`. If nested schema pruning is disabled, `requiredSchema` will have pruned only top-level columns. In that case, it is still safe to use data types from `requiredSchema`. CSV/JSON/Avro will always use `requiredSchema` as a reference but keep data types from `dataSchema`, so that even if Spark passes a schema with pruned nested columns, we still use data types defined in `dataSchema` to match the behavior before this PR. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org