Johan Lasperas created SPARK-48308: -------------------------------------- Summary: Unify getting data schema without partition columns in FileSourceStrategy Key: SPARK-48308 URL: https://issues.apache.org/jira/browse/SPARK-48308 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.5.1 Reporter: Johan Lasperas
In [FileSourceStrategy,|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L191] the schema of the data excluding partition columns is computed 2 times in a slightly different way: ``` val dataColumnsWithoutPartitionCols = dataColumns.filterNot(partitionSet.contains) ``` vs ``` val readDataColumns = dataColumns .filterNot(partitionColumns.contains) ``` This should be unified -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org