felipepessoto commented on PR #13042: URL: https://github.com/apache/gluten/pull/13042#issuecomment-5733607615
Replying to [this review](https://github.com/apache/gluten/pull/13042#pullrequestreview-5250547119): The name-based check matches Delta's reader semantics here. In both [Delta 3.3.2](https://github.com/delta-io/delta/blob/v3.3.2/spark/src/main/scala/org/apache/spark/sql/delta/DeltaParquetFileFormat.scala) and [Delta 4.0.1](https://github.com/delta-io/delta/blob/v4.0.1/spark/src/main/scala/org/apache/spark/sql/delta/DeltaParquetFileFormat.scala), `buildReaderWithPartitionValues` identifies `__delta_internal_row_index` and `__delta_internal_is_row_deleted` by exact field name. Their definitions are plain `StructField`s without a generated-column marker. `GlutenDeltaParquetFileFormat` follows the same logic. When `useMetadataRowIndex=false`, the JVM reader therefore also interprets matching required fields as synthetic metadata, rather than distinguishing physical user columns. A metadata-marker requirement in the fallback would miss legitimate synthetic fields. The rule is already restricted to Delta scans, so unrelated Parquet columns are unaffected. I'm keeping the fallback unchanged to preserve Delta's JVM reader behavior. This does not rely on assuming that table creation prohibits these names; the internal-column list alone does not establish that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
