sabbasani commented on code in PR #12726:
URL: https://github.com/apache/gluten/pull/12726#discussion_r4167475101
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PushDownInputFileExpression.scala:
##########
@@ -104,15 +103,44 @@ object PushDownInputFileExpression {
ProjectExec(f.output, FilterExec(newCondition, newChild))
}
+ /**
+ * Returns true when any of the injected metadata attribute names (all
lowercase) matches a
+ * column already present in the scan output at the Velox/case-insensitive
level.
+ *
+ * Velox is case-insensitive. If the scan has a user data column named
e.g. "Input_File_Name"
+ * its lowercase form "input_file_name" collides with the metadata
function of the same name.
+ * Velox rejects a TableScan that maps the same lowercase column name to
both a Regular handle
+ * (data column) and a PartitionKey/metadata handle (file-path metadata).
When this conflict is
+ * detected the scan must fall back to Vanilla so that Spark's own
FilePartitionReader sets the
+ * InputFileBlockHolder thread-local and input_file_name() returns the
correct file path.
+ */
+ private def hasVeloxColumnNameConflict(
+ scanOutput: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ replacedExprs: mutable.Map[String, Alias]): Boolean = {
+ val scanOutputLowerNames =
scanOutput.map(_.name.toLowerCase(java.util.Locale.ROOT)).toSet
Review Comment:
@infvg
Thanks for the additional reproduction. I tested the case with
`spark.sql.caseSensitive=false` and the `CREATE TABLE ... USING parquet` +
`INSERT` flow.
I can reproduce the issue where selecting only `Input_File_Name` can return
the file path instead of the actual column value. I’m investigating the
scan/output mapping in `PushDownInputFileExpression` to make sure the real
`Input_File_Name` attribute is not confused with the `input_file_name()`
metadata column.
I’ll add this exact regression case and update the fix accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
