sabbasani commented on code in PR #12726:
URL: https://github.com/apache/gluten/pull/12726#discussion_r4167475101
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PushDownInputFileExpression.scala:
##########
@@ -104,15 +103,44 @@ object PushDownInputFileExpression {
ProjectExec(f.output, FilterExec(newCondition, newChild))
}
+ /**
+ * Returns true when any of the injected metadata attribute names (all
lowercase) matches a
+ * column already present in the scan output at the Velox/case-insensitive
level.
+ *
+ * Velox is case-insensitive. If the scan has a user data column named
e.g. "Input_File_Name"
+ * its lowercase form "input_file_name" collides with the metadata
function of the same name.
+ * Velox rejects a TableScan that maps the same lowercase column name to
both a Regular handle
+ * (data column) and a PartitionKey/metadata handle (file-path metadata).
When this conflict is
+ * detected the scan must fall back to Vanilla so that Spark's own
FilePartitionReader sets the
+ * InputFileBlockHolder thread-local and input_file_name() returns the
correct file path.
+ */
+ private def hasVeloxColumnNameConflict(
+ scanOutput: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ replacedExprs: mutable.Map[String, Alias]): Boolean = {
+ val scanOutputLowerNames =
scanOutput.map(_.name.toLowerCase(java.util.Locale.ROOT)).toSet
Review Comment:
@infvg Thanks for pointing this out. Fixed the case-sensitive handling to
use Spark's configured resolver instead of unconditional lowercasing.
I also added regression tests covering `Input_File_Name` vs
`input_file_name()` with both `spark.sql.caseSensitive=true` and `false`. The
queries now return the correct column values and metadata file paths through
the Gluten/Velox path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
