sabbasani commented on code in PR #12726:
URL: https://github.com/apache/gluten/pull/12726#discussion_r4218588667
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala:
##########
@@ -117,15 +118,28 @@ trait BasicScanExecTransformer extends
LeafTransformSupport with BaseDataSource
val metadataFromSpark = getMetadataColumns().map(_.name)
- val inputFileRelatedMetadataKeys = Seq(
- InputFileName().prettyName,
- InputFileBlockStart().prettyName,
- InputFileBlockLength().prettyName)
-
- val neededInputFileRelatedMetadataKeys =
- inputFileRelatedMetadataKeys.filter(k => output.exists(_.name == k))
+ // In addition to the "proper" Spark metadata columns
(FileSourceConstantMetadataAttribute),
+ // PreOffload may have injected AttributeReferences to carry input-file
function values.
+ // These attributes are identified by isInjectedInputFileAttr (presence of
+ // GLUTEN_INPUT_FILE_COL_ATTR_KEY in metadata) rather than by name,
because under
+ // caseSensitive=false the schema name may have been mangled (e.g.
+ // "__gluten_input_file_col__input_file_name__") to avoid colliding with a
user column that
+ // lowercases to the same name. The canonical prettyName is recovered via
+ // injectedInputFileCanonName and used as the key in the Velox split
infoColumns map, while
+ // the schema name (attr.name) is the key under which Velox looks up the
value.
+ //
+ // When no mangling occurred (caseSensitive=true or no collision) schema
name == canonical
+ // prettyName and both lookups use the same string.
+ //
+ // IMPORTANT: do NOT use normalizeColName here. A user column like
"Input_File_Name" must
+ // never match -- we identify injected attrs exclusively via their
metadata key.
+ val injectedInputFileCols: Seq[(String, String)] = output.collect {
+ case a if PushDownInputFileExpression.isInjectedInputFileAttr(a) =>
+ // (schemaName, canonicalPrettyName)
+ a.name -> PushDownInputFileExpression.injectedInputFileCanonName(a)
+ }
Review Comment:
I have tested locally
```
ss.sql("""
| SELECT id, input_file_name() AS fname
| FROM (
| SELECT * FROM left_table
| UNION ALL
| SELECT * FROM left_table
| ) u
| """).show(false)
```
```
+---+---------------------------------------------------------------------------------------------+
|id |fname
|
+---+---------------------------------------------------------------------------------------------+
|2
|file:///tmp/gluten_join_test_1/part-00001-da259739-0934-4fe9-b689-15d7414dbb0b.snappy.parquet|
|1
|file:///tmp/gluten_join_test_1/part-00000-079afa01-efe4-4ab9-8a40-c33a8984dfac.snappy.parquet|
|2
|file:///tmp/gluten_join_test_1/part-00001-da259739-0934-4fe9-b689-15d7414dbb0b.snappy.parquet|
|1
|file:///tmp/gluten_join_test_1/part-00000-079afa01-efe4-4ab9-8a40-c33a8984dfac.snappy.parquet|
+---+---------------------------------------------------------------------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
