pan3793 commented on code in PR #7792:
URL: https://github.com/apache/kyuubi/pull/7792#discussion_r4385731101
##########
extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala:
##########
@@ -455,6 +455,17 @@ trait LineageParser {
nextColumnsLineage,
inputTablesByPlan)).reduce(mergeColumnsLineage)
+ case p: Filter =>
+ p.condition.foreach {
+ case expression: SubqueryExpression =>
+ extractColumnsLineage(
+ expression.plan,
+ ListMap[Attribute, AttributeSet](),
+ inputTablesByPlan)
Review Comment:
[P2] Handle bare outer-reference projections in filter subqueries
`SELECT * FROM v2_catalog.db.tb1 t1 WHERE t1.col1 = (SELECT t1.col1)`
executes successfully on Spark 3.5.8, but this call now traverses `Project
[outer(col1)] -> OneRowRelation`. `getSelectColumnLineage` handles only `Alias`
and `Attribute`, so the bare `OuterReference` throws `MatchError`.
`transformToLineage` consequently returns `None`, discarding the entire
statement's lineage under both `legacy=false` and `legacy=true`; the base
revision returns the normal input and column lineage. The same predicate also
drops lineage for a successfully executed `INSERT OVERWRITE ... SELECT`.
This is a regression of this change: the incomplete match is pre-existing,
but the new Filter traversal makes it reachable. Add explicit `OuterReference`
handling and cover the bare correlated scalar case in the parameterized tests
for both legacy settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
