[ 
https://issues.apache.org/jira/browse/SPARK-61768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61768:
-----------------------------------
    Labels: correctness pull-request-available  (was: correctness)

> DataFrame API: a column referenced inside a lambda function resolves to a 
> same-named alias of the same select list instead of the input column
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61768
>                 URL: https://issues.apache.org/jira/browse/SPARK-61768
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, PySpark, SQL
>    Affects Versions: 4.0.0, 4.2.0, 4.1.1
>            Reporter: Yaoxuan Wu
>            Priority: Major
>              Labels: correctness, pull-request-available
>
> In the DataFrame API, a column referenced inside a lambda function resolves 
> to a same-named alias defined earlier in the same select list (implicit 
> lateral column alias), not to the input column. Outside a lambda, the same 
> reference resolves to the input column, and the equivalent SQL query is 
> correct. The result is wrong when the types match; when they don't, the query 
> fails analysis instead.
> {code:python}
> from pyspark.sql import SparkSession, functions as F
> spark = SparkSession.builder.getOrCreate()
> df = spark.sql("SELECT 1 AS a, 100 AS b, array(1, 2) AS arr")
> df.createOrReplaceTempView("t")
> spark.sql("SELECT b AS a, transform(arr, x -> x + a) AS c FROM t").collect()
> # [Row(a=100, c=[2, 3])]                      correct: a is the input column 
> (1)
> df.select(F.col("b").alias("a"), F.transform("arr", lambda x: x + 
> F.col("a")).alias("c")).collect()
> # [Row(a=100, c=[101, 102])]                  expected c=[2, 3]
> df.withColumns({"a": F.col("b"), "c": F.transform("arr", lambda x: x + 
> F.col("a"))}).collect()
> # [Row(a=100, b=100, arr=[1, 2], c=[101, 102])]   expected c=[2, 3]
> df.withColumns({"a": F.col("b"), "c": F.col("a") + 0}).collect()
> # [Row(a=100, b=100, arr=[1, 2], c=1)]        correct outside a lambda
> m = spark.sql("SELECT map(1, 10) AS m, 7 AS b")
> m.withColumns({"m": F.col("b"),
>                "c": F.map_filter("m", lambda k, v: 
> F.array_contains(F.map_keys(F.col("m")), k))}).collect()
> # [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve 
> "map_keys(lateralAliasReference(m))" ...
> {code}
> Same on 3.5.8, 4.0.0, 4.1.1 and 4.2.0 (classic), and on 4.2.0 with Spark 
> Connect. Lateral column aliases should apply only to names the input does not 
> have, and {{withColumns}} is documented to evaluate every expression against 
> the input DataFrame.
> Analyzed plan of the {{select}} above: the lambda body refers to the new 
> alias {{a#6}}, not the input {{a#0}}:
> {noformat}
> Project [a#6, transform(arr#2, lambdafunction((lambda x_1#8 + a#6), lambda 
> x_1#8, false)) AS c#7]
> +- Project [a#0, b#1, arr#2, b#1 AS a#6]
>    +- Project [1 AS a#0, 100 AS b#1, array(1, 2) AS arr#2]
> {noformat}
> Cause (master d3f4a73a, ColumnResolutionHelper.scala):
> 1. {{resolveExpression}} skips unbound lambda functions ({{case f: 
> LambdaFunction if !f.bound => f}}). An input column used inside a lambda body 
> therefore stays an {{UnresolvedAttribute}} until {{ResolveLambdaVariables}} 
> binds the lambda.
> 2. In the same {{ResolveReferences}} pass, {{resolveLateralColumnAlias}} 
> walks the whole select list with {{transformUp}} and does not skip lambdas. 
> It matches that {{UnresolvedAttribute}} to the earlier alias {{b AS a}} and 
> wraps it in a {{LateralColumnAliasReference}}, before the reference has been 
> tried against the input.
> 3. SQL escapes because the parser builds every name in a lambda body as an 
> {{UnresolvedNamedLambdaVariable}}: the parsed plan has {{lambda 'a}}, which 
> {{ResolveLambdaVariables}} later resolves against the input. The Column API 
> builds an ordinary {{UnresolvedAttribute}} ({{'a}}).
> A possible fix is to make {{resolveLateralColumnAlias}} skip unbound 
> {{LambdaFunction}} bodies, as {{resolveExpression}} does. A lateral alias 
> inside a lambda would then only be used if the name doesn't resolve against 
> the input once the lambda is bound.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to