[ 
https://issues.apache.org/jira/browse/SPARK-61734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61734:
-----------------------------------
    Labels: correctness pull-request-available  (was: correctness)

> Python UDTF with PARTITION BY gets a partition's rows in several runs when a 
> LEFT ANTI / LEFT SEMI join with a constant-false condition is pushed below it
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61734
>                 URL: https://issues.apache.org/jira/browse/SPARK-61734
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>    Affects Versions: 4.0.0, 4.2.0, 4.1.1
>            Reporter: Yaoxuan Wu
>            Priority: Major
>              Labels: correctness, pull-request-available
>
> {code:python}
> import datetime
> from pyspark.sql import SparkSession, functions as F
> from pyspark.sql.functions import udtf
> spark = (SparkSession.builder
>          .config("spark.sql.shuffle.partitions", "1")
>          .config("spark.sql.execution.pythonUDTF.arrow.enabled", "false")
>          .getOrCreate())
> class CountPerKey:
>     def __init__(self):
>         self.k, self.n = None, 0
>     def eval(self, row):
>         self.k = row["k"]
>         self.n += 1
>     def terminate(self):
>         yield self.k, self.n
> spark.udtf.register(
>     "count_per_key",
>     udtf(CountPerKey, returnType="k: date, n: int", 
> useArrow=False).asDeterministic())
> d1, d2 = datetime.date(2004, 4, 12), datetime.date(2009, 11, 30)
> spark.createDataFrame([(d1,), (d2,), (d1,), (d2,)], "k 
> date").createOrReplaceTempView("t")
> u = spark.sql("SELECT * FROM count_per_key(TABLE(t) PARTITION BY k)")
> print(sorted(u.collect()))
> # [Row(k=datetime.date(2004, 4, 12), n=2), Row(k=datetime.date(2009, 11, 30), 
> n=2)]
> r = u.select(F.col("k").alias("k2"))
> v = u.join(r, F.lit(None).cast("boolean"), "left_anti")   # NOT EXISTS (... 
> WHERE NULL): every row of u
> print(sorted(v.collect()))
> # expected: the same 2 rows as u
> # actual:   [Row(k=datetime.date(2004, 4, 12), n=1), 
> Row(k=datetime.date(2004, 4, 12), n=1),
> #            Row(k=datetime.date(2009, 11, 30), n=1), 
> Row(k=datetime.date(2009, 11, 30), n=1)]
> v.explain()   # no Sort [partition_by_0] below BatchEvalPythonUDTF
> {code}
> Same on 4.0.0, 4.1.1 and 4.2.0. The UDTF sees each partition key in two 
> separate runs, so {{terminate()}} is called once per run. The condition 
> {{(u.k = r.k2) AND false}} gives the same result. In the query where this was 
> found, the condition was a comparison that the optimizer folds to false only 
> after the first pushdown step. Without {{asDeterministic()}} the result is 
> right, because the pushdown requires a deterministic Generate.
> Cause (master d3f4a73a):
> 1. {{PushDownLeftSemiAntiJoin}}, case "LeftSemi/LeftAnti over UnaryNode", 
> pushes the join through {{u}} when 
> {{PushPredicateThroughNonJoin.canPushThrough(u)}} holds, and 
> {{canPushThrough}} returns true for every {{Generate}}. A condition that 
> references no column passes the {{references.subsetOf(u.child.outputSet ++ 
> rightOp.outputSet)}} check. The join lands between the UDTF's {{Generate}} 
> and its {{Project <- Sort [partition_by_0] <- RepartitionByExpression}} input.
> 2. {{EliminateSorts}}, case {{Join}} with a deterministic condition, calls 
> {{recursiveRemoveSort}} on both children and drops that Sort.
> 3. The anti join with the false condition is removed later. The final plan is 
> {{BatchEvalPythonUDTF <- Project <- Exchange 
> hashpartitioning(partition_by_0)}} with no Sort, and the Python runner starts 
> a new partition at every key change.
> The Sort below a UDTF with PARTITION BY / ORDER BY is required for 
> correctness, so it should not be separable from the Generate. Possible fixes: 
> do not push semi / anti joins through a {{Generate}} of a Python UDTF that 
> has partitioning or ordering, or express the per-partition order as a 
> physical {{requiredChildOrdering}} of the UDTF exec node instead of a logical 
> Sort.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to