Hyukjin Kwon created SPARK-42115: ------------------------------------ Summary: Push down limit through Python UDFs Key: SPARK-42115 URL: https://issues.apache.org/jira/browse/SPARK-42115 Project: Spark Issue Type: Bug Components: PySpark, SQL Affects Versions: 3.4.0 Reporter: Hyukjin Kwon
{code} from pyspark.sql.functions import udf spark.range(10).write.mode("overwrite").parquet("/tmp/abc") @udf(returnType='string') def my_udf(arg): return arg df = spark.read.parquet("/tmp/abc") df = df.limit(10).withColumn("prediction", my_udf(df["id"])).explain() {code} As an example. since Python UDFs are executed asynchronously, so pushing limits benefit the performance. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org