dtenedor commented on code in PR #43356:
URL: https://github.com/apache/spark/pull/43356#discussion_r1358990069
##########
python/pyspark/worker.py:
##########
@@ -841,6 +841,27 @@ def _remove_partition_by_exprs(self, arg: Any) -> Any:
"the query again."
)
+ # Compute the set of UDTF result columns whose types are not nullable.
+ # We will check that the UDTF does not return None values for these
columns below.
+ non_nullable_result_cols = set()
+ for i, field in enumerate(return_type):
Review Comment:
Thanks for your reviews! I updated this PR to also check recursively for
`None` values within array, struct, and map values as well.
We can certainly consider other types of UDFs as well later (including Scala
UDFs); it seems possible to decouple that work from Python UDTFs here, so I'll
leave this PR to focus on the latter for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
