dtenedor commented on code in PR #43356: URL: https://github.com/apache/spark/pull/43356#discussion_r1364492809
########## python/pyspark/worker.py: ########## @@ -841,6 +845,63 @@ def _remove_partition_by_exprs(self, arg: Any) -> Any: "the query again." ) + # Compares each UDTF output row against the output schema for this particular UDTF call, + # raising an error if the two are incompatible. + def check_output_row_against_schema(row: Any) -> None: Review Comment: Note: In a previous iteration of this PR, I had a check to see if the schema contained any non-nullable columns in order to enable this. However, I would like to extend these checks to compare provided row values against the expected output schema column types, which currently produce internal exceptions instead of good error messages if they don't match. We would need to check every value in every row in that case, so I figured it was OK to just do that here as well. ########## python/pyspark/worker.py: ########## @@ -841,6 +845,63 @@ def _remove_partition_by_exprs(self, arg: Any) -> Any: "the query again." ) + # Compares each UDTF output row against the output schema for this particular UDTF call, + # raising an error if the two are incompatible. + def check_output_row_against_schema(row: Any) -> None: Review Comment: Note: In a previous iteration of this PR, I had a check to see if the schema contained any non-nullable columns in order to enable this. However, I would like to later extend these checks to compare provided row values against the expected output schema column types, which currently produce internal exceptions instead of good error messages if they don't match. We would need to check every value in every row in that case, so I figured it was OK to just do that here as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org