[ https://issues.apache.org/jira/browse/SPARK-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136250#comment-14136250 ]
Apache Spark commented on SPARK-3554: ------------------------------------- User 'davies' has created a pull request for this issue: https://github.com/apache/spark/pull/2417 > handle large dataset in closure of PySpark > ------------------------------------------ > > Key: SPARK-3554 > URL: https://issues.apache.org/jira/browse/SPARK-3554 > Project: Spark > Issue Type: Improvement > Components: PySpark > Reporter: Davies Liu > Assignee: Davies Liu > > Sometimes there are large dataset used in closure and user forget to use > broadcast for it, then the serialized command will become huge. > py4j can not handle large objects efficiently, we should compress the > serialized command and user broadcast for it if it's huge. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org