[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751785#comment-16751785 ]
Bryan Cutler commented on SPARK-26412: -------------------------------------- [~mengxr] I think Arrow record batches would be a much more ideal way to connect with other frameworks. Making the conversion to Pandas carries some overhead and the Arrow format/types are more solidly defined. It is also better suited to be used with an iterator - most of the Arrow IPC mechanisms operate on streams of record batches. Is this proposal instead of SPARK-24579 SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks? > Allow Pandas UDF to take an iterator of pd.DataFrames for the entire partition > ------------------------------------------------------------------------------ > > Key: SPARK-26412 > URL: https://issues.apache.org/jira/browse/SPARK-26412 > Project: Spark > Issue Type: New Feature > Components: PySpark > Affects Versions: 3.0.0 > Reporter: Xiangrui Meng > Priority: Major > > Pandas UDF is the ideal connection between PySpark and DL model inference > workload. However, user needs to load the model file first to make > predictions. It is common to see models of size ~100MB or bigger. If the > Pandas UDF execution is limited to batch scope, user need to repeatedly load > the same model for every batch in the same python worker process, which is > inefficient. I created this JIRA to discuss possible solutions. > Essentially we need to support "start()" and "finish()" besides "apply". We > can either provide those interfaces or simply provide users the iterator of > batches in pd.DataFrame and let user code handle it. > cc: [~icexelloss] [~bryanc] [~holdenk] [~hyukjin.kwon] [~ueshin] [~smilegator] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org