[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552708#comment-15552708 ]
Jacob Eisinger commented on SPARK-17728: ---------------------------------------- [~hvanhovell], thanks for looking into this! Unless you can think of a better way to ensure the UDF doesn't get executed multiple times, we are going to go with your workaround of: {code} val exploded = as .withColumn("structured_information", explode(array(fUdf('a)))) .withColumn("plus_one", 'structured_information("plusOne")) .withColumn("squared", 'structured_information("squared")) {code} {code} exploded2.explain == Physical Plan == *Project [a#10, structured_information#159, structured_information#159.plusOne AS plus_one#161, structured_information#159.squared AS squared#166] +- Generate explode(array(if (isnull(a#10)) null else UDF(a#10))), true, false, [a#10, structured_information#159] +- *BatchedScan parquet [a#10] Format: ParquetFormat, InputPaths: file:/tmp/as.parquet, PushedFilters: [], ReadSchema: struct<a:int> {code} I reckon it might impact GC a bit with the creation of the extra arrays --- but, that sure beats the cost of running those expensive UDFs! Thanks again for the excellent explanations! > UDFs are run too many times > --------------------------- > > Key: SPARK-17728 > URL: https://issues.apache.org/jira/browse/SPARK-17728 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.0.0 > Environment: Databricks Cloud / Spark 2.0.0 > Reporter: Jacob Eisinger > Priority: Minor > Attachments: over_optimized_udf.html > > > h3. Background > Llonger running processes that might run analytics or contact external > services from UDFs. The response might not just be a field, but instead a > structure of information. When attempting to break out this information, it > is critical that query is optimized correctly. > h3. Steps to Reproduce > # Create some sample data. > # Create a UDF that returns a multiple attributes. > # Run UDF over some data. > # Create new columns from the multiple attributes. > # Observe run time. > h3. Actual Results > The UDF is executed *multiple times* _per row._ > h3. Expected Results > The UDF should only be executed *once* _per row._ > h3. Workaround > Cache the Dataset after UDF execution. > h3. Details > For code and more details, see [^over_optimized_udf.html] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org