[ https://issues.apache.org/jira/browse/SPARK-17986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yanbo Liang resolved SPARK-17986. --------------------------------- Resolution: Fixed Assignee: Drew Robb Fix Version/s: 2.1.0 2.0.2 > SQLTransformer leaks temporary tables > ------------------------------------- > > Key: SPARK-17986 > URL: https://issues.apache.org/jira/browse/SPARK-17986 > Project: Spark > Issue Type: Bug > Components: ML > Affects Versions: 2.0.1 > Reporter: Drew Robb > Assignee: Drew Robb > Priority: Minor > Fix For: 2.0.2, 2.1.0 > > > The SQLTransformer creates a temporary table when called, and does not delete > this temporary table. When using a SQLTransformer in a long running Spark > Streaming task, these temporary tables accumulate. > I believe that the fix would be as simple as calling > `dataset.sparkSession.catalog.dropTempView(tableName)` in the last part of > `transform`: > https://github.com/apache/spark/blob/v2.0.1/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala#L65. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org