[ https://issues.apache.org/jira/browse/SPARK-32063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17143233#comment-17143233 ]
L. C. Hsieh commented on SPARK-32063: ------------------------------------- For 1 and 2, it seems all related to performance. In Spark, we have caching mechanism that materializes complex query. I think it can complement the shortage of temporary view. For 3, I'm not sure about this point. Can you elaborate it more? > Spark native temporary table > ---------------------------- > > Key: SPARK-32063 > URL: https://issues.apache.org/jira/browse/SPARK-32063 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 3.1.0 > Reporter: Lantao Jin > Priority: Major > > Many databases and data warehouse SQL engines support temporary tables. A > temporary table, as its named implied, is a short-lived table that its life > will be only for current session. > In Spark, there is no temporary table. the DDL “CREATE TEMPORARY TABLE AS > SELECT” will create a temporary view. A temporary view is totally different > with a temporary table. > A temporary view is just a VIEW. It doesn’t materialize data in storage. So > it has below shortage: > # View will not give improved performance. Materialize intermediate data in > temporary tables for a complex query will accurate queries, especially in an > ETL pipeline. > # View which calls other views can cause severe performance issues. Even, > executing a very complex view may fail in Spark. > # Temporary view has no database namespace. In some complex ETL pipelines or > data warehouse applications, without database prefix is not convenient. It > needs some tables which only used in current session. > > More details are described in [Design > Docs|https://docs.google.com/document/d/1RS4Q3VbxlZ_Yy0fdWgTJ-k0QxFd1dToCqpLAYvIJ34U/edit?usp=sharing] -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org