I do not think the issue is with DROP MATERIALIZED VIEW only, but also with CREATE MATERIALIZED VIEW, because neither is supported in Spark. I guess you must have created the view from Hive and are trying to drop it from Spark and that is why you are running to the issue with DROP first.
There is some work in the Iceberg community to add the support to Spark through SQL extensions, and Iceberg support for views and materialization tables. Some recent discussions can be found here [1] along with a WIP Iceberg-Spark PR. [1] https://lists.apache.org/thread/rotmqzmwk5jrcsyxhzjhrvcjs5v3yjcc Thanks, Walaa. On Thu, May 2, 2024 at 4:55 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > An issue I encountered while working with Materialized Views in Spark SQL. > It appears that there is an inconsistency between the behavior of > Materialized Views in Spark SQL and Hive. > > When attempting to execute a statement like DROP MATERIALIZED VIEW IF > EXISTS test.mv in Spark SQL, I encountered a syntax error indicating that > the keyword MATERIALIZED is not recognized. However, the same statement > executes successfully in Hive without any errors. > > pyspark.errors.exceptions.captured.ParseException: > [PARSE_SYNTAX_ERROR] Syntax error at or near 'MATERIALIZED'.(line 1, pos 5) > > == SQL == > DROP MATERIALIZED VIEW IF EXISTS test.mv > -----^^^ > > Here are the versions I am using: > > > > *Hive: 3.1.1Spark: 3.4* > my Spark session: > > spark = SparkSession.builder \ > .appName("test") \ > .enableHiveSupport() \ > .getOrCreate() > > Has anyone seen this behaviour or encountered a similar issue or if there > are any insights into why this discrepancy exists between Spark SQL and > Hive. > > Thanks > > Mich Talebzadeh, > > Technologist | Architect | Data Engineer | Generative AI | FinCrime > > London > United Kingdom > > > view my Linkedin profile > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > Disclaimer: The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner Von Braun)". >
