Hi team, I would like to know the behaviour of Append & Overwrite modes when table is not present and whether automatic table creation is supported/unsupported when df.write is used in Spark 3 when the underlying custom datasource is using SupportCatalogOptions.
As per my knowledge, in the current implementation in master, df.write in Append and Overwrite mode tries to load the table and look for the schema of the table. Hence the table is expected to be created beforehand and hence the table will not automatically be created. Attaching the below code link for reference. Code link - https://github.com/apache/spark/blob/b065c945fe27dd5869b39bfeaad8e2b23a8835b5/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L287 This is slightly different from the behaviour we observed in Spark 2 - https://lists.apache.org/thread/y468ngqhfhxhv0fygvwvy8r3g4sw9v7n Please confirm if I am correct and if this is the intended behaviour in Spark 3. Thanks and regards, Sourabh Badhya
