Assaf Mendelson created SPARK-25709: ---------------------------------------
Summary: Can't append to a data source V2 unless target already has schema information or fixed schema matching dataframe Key: SPARK-25709 URL: https://issues.apache.org/jira/browse/SPARK-25709 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.4.0 Reporter: Assaf Mendelson Let's assume we have a writer data source which writes to a dynamic schema (e.g. writing to a database table). If we have mode append but the table is missing then writing will fail. The reason for this is that in the dataframeWriter save method we have val relation = DataSourceV2Relation.create(source, options) which in turn contains: val reader = source.createReader(options, userSpecifiedSchema) The problem is that this sets the schema from the reader portion of the data source. If the data source can't infer the schema then we will fail to write (even thought the schema is actually part of the dataframe). -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org