[ https://issues.apache.org/jira/browse/SPARK-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-20734: --------------------------------- Labels: bulk-closed (was: ) > Structured Streaming spark.sql.streaming.schemaInference not handling schema > changes > ------------------------------------------------------------------------------------ > > Key: SPARK-20734 > URL: https://issues.apache.org/jira/browse/SPARK-20734 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming > Affects Versions: 2.1.1 > Reporter: Ram > Priority: Major > Labels: bulk-closed > > sparkSession.config("spark.sql.streaming.schemaInference", > true).getOrCreate(); > Dataset<Row> dataset = > sparkSession.readStream().parquet("file:/files-to-process"); > StreamingQuery streamingQuery = > dataset.writeStream().option("checkpointLocation", > "file:/checkpoint-location") > .outputMode(Append()).start("file:/save-parquet-files"); > streamingQuery.awaitTermination(); > After streaming query started If there's a schema changes on new paruet > files under files-to-process directory. Structured Streaming not writing new > schema changes. Is it possible to handle these schema changes in Structured > Streaming. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org