Tathagata Das created SPARK-14741:
-------------------------------------

             Summary: Streaming from partitioned directory structure captures 
unintended partition columns
                 Key: SPARK-14741
                 URL: https://issues.apache.org/jira/browse/SPARK-14741
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: Tathagata Das
            Assignee: Tathagata Das


Consider the following directory structure

    dir/col=X/some-files

If we create a text format streaming dataframe on {dir/col=X/}, then it should 
not consider as partitioning in columns. Even though the streaming dataframe 
does not do so, the generated batch dataframes pick up col as a partitioning 
columns, causing mismatch streaming source schema and generated df schema. This 
leads to runtime failure: 


18:55:11.262 ERROR org.apache.spark.sql.execution.streaming.StreamExecution: 
Query query-0 terminated with error
java.lang.AssertionError: assertion failed: Invalid batch: c#2 != c#7,type#8





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to