Sudhanva Huruli created SPARK-62025:
---------------------------------------

             Summary: Streaming table that reads a materialized view from the 
same pipeline duplicates rows on every run
                 Key: SPARK-62025
                 URL: https://issues.apache.org/jira/browse/SPARK-62025
             Project: Spark
          Issue Type: Bug
          Components: Declarative Pipelines
    Affects Versions: 4.3.0
            Reporter: Sudhanva Huruli


A materialized view sales_mv reads a directory of JSON files, and a streaming 
table is defined as spark.readStream.table("sales_mv"). After a run with 5 
orders and a second run with 3 more, the streaming table holds 13 rows instead 
of 8, because the view's files are rewritten on each run and the file source 
treats them as new data. Each later run adds another full copy, and the run 
still reports COMPLETED. Expected: the pipeline rejects a streaming read of a 
materialized view in the same pipeline at startup with a clear error, or each 
row appears once.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to