Sudhanva Huruli created SPARK-62025:
---------------------------------------
Summary: Streaming table that reads a materialized view from the
same pipeline duplicates rows on every run
Key: SPARK-62025
URL: https://issues.apache.org/jira/browse/SPARK-62025
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Sudhanva Huruli
A materialized view sales_mv reads a directory of JSON files, and a streaming
table is defined as spark.readStream.table("sales_mv"). After a run with 5
orders and a second run with 3 more, the streaming table holds 13 rows instead
of 8, because the view's files are rewritten on each run and the file source
treats them as new data. Each later run adds another full copy, and the run
still reports COMPLETED. Expected: the pipeline rejects a streaming read of a
materialized view in the same pipeline at startup with a clear error, or each
row appears once.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]