Guo Wei created SPARK-26636:
-------------------------------
Summary: How to know a partition is ready when using Structured
Streaming
Key: SPARK-26636
URL: https://issues.apache.org/jira/browse/SPARK-26636
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.3.2
Reporter: Guo Wei
When using structured streaming, we use "partitionBy" api to partition the
output data, and use the watermark based on event-time to handle delay records,
but how to tell downstream users that a partition data is ready? For example,
when to write an empty "hadoop.done" file in a paritition directory?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
