Guo Wei created SPARK-26636: ------------------------------- Summary: How to know a partition is ready when using Structured Streaming Key: SPARK-26636 URL: https://issues.apache.org/jira/browse/SPARK-26636 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 2.3.2 Reporter: Guo Wei
When using structured streaming, we use "partitionBy" api to partition the output data, and use the watermark based on event-time to handle delay records, but how to tell downstream users that a partition data is ready? For example, when to write an empty "hadoop.done" file in a paritition directory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org