[
https://issues.apache.org/jira/browse/SPARK-22480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jochen Niebuhr updated SPARK-22480:
-----------------------------------
Description: When you're using the watermark feature, you're forced to
provide an absolute duration to identify late events. For the case we're using
structured streaming for, this is not completely working. In our case, late
events will be possible on the next business day. So I'd like to use 24 hours
watermark for Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I
would suggest is being able to use a function or expression to `withWatermark`
so people can implement this or similar behaviors. If this sounds like a good
idea, I can probably supply a pull request. (was: When you're using the
watermark feature, you're forced to provide an absolute duration to identify
late events. For the case we're using structured streaming for, this is not
completely working. In our case, late events will be possible on the next
business day. So I'd like to use 24 hours watermark for Sunday-Thursday, 72
hours for Friday, 48 for Saturday. If this sounds like a good idea, I can
probably supply a pull request.)
> Dynamic Watermarking
> --------------------
>
> Key: SPARK-22480
> URL: https://issues.apache.org/jira/browse/SPARK-22480
> Project: Spark
> Issue Type: Wish
> Components: Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Jochen Niebuhr
>
> When you're using the watermark feature, you're forced to provide an absolute
> duration to identify late events. For the case we're using structured
> streaming for, this is not completely working. In our case, late events will
> be possible on the next business day. So I'd like to use 24 hours watermark
> for Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I would
> suggest is being able to use a function or expression to `withWatermark` so
> people can implement this or similar behaviors. If this sounds like a good
> idea, I can probably supply a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
