[
https://issues.apache.org/jira/browse/SPARK-22480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jochen Niebuhr updated SPARK-22480:
-----------------------------------
Description: When you're using the watermark feature, you're forced to
provide an absolute duration to identify late events. For the case we're using
structured streaming for, this is not completely working. In our case, late
events will be possible on the next business day. So I'd like to use 24 hours
watermark for Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I
would suggest is being able to use a function or expression to
{{withWatermark}} so people can implement this or similar behaviors. If this
sounds like a good idea, I can probably supply a pull request. (was: When
you're using the watermark feature, you're forced to provide an absolute
duration to identify late events. For the case we're using structured streaming
for, this is not completely working. In our case, late events will be possible
on the next business day. So I'd like to use 24 hours watermark for
Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I would suggest is
being able to use a function or expression to `withWatermark` so people can
implement this or similar behaviors. If this sounds like a good idea, I can
probably supply a pull request.)
> Dynamic Watermarking
> --------------------
>
> Key: SPARK-22480
> URL: https://issues.apache.org/jira/browse/SPARK-22480
> Project: Spark
> Issue Type: Wish
> Components: Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Jochen Niebuhr
>
> When you're using the watermark feature, you're forced to provide an absolute
> duration to identify late events. For the case we're using structured
> streaming for, this is not completely working. In our case, late events will
> be possible on the next business day. So I'd like to use 24 hours watermark
> for Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I would
> suggest is being able to use a function or expression to {{withWatermark}} so
> people can implement this or similar behaviors. If this sounds like a good
> idea, I can probably supply a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
