[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674147#comment-15674147
]
Amit Sela commented on SPARK-10816:
-----------------------------------
[~rxin] it might be worth taking into account a more generic implementation of
"Merging Windows".
Sessions are merging windows that use "gap duration" to determine when to close
the session, such that if we say that the first element arrives in window
{{[event_time1, event_time1 + gap_duration)}} and the next one at
{{[event_time2, event_time2 + gap_duration)}} and {{event_time1 <
event_time2}}, their combined value will belong to {{[event_time1, event_time2
+ gap_duration)}}, right ?
But the same "merge" of windows could very well be determined by a "close
session" element (using Kafka for example would guarantee order of messages),
or any user defined logic for that matter, as long as the "merge function" is
provided by the user.
Of course providing Sessions API out-of-the-box would prove most useful as it
is the most common, but I don't see any downside to also have a more "advanced"
API here.
Thanks!
> EventTime based sessionization
> ------------------------------
>
> Key: SPARK-10816
> URL: https://issues.apache.org/jira/browse/SPARK-10816
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Reporter: Reynold Xin
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
