Forgive the slight tangent… For anyone following this thread who may be wondering about a quick, simple solution they can apply (and a walk-through on how) for more straight-forward sessionization needs:
There’s a nice section on sessionization in “Advanced Analytics with Spark”, by Ryza, Laserson, Owen, and Wills (starts on p.167). It was excellent for my job that needed to take events, get them in time order, and calculate the time between them (that particular job’s def of a “session”). I used their groupByKeyAndSortValues() function. As the authors state, “Work is progressing on Spark JIRA SPARK-3655 to add a transformation like this to core Spark.” From: Ofir Manor <ofir.ma...@equalum.io> Date: Thursday, November 17, 2016 at 9:57 AM To: "assaf.mendelson" <assaf.mendel...@rsa.com> Cc: dev <dev@spark.apache.org> Subject: Re: structured streaming and window functions I agree with you, I think that once we will have sessionization, we could aim for richer processing capabilities per session. As far as I image it, a session is an ordered sequence of data, that we could apply computation on it (like CEP). Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286<tel:%2B972-54-7801286> | Email: ofir.ma...@equalum.io<mailto:ofir.ma...@equalum.io> On Thu, Nov 17, 2016 at 5:16 PM, assaf.mendelson <assaf.mendel...@rsa.com<mailto:assaf.mendel...@rsa.com>> wrote: It is true that this is sessionizing but I brought it as an example for finding an ordered pattern in the data. In general, using simple window (e.g. 24 hours) in structured streaming is explain in the grouping by time and is very clear. What I was trying to figure out is how to do streaming of cases where you actually have to have some sorting to find patterns, especially when some of the data may come in late. I was trying to figure out if there is plan to support this and if so, what would be the performance implications. Assaf. From: Ofir Manor [via Apache Spark Developers List] [mailto:ml-node+<mailto:ml-node%2B>[hidden email]<http:///user/SendEmail.jtp?type=node&node=19936&i=0>] Sent: Thursday, November 17, 2016 5:13 PM To: Mendelson, Assaf Subject: Re: structured streaming and window functions Assaf, I think what you are describing is actually sessionizing, by user, where a session is ended by a successful login event. On each session, you want to count number of failed login events. If so, this is tracked by https://issues.apache.org/jira/browse/SPARK-10816 (didn't start yet) Ofir Manor Co-Founder & CTO | Equalum Mobile: <a href="<a href="tel:%2B972-54-7801286">tel:%2B972-54-7801286" value="+972507470820<tel:%2B972507470820>" target="_blank">+972-54-7801286<tel:%2B972-54-7801286> | Email: [hidden email]<http:///user/SendEmail.jtp?type=node&node=19935&i=0> On Thu, Nov 17, 2016 at 2:52 PM, assaf.mendelson <[hidden email]<http:///user/SendEmail.jtp?type=node&node=19935&i=1>> wrote: Is there a plan to support sql window functions? I will give an example of use: Let’s say we have login logs. What we want to do is for each user we would want to add the number of failed logins for each successful login. How would you do it with structured streaming? As this is currently not supported, is there a plan on how to support it in the future? Assaf. From: Herman van Hövell tot Westerflier-2 [via Apache Spark Developers List] [mailto:[hidden email]<http:///user/SendEmail.jtp?type=node&node=19935&i=2>[hidden email]<http://user/SendEmail.jtp?type=node&node=19934&i=0>] Sent: Thursday, November 17, 2016 1:27 PM To: Mendelson, Assaf Subject: Re: structured streaming and window functions What kind of window functions are we talking about? Structured streaming only supports time window aggregates, not the more general sql window function (sum(x) over (partition by ... order by ...)) aggregates. The basic idea is that you use incremental aggregation and store the aggregation buffer (not the end result) in a state store after each increment. When an new batch comes in, you perform aggregation on that batch, merge the result of that aggregation with the buffer in the state store, update the state store and return the new result. This is much harder than it sounds, because you need to maintain state in a fault tolerant way and you need to have some eviction policy (watermarks for instance) for aggregation buffers to prevent the state store from reaching an infinite size. On Thu, Nov 17, 2016 at 12:19 AM, assaf.mendelson <[hidden email]<http://user/SendEmail.jtp?type=node&node=19933&i=0>> wrote: Hi, I have been trying to figure out how structured streaming handles window functions efficiently. The portion I understand is that whenever new data arrived, it is grouped by the time and the aggregated data is added to the state. However, unlike operations like sum etc. window functions need the original data and can change when data arrives late. So if I understand correctly, this would mean that we would have to save the original data and rerun on it to calculate the window function every time new data arrives. Is this correct? Are there ways to go around this issue? Assaf. ________________________________ View this message in context: structured streaming and window functions<http://apache-spark-developers-list.1001551.n3.nabble.com/structured-streaming-and-window-functions-tp19930.html> Sent from the Apache Spark Developers List mailing list archive<http://apache-spark-developers-list.1001551.n3.nabble.com/> at Nabble.com. ________________________________ If you reply to this email, your message will be added to the discussion below: http://apache-spark-developers-list.1001551.n3.nabble.com/structured-streaming-and-window-functions-tp19930p19933.html To start a new topic under Apache Spark Developers List, email [hidden email]<http://user/SendEmail.jtp?type=node&node=19934&i=1> To unsubscribe from Apache Spark Developers List, click here. NAML<http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> ________________________________ View this message in context: RE: structured streaming and window functions<http://apache-spark-developers-list.1001551.n3.nabble.com/structured-streaming-and-window-functions-tp19930p19934.html> Sent from the Apache Spark Developers List mailing list archive<http://apache-spark-developers-list.1001551.n3.nabble.com/> at Nabble.com. ________________________________ If you reply to this email, your message will be added to the discussion below: http://apache-spark-developers-list.1001551.n3.nabble.com/structured-streaming-and-window-functions-tp19930p19935.html To start a new topic under Apache Spark Developers List, email [hidden email]<http:///user/SendEmail.jtp?type=node&node=19936&i=1> To unsubscribe from Apache Spark Developers List, click here. NAML<http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> ________________________________ View this message in context: RE: structured streaming and window functions<http://apache-spark-developers-list.1001551.n3.nabble.com/structured-streaming-and-window-functions-tp19930p19936.html> Sent from the Apache Spark Developers List mailing list archive<http://apache-spark-developers-list.1001551.n3.nabble.com/> at Nabble.com.
