There were some prototypes/discussions being done on top of Spark Streaming, and they were discussing how that would fit with regards to Structured Streaming which was in design mode at that time. See https://issues.apache.org/jira/browse/SPARK-14745 for some details and link to PR.
On Mon, Aug 29, 2016 at 1:13 PM, Tomasz Gawęda <tomasz.gaw...@outlook.com> wrote: > Hi everyone, > > > I wonder if there are plans to implement real time streaming in Spark. I > see that in Spark 2.0 Trigger can have more implementations than > ProcessingTime. > > > In my opinion Real Time streaming (so reaction on every event - like > continous queries in Apache Ignite) will be very useful and will fill gap > that is currently in Spark. Now, if we must implement both real-time > streaming and batch jobs, the streaming must be done in other frameworks as > Spark allows us only to process event in Micro Batches. Matei Zaharia > wrote in Databricks blog about Continuous Applications [1], in my > opinion adding EventTrigger will be next big step to Continuous > Applications. > > > What do you think about it? Are there any plans to implement such > event-based trigger? Of course I can help with implementation, however I'm > just starting learning Spark internals and it will take a while before I > would be able to write something. > > > Pozdrawiam / Best regards, > > Tomek > > > [1] https://databricks.com/blog/2016/07/28/continuous- > applications-evolving-streaming-in-apache-spark-2-0.html > > <https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html> > Continuous Applications: Evolving Streaming in Apache Spark 2.0 > <https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html> > databricks.com > Apache Spark 2.0 lays the foundation for Continuous Applications, a > simplified and unified way to write end-to-end streaming applications that > reacts to data in real-time. > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
