I have been doing several Spark PoC projects recently and the latest one involved the new 2.0 experimental feature Structured Streaming. My PoC ended up being a non-starter as I quickly realized the stream to stream joins are not implemented yet. I believe this feature will be immensely powerful and allow applications to migrate from batch to streaming with ease, which is exactly what I would like to accomplish.
I came across an interesting article (link below) from another streaming platform called AMPS which I believe describes a common use case where Spark's Structured Streaming would be perfect for and I hope the Spark dev team is looking at the AMPS platform for some inspiration. http://www.crankuptheamps.com/blog/posts/2014/04/07/real-time-streaming-joins-reinvented/ I am posting this for two reason: - Share AMPS approach to this problem and get the communities thoughts on Spark's Structured Streaming approach if it will provide similar semantics - Inquire about the availability of stream to stream joins and where I can follow along (is there a JIRA or dev mailing list topic I can follow). Thank you in advance for any replies. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Structured-Streaming-Comparison-to-AMPS-tp27303.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
