I don't understand what kind of low level control that DStream can do while Structure Streaming can not
Thanks Chang On Wednesday, July 27, 2016, Matei Zaharia <matei.zaha...@gmail.com> wrote: > Yup, they will definitely coexist. Structured Streaming is currently alpha > and will probably be complete in the next few releases, but Spark Streaming > will continue to exist, because it gives the user more low-level control. > It's similar to DataFrames vs RDDs (RDDs are the lower-level API for when > you want control, while DataFrames do more optimizations automatically by > restricting the computation model). > > Matei > > On Jul 27, 2016, at 12:03 AM, Ofir Manor <ofir.ma...@equalum.io > <javascript:_e(%7B%7D,'cvml','ofir.ma...@equalum.io');>> wrote: > > Structured Streaming in 2.0 is declared as alpha - plenty of bits still > missing: > > http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html > I assume that it will be declared stable / GA in a future 2.x release, and > then it will co-exist with DStream for quite a while before someone will > suggest to start a deprecation process that will eventually lead to its > removal... > As a user, I guess we will need to apply judgement about when to switch to > Structured Streaming - each of us have a different risk/value tradeoff, > based on our specific situation... > > Ofir Manor > > Co-Founder & CTO | Equalum > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > <javascript:_e(%7B%7D,'cvml','ofir.ma...@equalum.io');> > > On Wed, Jul 27, 2016 at 8:02 AM, Chang Chen <baibaic...@gmail.com > <javascript:_e(%7B%7D,'cvml','baibaic...@gmail.com');>> wrote: > >> Hi guys >> >> Structure Stream is coming with spark 2.0, but I noticed that DStream is >> still here >> >> What's the future of the DStream, will it be deprecated and removed >> eventually? Or co-existed with Structure Stream forever? >> >> Thanks >> Chang >> >> > >
