+1 on the Dstreams deprecation proposal On Fri, Jan 13, 2023 at 10:47 AM Jerry Peng <jerry.boyang.p...@gmail.com> wrote:
> +1 in general for marking the DStreams API as deprecated > > Jungtaek, can you please provide / elaborate on the concrete actions you > intend on taking for the depreciation process? > > Best, > > Jerry > > On Thu, Jan 12, 2023 at 11:16 PM L. C. Hsieh <vii...@gmail.com> wrote: > >> +1 >> >> On Thu, Jan 12, 2023 at 10:39 PM Jungtaek Lim >> <kabhwan.opensou...@gmail.com> wrote: >> > >> > Yes, exactly. I'm sorry to bring confusion - should have clarified >> action items on the proposal. >> > >> > On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun <dongjoon.h...@gmail.com> >> wrote: >> >> >> >> Then, could you elaborate `the proposed code change` specifically? >> >> Maybe, usual deprecation warning logs and annotation on the API? >> >> >> >> >> >> On Thu, Jan 12, 2023 at 10:05 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> >> >>> Maybe I need to clarify - my proposal is "explicitly" deprecating it, >> which incurs code change for sure. Guidance on the Spark website is done >> already as I mentioned - we updated the DStream doc page to mention that >> DStream is a "legacy" project and users should move to SS. I don't feel >> this is sufficient to refrain users from using it, hence initiating this >> proposal. >> >>> >> >>> Sorry to make confusion. I just wanted to make sure the goal of the >> proposal is not "removing" the API. The discussion on the removal of API >> doesn't tend to go well, so I wanted to make sure I don't mean that. >> >>> >> >>> On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun < >> dongjoon.h...@gmail.com> wrote: >> >>>> >> >>>> +1 for the proposal (guiding only without any code change). >> >>>> >> >>>> Thanks, >> >>>> Dongjoon. >> >>>> >> >>>> On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu <zsxw...@gmail.com> >> wrote: >> >>>>> >> >>>>> +1 >> >>>>> >> >>>>> >> >>>>> On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>>>>> >> >>>>>> +1 >> >>>>>> >> >>>>>> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon <gurwls...@gmail.com> >> wrote: >> >>>>>>> >> >>>>>>> +1 >> >>>>>>> >> >>>>>>> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>>>>>>> >> >>>>>>>> bump for more visibility. >> >>>>>>>> >> >>>>>>>> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>>>>>>>> >> >>>>>>>>> Hi dev, >> >>>>>>>>> >> >>>>>>>>> I'd like to propose the deprecation of DStream in Spark 3.4, in >> favor of promoting Structured Streaming. >> >>>>>>>>> (Sorry for the late proposal, if we don't make the change in >> 3.4, we will have to wait for another 6 months.) >> >>>>>>>>> >> >>>>>>>>> We have been focusing on Structured Streaming for years (across >> multiple major and minor versions), and during the time we haven't made any >> improvements for DStream. Furthermore, recently we updated the DStream doc >> to explicitly say DStream is a legacy project. >> >>>>>>>>> >> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >> >>>>>>>>> >> >>>>>>>>> The baseline of deprecation is that we don't see a particular >> use case which only DStream solves. This is a different story with GraphX >> and MLLIB, as we don't have replacements for that. >> >>>>>>>>> >> >>>>>>>>> The proposal does not mean we will remove the API soon, as the >> Spark project has been making deprecation against public API. I don't >> intend to propose the target version for removal. The goal is to guide >> users to refrain from constructing a new workload with DStream. We might >> want to go with this in future, but it would require a new discussion >> thread at that time. >> >>>>>>>>> >> >>>>>>>>> What do you think? >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> Jungtaek Lim (HeartSaVioR) >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
