Maybe I need to clarify - my proposal is "explicitly" deprecating it, which incurs code change for sure. Guidance on the Spark website is done already as I mentioned - we updated the DStream doc page to mention that DStream is a "legacy" project and users should move to SS. I don't feel this is sufficient to refrain users from using it, hence initiating this proposal.
Sorry to make confusion. I just wanted to make sure the goal of the proposal is not "removing" the API. The discussion on the removal of API doesn't tend to go well, so I wanted to make sure I don't mean that. On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > +1 for the proposal (guiding only without any code change). > > Thanks, > Dongjoon. > > On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu <zsxw...@gmail.com> wrote: > >> +1 >> >> >> On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>> +1 >>> >>> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon <gurwls...@gmail.com> >>> wrote: >>> >>>> +1 >>>> >>>> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim < >>>> kabhwan.opensou...@gmail.com> wrote: >>>> >>>>> bump for more visibility. >>>>> >>>>> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >>>>> kabhwan.opensou...@gmail.com> wrote: >>>>> >>>>>> Hi dev, >>>>>> >>>>>> I'd like to propose the deprecation of DStream in Spark 3.4, in favor >>>>>> of promoting Structured Streaming. >>>>>> (Sorry for the late proposal, if we don't make the change in 3.4, we >>>>>> will have to wait for another 6 months.) >>>>>> >>>>>> We have been focusing on Structured Streaming for years (across >>>>>> multiple major and minor versions), and during the time we haven't made >>>>>> any >>>>>> improvements for DStream. Furthermore, recently we updated the DStream >>>>>> doc >>>>>> to explicitly say DStream is a legacy project. >>>>>> >>>>>> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >>>>>> >>>>>> The baseline of deprecation is that we don't see a particular use >>>>>> case which only DStream solves. This is a different story with GraphX and >>>>>> MLLIB, as we don't have replacements for that. >>>>>> >>>>>> The proposal does not mean we will remove the API soon, as the Spark >>>>>> project has been making deprecation against public API. I don't intend to >>>>>> propose the target version for removal. The goal is to guide users to >>>>>> refrain from constructing a new workload with DStream. We might want to >>>>>> go >>>>>> with this in future, but it would require a new discussion thread at that >>>>>> time. >>>>>> >>>>>> What do you think? >>>>>> >>>>>> Thanks, >>>>>> Jungtaek Lim (HeartSaVioR) >>>>>> >>>>>
