+1 On Thu, Jan 12, 2023 at 10:39 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > > Yes, exactly. I'm sorry to bring confusion - should have clarified action > items on the proposal. > > On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: >> >> Then, could you elaborate `the proposed code change` specifically? >> Maybe, usual deprecation warning logs and annotation on the API? >> >> >> On Thu, Jan 12, 2023 at 10:05 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> >> wrote: >>> >>> Maybe I need to clarify - my proposal is "explicitly" deprecating it, which >>> incurs code change for sure. Guidance on the Spark website is done already >>> as I mentioned - we updated the DStream doc page to mention that DStream is >>> a "legacy" project and users should move to SS. I don't feel this is >>> sufficient to refrain users from using it, hence initiating this proposal. >>> >>> Sorry to make confusion. I just wanted to make sure the goal of the >>> proposal is not "removing" the API. The discussion on the removal of API >>> doesn't tend to go well, so I wanted to make sure I don't mean that. >>> >>> On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun <dongjoon.h...@gmail.com> >>> wrote: >>>> >>>> +1 for the proposal (guiding only without any code change). >>>> >>>> Thanks, >>>> Dongjoon. >>>> >>>> On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu <zsxw...@gmail.com> wrote: >>>>> >>>>> +1 >>>>> >>>>> >>>>> On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das >>>>> <tathagata.das1...@gmail.com> wrote: >>>>>> >>>>>> +1 >>>>>> >>>>>> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: >>>>>>> >>>>>>> +1 >>>>>>> >>>>>>> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim >>>>>>> <kabhwan.opensou...@gmail.com> wrote: >>>>>>>> >>>>>>>> bump for more visibility. >>>>>>>> >>>>>>>> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim >>>>>>>> <kabhwan.opensou...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hi dev, >>>>>>>>> >>>>>>>>> I'd like to propose the deprecation of DStream in Spark 3.4, in favor >>>>>>>>> of promoting Structured Streaming. >>>>>>>>> (Sorry for the late proposal, if we don't make the change in 3.4, we >>>>>>>>> will have to wait for another 6 months.) >>>>>>>>> >>>>>>>>> We have been focusing on Structured Streaming for years (across >>>>>>>>> multiple major and minor versions), and during the time we haven't >>>>>>>>> made any improvements for DStream. Furthermore, recently we updated >>>>>>>>> the DStream doc to explicitly say DStream is a legacy project. >>>>>>>>> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >>>>>>>>> >>>>>>>>> The baseline of deprecation is that we don't see a particular use >>>>>>>>> case which only DStream solves. This is a different story with GraphX >>>>>>>>> and MLLIB, as we don't have replacements for that. >>>>>>>>> >>>>>>>>> The proposal does not mean we will remove the API soon, as the Spark >>>>>>>>> project has been making deprecation against public API. I don't >>>>>>>>> intend to propose the target version for removal. The goal is to >>>>>>>>> guide users to refrain from constructing a new workload with DStream. >>>>>>>>> We might want to go with this in future, but it would require a new >>>>>>>>> discussion thread at that time. >>>>>>>>> >>>>>>>>> What do you think? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Jungtaek Lim (HeartSaVioR)
--------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
