Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

2024-01-10 Thread Bartosz Konieczny
n-binding). Thanks! >>>>>> >>>>>> On Tue, Jan 9, 2024 at 9:37 AM Jungtaek Lim < >>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>&g

Re: [VOTE] SPIP: State Data Source - Reader

2023-10-24 Thread Bartosz Konieczny
et >> >>>> SPIP doc >> >>>> Discussion thread >> >>>> >> >>>> Please vote on the SPIP for the next 72 hours: >> >>>> >> >>>> [ ] +1: Accept the proposal as an official SPIP >> >>>> [ ] +0 >> >>>> [ ] -1: I don’t think this is a good idea because … >> >>>> >> >>>> Thanks! >> >>>> Jungtaek Lim (HeartSaVioR) >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> -- Bartosz Konieczny freelance data engineer https://www.waitingforcode.com https://github.com/bartosz25/ https://twitter.com/waitingforcode

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-16 Thread Bartosz Konieczny
://issues.apache.org/jira/browse/SPARK-45511 >>> >>> Looking forward to your feedback! >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>> ps. The scope of the project is narrowed to the reader in this SPIP, >>> since the writer requires us to consider more cases. We are planning on it. >>> >> -- Bartosz Konieczny freelance data engineer https://www.waitingforcode.com https://github.com/bartosz25/ https://twitter.com/waitingforcode

Re: Watermark on late data only

2023-10-10 Thread Bartosz Konieczny
consistent demand on it, we could probably look into it and > maybe introduce a new SQL function (which works only on streaming - that's > probably a major blocker on introduction) on it. > > On Mon, Oct 9, 2023 at 11:03 AM Bartosz Konieczny > wrote: > >> Hi, >> >&

Watermark on late data only

2023-10-08 Thread Bartosz Konieczny
Hi, I've been analyzing the watermark propagation added in the 3.5.0 recently and had to return to the basics of watermarks. One question is still unanswered in my head. Why are the watermarks reserved to stateful queries? Can't they apply to the filtering late date out only? The reason is only