Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

Thanks Jungtaek for creating the Vote thread. +1 (non-binding) from my side too. Thanks, Anish On Tue, Jan 9, 2024 at 6:09 AM Jungtaek Lim wrote: > Starting with my +1 (non-binding). Thanks! > > On Tue, Jan 9, 2024 at 9:37 AM Jungtaek Lim > wrote: > >> Hi all, >> >> I'd like to start the

Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

Starting with my +1 (non-binding). Thanks! On Tue, Jan 9, 2024 at 9:37 AM Jungtaek Lim wrote: > Hi all, > > I'd like to start the vote for SPIP: Structured Streaming - Arbitrary > State API v2. > > References: > >- JIRA ticket >- SPIP

[VOTE] SPIP: Structured Streaming - Arbitrary State API v2

Hi all, I'd like to start the vote for SPIP: Structured Streaming - Arbitrary State API v2. References: - JIRA ticket - SPIP doc -

Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

Thanks everyone for the feedback! Given that we get positive feedback without major concerns, I will initiate the vote thread soon. Please make a vote in that thread as well. Thanks again! On Tue, Jan 9, 2024 at 7:44 AM Bhuwan Sahni wrote: > +1 on the newer APIs. I believe these APIs provide

Re: Spark Structured Streaming and Flask REST API for Real-Time Data Ingestion and Analytics.

Please also note that Flask, by default, is a single-threaded web framework. While it is suitable for development and small-scale applications, it may not handle concurrent requests efficiently in a production environment. In production, one can utilise Gunicorn (Green Unicorn) which is a WSGI (

Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

+1 on the newer APIs. I believe these APIs provide a much powerful mechanism for the user to perform arbitrary state management in Structured Streaming queries. Thanks Bhuwan Sahni On Mon, Jan 8, 2024 at 10:07 AM L. C. Hsieh wrote: > +1 > > I left some comments in the SPIP doc and got replies

Spark Structured Streaming and Flask REST API for Real-Time Data Ingestion and Analytics.

Thought it might be useful to share my idea with fellow forum members. During the breaks, I worked on the *seamless integration of Spark Structured Streaming with Flask REST API for real-time data ingestion and analytics*. The use case revolves around a scenario where data is generated through

Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

+1 I left some comments in the SPIP doc and got replies quickly. The new API looks good and more comprehensive. I think it will help Spark Structured Streaming to be more useful in more complicated streaming use cases. On Fri, Jan 5, 2024 at 8:15 PM Burak Yavuz wrote: > > I'm also a +1 on the

Re: Regression? - UIUtils::formatBatchTime - [SPARK-46611][CORE] Remove ThreadLocal by replace SimpleDateFormat with DateTimeFormatter

Agreed, that looks wrong. From the code, it seems that "timezone" is only used for testing, though apparently no test caught this. I'll submit a PR to patch it in any event: https://github.com/apache/spark/pull/44619 On Mon, Jan 8, 2024 at 1:33 AM Janda Martin wrote: > I think that >