Re: Spark Structured Streaming and Flask REST API for Real-Time Data Ingestion and Analytics.

Hi Ashok, Thanks for pointing out the databricks article Scalable Spark Structured Streaming for REST API Destinations | Databricks Blog I browsed it and it is basically similar to many of us involved

Re: Spark Structured Streaming and Flask REST API for Real-Time Data Ingestion and Analytics.

Hey Mich, Thanks for this introduction on your forthcoming proposal "Spark Structured Streaming and Flask REST API for Real-Time Data Ingestion and Analytics". I recently came across an article by Databricks with title Scalable Spark Structured Streaming for REST API Destinations. Their use

Re: Spark Structured Streaming and Flask REST API for Real-Time Data Ingestion and Analytics.

Please also note that Flask, by default, is a single-threaded web framework. While it is suitable for development and small-scale applications, it may not handle concurrent requests efficiently in a production environment. In production, one can utilise Gunicorn (Green Unicorn) which is a WSGI (

Spark Structured Streaming and Flask REST API for Real-Time Data Ingestion and Analytics.

Thought it might be useful to share my idea with fellow forum members. During the breaks, I worked on the *seamless integration of Spark Structured Streaming with Flask REST API for real-time data ingestion and analytics*. The use case revolves around a scenario where data is generated through