I am currently pondering and sharing my thoughts openly. Given our reliance on gathered statistics, it prompts the question of whether we could integrate specific machine learning components into Spark Structured Streaming. Consider a scenario where we aim to adjust configuration values on the fly based on collected statistics. For instance, Spark GUI could gather some statistics via Spark listeners. Is this a viable approach? This aligns with use cases involving Spark on Kubernetes autopilot and Spark Structured Streaming. While this might slightly veer from our usual discussions, I admit my lack of expertise in machine learning. However, there are experts among us who could lend their knowledge to explore this potential avenue.
Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
