Hi, Your intended work on improving the Structured Streaming documentation is great! Clear and well-organized instructions are important for everyone using Spark, beginners and experts alike. Having said that, Spark Structured Streaming much like other specialist topics with Spark say (k8s) or otherwise cannot be mastered by documentation alone. These topics require a considerable amount of practice and trench warfare so to speak to master them. Suffice to say that I agree with the proposals of making examples. However, it is an area that many try to master but fail( judging by typical issues brought up in the user group and otherwise). Perhaps using a section such as the proposed "Knowledge Sharing Hub'', may become more relevant. Moreover, the examples have to reflect real life scenarios and conversly will be of limited use otherwise.
HTH Mich Talebzadeh, Technologist | Data | Generative AI | Financial Fraud London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh Disclaimer: The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner Von Braun)". Mich Talebzadeh, Technologist | Data | Generative AI | Financial Fraud London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Mon, 25 Mar 2024 at 21:19, Neil Ramaswamy <n...@ramaswamy.org> wrote: > Hi all, > > I recently started an effort to improve the Structured Streaming > documentation. I thought that the current documentation, while very > comprehensive, could be improved in terms of organization, clarity, and > presence of examples. > > You can view the repo here > <https://github.com/neilramaswamy/structured-streaming>, and you can see > a preview of the site here <https://structured-streaming.vercel.app/>. > It's almost at full parity with the programming guide, and it also has > additional content, like a guide on unit testing and an in-depth > explanation of watermarks. I think it's at a point where we can bring this > to completion if it's something that the community wants. > > I'd love to hear feedback from everyone: is this something that we would > want to move forward with? As it borrows certain parts from the programming > guide, it has an Apache License, so I'd be more than happy if it is adopted > by an official Spark repo. > > Best, > Neil >
