Thanks, Alexander for your reply. Our solution initiated from this inquiry on Stack Overflow: https://stackoverflow.com/questions/52179898/does-flink-support-keyed-watermarks-if-not-is-there-any-plan-of-implementing-i
The idea is to generate a watermark for each key (sub-stream), in order to avoid the fast progress of the global watermark which affects low-rate sources. Instead of using only one watermark (vanilla/global watermark), we changed the API to allow moving the keyBy() before the assignTimestampsAndWatermarks() so the stream will be partitioned then the TimestampsAndWatermarkOperator will handle the generation of each watermark for each key (source/sub-stream/partition). Let's discuss more if you want I have a presentation at a conference, we can meet or whatever is suitable. Also, I contacted David Anderson one year ago and he followed me step by step and helped me a lot. I attached some messages with David. Thanks & BR, [https://lh5.googleusercontent.com/MiACj615x33GxCbMNBzNppSALcuxUYChqrKMR1YZdR4zYSu1HFg3rbI7HUhmzrZhlHwHHF3QAhx2M4wlB_OQ1mmq1C6gQHVjhrfeGzmSvZdfszJeZHB8ll_WKwuxN6b6BRih4lkw5P-LuBtMKZweJD0]<http://www.nu.edu.eg/> [https://lh4.googleusercontent.com/YakyNaMyo42CikAlRHeAN0s-s5bwjaJsThDsWKKWN0Hc7BXS4Fg57ZGWnvCGm3tZxfifJWvPrj9fyDBvtg61xDQ0ZF6eGM8kCFU-qa3gPXHTSxWlYugBFDGH2hGw-gfsz546mPGLIQSI__MruVabQCs] Tawfik Yasser Tawfik Teaching Assistant | AI-ITCS-NU Office: UB1-B, Room 229 26th of July Corridor, Sheikh Zayed City, Giza, Egypt ________________________________ From: Alexander Fedulov <alexander.fedu...@gmail.com> Sent: 27 October 2023 20:09 To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: Proposal for Implementing Keyed Watermarks in Apache Flink [You don't often get email from alexander.fedu...@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Hi Tawfek, Thanks for sharing. I am trying to understand what exact real-life problem you are tackling with this approach. My understanding from skimming through the paper is that you are concerned about some outlier event producers from which the events can be delayed beyond what is expected in the overall system. Do I get it correctly that the keyed watermarking only targets scenarios of calculating keyed windows (which are also keyed by the same producer ids)? Best, Alexander Fedulov On Fri, 27 Oct 2023 at 19:07, Tawfek Yasser Tawfek <tyas...@nu.edu.eg> wrote: > Dear Apache Flink Development Team, > > I hope this email finds you well. I propose an exciting new feature for > Apache Flink that has the potential to significantly enhance its > capabilities in handling unbounded streams of events, particularly in the > context of event-time windowing. > > As you may be aware, Apache Flink has been at the forefront of Big Data > Stream processing engines, leveraging windowing techniques to manage > unbounded event streams effectively. The accuracy of the results obtained > from these streams relies heavily on the ability to gather all relevant > input within a window. At the core of this process are watermarks, which > serve as unique timestamps marking the progression of events in time. > > However, our analysis has revealed a critical issue with the current > watermark generation method in Apache Flink. This method, which operates at > the input stream level, exhibits a bias towards faster sub-streams, > resulting in the unfortunate consequence of dropped events from slower > sub-streams. Our investigations showed that Apache Flink's conventional > watermark generation approach led to an alarming data loss of approximately > 33% when 50% of the keys around the median experienced delays. This loss > further escalated to over 37% when 50% of random keys were delayed. > > In response to this issue, we have authored a research paper outlining a > novel strategy named "keyed watermarks" to address data loss and > substantially enhance data processing accuracy, achieving at least 99% > accuracy in most scenarios. > > Moreover, we have conducted comprehensive comparative studies to evaluate > the effectiveness of our strategy against the conventional watermark > generation method, specifically in terms of event-time tracking accuracy. > > We believe that implementing keyed watermarks in Apache Flink can greatly > enhance its performance and reliability, making it an even more valuable > tool for organizations dealing with complex, high-throughput data > processing tasks. > > We kindly request your consideration of this proposal. We would be eager > to discuss further details, provide the full research paper, or collaborate > closely to facilitate the integration of this feature into Apache Flink. > > Please check this preprint on Research Square: > https://www.researchsquare.com/article/rs-3395909/< > https://www.researchsquare.com/article/rs-3395909/v1> > > Thank you for your time and attention to this proposal. We look forward to > the opportunity to contribute to the continued success and evolution of > Apache Flink. > > Best Regards, > > Tawfik Yasser > Senior Teaching Assistant @ Nile University, Egypt > Email: tyas...@nu.edu.eg > LinkedIn: https://www.linkedin.com/in/tawfikyasser/ >
