Hi Bill, John, I've made some changes to the KIP after some thinking. I've come up with a reasonable solution, I believe, to relieve the problem that is associated with low traffic suppression buffers. Would be great if I can get your input on this. :)
Cheers, Richard On Fri, Oct 18, 2019 at 7:16 PM Richard Yu <yohan.richard...@gmail.com> wrote: > Hi Bill, > > Thanks for the input! > TBH, I am think that suppression buffers are not used *in response *to > low traffic conditions. > Rather, we are trying to fix the situation when low traffic conditions > occur in a suppression buffer (for example, previously, the same > suppression buffer had a decent volume of records entering it, thus > advancing the stream time). > In summary, when these conditions do hit, we want to advance the stream > time somehow to resolve this issue. > Reflecting on this though, I am not completely certain if this really > stops us from implementing per key stream time tracking because the problem > wouldn't be made *that *much worse. > > Cheers, > Richard > > On Fri, Oct 18, 2019 at 11:46 AM Bill Bejeck <bbej...@gmail.com> wrote: > >> Hi Richard, >> >> Thanks for the KIP proposal. I understand the situation you are >> describing. >> But in my mind, if there is a low traffic condition and you need to keep >> records going downstream at regular intervals, I'm wondering if using >> suppression is the correct approach. >> IMHO it seems it would be better to use the PAPI or a Transform on the DSL >> with a scheduled punctuation call. >> >> Just my 2 cents. >> >> Thanks, >> Bill >> >> On Thu, Oct 17, 2019 at 7:42 PM Richard Yu <yohan.richard...@gmail.com> >> wrote: >> >> > Hi all, >> > >> > I wish to discuss this KIP which would help us in resolving some issues >> we >> > have with suppression buffers. >> > Below is the link: >> > >> > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers >> > >> > @John Roesler if you have time, would be great if we could get your >> input. >> > >> > Cheers, >> > Richard >> > >> >
