Hi Zahari, Created the patch here : https://github.com/apache/kafka/pull/5844
Thanks, Mayuresh On Thu, Oct 25, 2018 at 4:42 PM Mayuresh Gharat <gharatmayures...@gmail.com> wrote: > Hi Zahari, > > Oops. We had planned to put this patch upstream but somehow slipped my > mind. We were recently going over hotfixes that we have and this seemed > something that had been due for sometime now. Glad to know that someone > else apart from us might also benefit from this :) > > Thanks, > > Mayuresh > > On Thu, Oct 25, 2018 at 12:25 PM Zahari Dichev <zaharidic...@gmail.com> > wrote: > >> Hi there Mayuresh, >> >> Great to heat that this is actually working well in production for some >> time now. I have changed the details of the KIP to reflect the fact that >> as >> already discussed - we do not really need any kind of configuration as >> this >> data should not be thrown away at all. Submitting a PR sounds great, >> although I feel a bit jealous you (LinkedIn) beat me to my first kafka >> commit ;) Not sure how things stand with the voting process ? >> >> Zahari >> >> >> >> On Thu, Oct 25, 2018 at 7:39 PM Mayuresh Gharat < >> gharatmayures...@gmail.com> >> wrote: >> >> > Hi Colin/Zahari, >> > >> > I have created a ticket for the similar/same feature : >> > https://issues.apache.org/jira/browse/KAFKA-7548 >> > We (Linkedin) had a use case in Samza at Linkedin when they moved from >> the >> > SimpleConsumer to KafkaConsumer and they wanted to do this pause and >> resume >> > pattern. >> > They realized there was performance degradation when they started using >> > KafkaConsumer.assign() and pausing and unPausing partitions. We realized >> > that not throwing away the prefetched data for paused partitions might >> > improve the performance. We wrote a benchmark (I can share it if >> needed) to >> > prove this. I have attached the findings in the ticket. >> > We have been running the hotfix internally for quite a while now. When >> > samza ran this fix in production, they realized 30% improvement in there >> > app performance. >> > I have the patch ready on our internal branch and would like to submit >> a PR >> > for this on the above ticket asap. >> > I am not sure, if we need a separate config for this as we haven't seen >> a >> > lot of memory overhead due to this in our systems. We have had this >> running >> > in production for a considerable amount of time without any issues. >> > It would be great if you guys can review the PR once its up and see if >> that >> > satisfies your requirement. If it doesn't then we can think more on the >> > config driven approach. >> > Thoughts?? >> > >> > Thanks, >> > >> > Mayuresh >> > >> > >> > On Thu, Oct 25, 2018 at 8:21 AM Colin McCabe <cmcc...@apache.org> >> wrote: >> > >> > > Hi Zahari, >> > > >> > > One question we didn't figure out earlier was who would actually want >> > this >> > > cached data to be thrown away. If there's nobody who actually wants >> > this, >> > > then perhaps we can simplify the proposal by just unconditionally >> > retaining >> > > the cache until the partition is resumed, or we unsubscribe from the >> > > partition. This would avoid adding a new configuration. >> > > >> > > best, >> > > Colin >> > > >> > > >> > > On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote: >> > > > Hi there, although it has been discussed briefly already in this >> thread >> > > > < >> > > >> > >> https://lists.apache.org/thread.html/fbb7e9ccc41084fc2ff8612e6edf307fb400f806126b644d383b4a64@%3Cdev.kafka.apache.org%3E >> > > >, >> > > > I decided to follow the process and initiate a DISCUSS thread. >> Comments >> > > > and >> > > > suggestions are more than welcome. >> > > > >> > > > >> > > > Zahari Dichev >> > > >> > >> > >> > -- >> > -Regards, >> > Mayuresh R. Gharat >> > (862) 250-7125 >> > >> > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 > -- -Regards, Mayuresh R. Gharat (862) 250-7125
