Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi everyone, I want to revive a solution to this issue. I created a new PR that accomodates Jason Gustafson's suggestion in the original PR to re-add paused completed fetches back to the completed fetches queue for less bookeeping. If someone could jump in and do a review it would be

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Just looked at it, Great work. Thanks a lot for the patch. This should certainly improve things ! Zahari On Wed, Oct 31, 2018 at 6:25 PM wrote: > Hi there, I will take a look first thing i get home. > > Zahari > > > On 31 Oct 2018, at 18:23, Mayuresh Gharat > wrote: > > > > Hi Colin,

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi there, I will take a look first thing i get home. Zahari > On 31 Oct 2018, at 18:23, Mayuresh Gharat wrote: > > Hi Colin, Zahari, > > Wanted to check if you can review the patch and let me know, if we need to > make any changes? > > Thanks, > > Mayuresh > > On Fri, Oct 26, 2018 at

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Colin, Zahari, Wanted to check if you can review the patch and let me know, if we need to make any changes? Thanks, Mayuresh On Fri, Oct 26, 2018 at 1:41 PM Zahari Dichev wrote: > Thanks for participating the discussion. Indeed, I learned quite a lot. > Will take a look at the patch as

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Thanks for participating the discussion. Indeed, I learned quite a lot. Will take a look at the patch as well and spend some time hunting for some other interesting issue to work on :) Cheers, Zahari On Fri, Oct 26, 2018 at 8:49 PM Colin McCabe wrote: > Hi Zahari, > > I think we can retire the

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Zahari, I think we can retire the KIP, since the KAFKA-7548 patch should solve the issue without any changes that require a KIP. This is actually the best thing we could do for our users, since things will "just work" more efficiently without a lot of configuration knobs. I think you did

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Zahari, Created the patch here : https://github.com/apache/kafka/pull/5844 Thanks, Mayuresh On Thu, Oct 25, 2018 at 4:42 PM Mayuresh Gharat wrote: > Hi Zahari, > > Oops. We had planned to put this patch upstream but somehow slipped my > mind. We were recently going over hotfixes that we

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Zahari, Oops. We had planned to put this patch upstream but somehow slipped my mind. We were recently going over hotfixes that we have and this seemed something that had been due for sometime now. Glad to know that someone else apart from us might also benefit from this :) Thanks, Mayuresh

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi there Mayuresh, Great to heat that this is actually working well in production for some time now. I have changed the details of the KIP to reflect the fact that as already discussed - we do not really need any kind of configuration as this data should not be thrown away at all. Submitting a

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Colin/Zahari, I have created a ticket for the similar/same feature : https://issues.apache.org/jira/browse/KAFKA-7548 We (Linkedin) had a use case in Samza at Linkedin when they moved from the SimpleConsumer to KafkaConsumer and they wanted to do this pause and resume pattern. They realized

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Zahari, One question we didn't figure out earlier was who would actually want this cached data to be thrown away. If there's nobody who actually wants this, then perhaps we can simplify the proposal by just unconditionally retaining the cache until the partition is resumed, or we

[DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi there, although it has been discussed briefly already in this thread , I decided to follow the process and initiate a DISCUSS thread. Comments and suggestions are more than