Re?? [DISCUSS] KIP-68 Add a consumed log retention before log retention

kafka.apache.org"<dev@kafka.apache.org>; ????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention David, I think it would be better implementing such synchronization (i.e. making sure all consumers has done fetching to that point, and no one will ever want to go b

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

; > > > > > > > -- 原始邮件 -- > 发件人: "Becket Qin";<becket....@gmail.com>; > 发送时间: 2016年11月6日(星期天) 晚上11:39 > 收件人: "dev"<dev@kafka.apache.org>; > > 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before

Re?? [DISCUSS] KIP-68 Add a consumed log retention before log retention

? -- > ??: "Becket Qin";<becket....@gmail.com>; > ????: 2016??11??1??(??) 3:57 > ??: "dev"<dev@kafka.apache.org>; > > : Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention > > &g

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

....@gmail.com>; > 发送时间: 2016年11月1日(星期二) 凌晨3:57 > 收件人: "dev"<dev@kafka.apache.org>; > > 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention > > > > Hi David, > > I think the trim() API is generally useful for the consume based re

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

--- > ??: "Mayuresh Gharat";<gharatmayures...@gmail.com>; > : 2016??10??29??(??) 1:43 > ??: "dev"<dev@kafka.apache.org>; > > : Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention > > > > I do

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

min > > > > > > > > > > offset based on their committed offsets, by either > talking > > to > > > > the > > > > > > > > > consumer > > > >

Re?? [DISCUSS] KIP-68 Add a consumed log retention before log retention

??: 2016??10??29??(??) 1:43 ??: "dev"<dev@kafka.apache.org>; ????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention I do agree with Guozhang on having applications request an external service(admin) that talks to kafka, for trimming, in which case

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

> can > > > be > > > > > > honor > > > > > > > > by > > > > > > > > > the same implementation of KIP-47 on broker side). > > > > > > >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

t; > > > > from > > > > > > > > client admin requests, and 2) allowing more generalized > > > > client-driven > > > > > > log > > > > > > > > retention policies with KIP-47 (i.e. broker is brainless and >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

> > > > > that > > > > > > > they send to the brokers). > > > > > > > > > > > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

t; > > > > > > > > > > One scenario is stream processing pipeline. In a stream > > processing > > > > DAG, > > > > > > > there will be a bunch of intermediate result, we only care > about > > > the > > > > > > &

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

; but > > > > not > > > > > > other groups. That means if a down stream job did not commit > offset > > > for > > > > > > some reason, we want to wait for that job. Without the predefined > > > > > > interested group,

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

ks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > On Fri, Oct 21, 2016 at 7:40 AM, 东方甲乙 <254479...@qq.com> wrote: > > > > > > > > > > > Hi Mayuresh, > > > > > >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

gt; > > the > > > > > consumed group which are consuming this topic, and query the commit > > > > offset > > > > > of this consumed group for the topic > > > > > using the OffsetFetch API. And the min commit offset is the minimal > > >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

mit > > > > offset between these commit offsets. > > > > > > > > > > > > 2. If the console consumer reading and commit, its commit offset > will > > be > > > > used to calculate the min commit offset for this topic. > > > >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

will not delete the log immediately, the log will stay some time > ( > > > retention.commitoffset.ms), and after that we only delete > > > the log segments whose offsets are less than the min commit offset. So > > > the user can rewind its offset in the log.retention.ms. &g

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

segments whose offsets are less than the min commit offset. So > > the user can rewind its offset in the log.retention.ms. > > > > > > Thanks, > > David > > > > > > > > > > -- 原始邮件 -- > > 发件

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

gt; > > > > -- 原始邮件 -- > 发件人: "Mayuresh Gharat";<gharatmayures...@gmail.com>; > 发送时间: 2016年10月19日(星期三) 上午10:25 > 收件人: "dev"<dev@kafka.apache.org>; > > 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention befor

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

harat";<gharatmayures...@gmail.com>; : 2016??10??19??(??) 10:25 ??: "dev"<dev@kafka.apache.org>; ????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention Hi David, Thanks for the KIP. I had some questions/suggestions : It w

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

-- > ??: "Dong Lin";<lindon...@gmail.com>; > : 2016??10??14??(??) 5:01 > ??: "dev"<dev@kafka.apache.org>; > > : Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention > > >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

is in inactive, but it's commit offset is > > also in the __commit_offsets topic and > > stay in the offset cache, we can delete it via this API. > > > > > > Thanks, > > David > > > > > > -- 原始邮件 -- > > 发件人: "Dong Lin&

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

I would argue that the 2nd scheme is better. Say the > > consumed log retention is enabled. The 1st scheme basically interprets > > forced log retention as the upper bound of the time the data can stay in > > Kafka. The 2nd scheme interprets forced log retention as the lower bound > of > &

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

-- -- ??: "Dong Lin";<lindon...@gmail.com>; : 2016??10??14??(??) 5:01 ??: "dev"<dev@kafka.apache.org>; ????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention Hi David, As explained in the motivation section of the

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

forced log retention to 1 day and enable consumed log retention. What do you think? > > Thanks, > David > > > > > -- -- > ??: "Dong Lin";<lindon...@gmail.com>; > : 2016??10??10??(??) 4:05 > ??: "dev"<dev@kafka.apa

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

?: "dev"<dev@kafka.apache.org>; ????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention Hi David This is a very timely KIP given the number of use cases in the streams processing pipeline than need consumed log retention management. Some questions

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

can stay in Kafka, which is more consistent with the > purpose of having this forced log retention (to save disk space). And if we > adopt the 2nd solution, the use-case you suggested can be easily addressed > by setting forced log retention to 1 day and enable consumed log retention. > What

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

David > > > > > -- 原始邮件 -- > 发件人: "Dong Lin";<lindon...@gmail.com>; > 发送时间: 2016年10月10日(星期一) 下午4:05 > 收件人: "dev"<dev@kafka.apache.org>; > > 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log rete

RE?? [DISCUSS] KIP-68 Add a consumed log retention before log retention

t;;<lindon...@gmail.com>; : 2016??10??10??(??) 4:05 ??: "dev"<dev@kafka.apache.org>; ????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention Hey David, Thanks for reply. Please see comment inline. On Mon, Oct 10, 2016 at 12:40 AM, Pe

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hi David This is a very timely KIP given the number of use cases in the streams processing pipeline than need consumed log retention management. Some questions that Becket and Dong asked just wanted to make sure are described in the KIP. 1. How is the configuration setup per topic to know what

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hey David, Thanks for reply. Please see comment inline. On Mon, Oct 10, 2016 at 12:40 AM, Pengwei (L) wrote: > Hi Dong >Thanks for the questions: > > 1. Now we don't distinguish inactive or active groups. Because in some > case maybe inactive group will become

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hi Dong Thanks for the questions: 1. Now we don't distinguish inactive or active groups. Because in some case maybe inactive group will become active again, and using the previous commit offset. So we will not delete the log segment in the consumer retention if there are some groups

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

this > KIP, though we are all modifying the log retention. > > > Thanks, > David. > > > > > > > > > -- 原始邮件 -- > 发件人: "Becket Qin";<becket....@gmail.com>; > 发送时间: 2016年10月9日(星期天) 中午1:00 > 收件人: "dev&qu

RE?? [DISCUSS] KIP-68 Add a consumed log retention before log retention

. -- -- ??: "Becket Qin";<becket@gmail.com>; : 2016??10??9??(??) 1:00 ??: "dev"<dev@kafka.apache.org>; ????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention Hi David, Thanks for the explanation. Could you u

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hey David, Thanks for the KIP. Can you help with the following two questions: 1) If someone start a consumer (e.g. kafka-console-consumer) to consume a topic for debug/validation purpose, a randome consumer group may be created and offset may be committed for this consumer group. If no offset

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hi David, Thanks for the explanation. Could you update the KIP-68 wiki to include the changes that need to be made? I have a few more comments below: 1. We already have an internal topic __consumer_offsets to store all the committed offsets. So the brokers can probably just consume from that to

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Hi Becket, Thanks for the feedback: 1. We use the simple consumer api to query the commit offset, so we don't need to specify the consumer group. 2. Every broker using the simple consumer api(OffsetFetchKey) to query the commit offset in the log retention process. The client can commit

[DISCUSS] KIP-68 Add a consumed log retention before log retention

Hi All, I have made a KIP to enhance the log retention, details as follows: https://cwiki.apache.org/confluence/display/KAFKA/KIP-68+Add+a+consumed+log+retention+before+log+retention Now start a discuss thread for this KIP , looking forward to the feedback. Thanks, David