I am not sure how MM is going to be rewritten. Based on the current implementation in trunk offset is not committed unless it is produced to destination. With assumption that this logic remains MM will not acknowledge the offset back to source for filtered message. So I think it is safe to filter messages out while keeping committed offset unchanged for that particular topic. Please correct me if I am wrong
On Fri, Mar 13, 2015 at 1:12 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Note that with filtering in message handler, records from the source > cluster are still considered as "consumed" since the offsets will be > committed. If you change the filtering dynamically back to whilelist these > topics, you will lose the data that gets consumed during the period of the > blacklist. > > Guozhang > > On Thu, Mar 12, 2015 at 10:01 PM, tao xiao <xiaotao...@gmail.com> wrote: > > > Yes, that will work. message handle can filter out message sent from > > certain topics > > > > On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin <j...@linkedin.com.invalid > > > > wrote: > > > > > No sure if it is an option. But does filtering out topics with message > > > handler works for you? Are you going to resume consuming from a topic > > > after you stop consuming from it? > > > > > > Jiangjie (Becket) Qin > > > > > > On 3/12/15, 8:05 AM, "tao xiao" <xiaotao...@gmail.com> wrote: > > > > > > >Yes, you are right. a dynamic topicfilter is more appropriate where I > > can > > > >filter topics at runtime via some kind of interface e.g. JMX > > > > > > > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wangg...@gmail.com> > > > >wrote: > > > > > > > >> Tao, > > > >> > > > >> Based on your description I think the combination of whitelist / > > > >>blacklist > > > >> will not achieve your goal, since it is still static. > > > >> > > > >> Guozhang > > > >> > > > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xiaotao...@gmail.com> > > wrote: > > > >> > > > >> > Thank you Guozhang for your advice. A dynamic topic filter is > what I > > > >>need > > > >> > so that I can stop a topic consumption when I need to at runtime. > > > >> > > > > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang < > wangg...@gmail.com> > > > >> wrote: > > > >> > > > > >> > > 1. Dynamic: yeah that is sth. we could think of, this could be > > > >>useful > > > >> > > operationally. > > > >> > > 2. Regex: I think in terms of expressiveness it should be > > sufficient > > > >> for > > > >> > > almost all subset of topics. In practice usually the rule of > thumb > > > >>is > > > >> > that > > > >> > > you will create your topics that belongs to the same "group" > with > > > >>some > > > >> > > prefix / suffix so that regex expression would not be crazily > > long. > > > >> > > > > > >> > > Guozhang > > > >> > > > > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xiaotao...@gmail.com > > > > > >> wrote: > > > >> > > > > > >> > > > something like dynamic filtering that can be updated at > runtime > > or > > > >> deny > > > >> > > all > > > >> > > > but allow a certain set of topics that cannot be specified > > easily > > > >>by > > > >> > > regex > > > >> > > > > > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang > > > >><wangg...@gmail.com> > > > >> > > wrote: > > > >> > > > > > > >> > > > > Hmm, what kind of customized filtering do you have in mind? > I > > > >> thought > > > >> > > > with > > > >> > > > > "--whitelist" you could already specify regex to do > filtering. > > > >> > > > > > > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao < > > xiaotao...@gmail.com > > > > > > > >> > > wrote: > > > >> > > > > > > > >> > > > > > Hi Guozhang, > > > >> > > > > > > > > >> > > > > > I was meant to be topicfilter not topic-count. sorry for > the > > > >> > > confusion. > > > >> > > > > > What I want to achieve is to pass my own customized > > > >>topicfilter > > > >> to > > > >> > MM > > > >> > > > so > > > >> > > > > > that I can filter out topics what ever I like. I know MM > > > >>doesn't > > > >> > > > support > > > >> > > > > > this now. I am just thinking if this is a good feature to > > add > > > >>in > > > >> > > > > > > > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang < > > > >> wangg...@gmail.com > > > >> > > > > > >> > > > > wrote: > > > >> > > > > > > > > >> > > > > > > Hi Tao, > > > >> > > > > > > > > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only > > directly > > > >> > > specify > > > >> > > > > > > "--whitelist", "--blacklist", and the number of streams > > you > > > >> want > > > >> > > to > > > >> > > > > > create > > > >> > > > > > > via "--num.streams", but cannot set specific > topic-count. > > > >>This > > > >> is > > > >> > > > > because > > > >> > > > > > > MM is mainly used for cross DC replication, and hence > > > >>usually > > > >> > will > > > >> > > > pipe > > > >> > > > > > all > > > >> > > > > > > topics or a majority of the topics from the source > cluster > > > >>to > > > >> the > > > >> > > > > > > destination, hence usually you do not care about "some > > > >>topics > > > >> > > should > > > >> > > > > get > > > >> > > > > > X > > > >> > > > > > > streams, while some other topics should get Y streams". > > > >> > > > > > > > > > >> > > > > > > Guozhang > > > >> > > > > > > > > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao < > > > >> xiaotao...@gmail.com > > > >> > > > > > >> > > > > wrote: > > > >> > > > > > > > > > >> > > > > > > > The topic list is not specified in consumer.properties > > > >>and I > > > >> > > don't > > > >> > > > > > think > > > >> > > > > > > > there is any property in consumer config that allows > us > > to > > > >> > > specify > > > >> > > > > what > > > >> > > > > > > > topics we want to consume. Can you point me to the > > > >>property > > > >> if > > > >> > > > there > > > >> > > > > is > > > >> > > > > > > > any? > > > >> > > > > > > > > > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < > > > >> > > > wangg...@gmail.com> > > > >> > > > > > > > wrote: > > > >> > > > > > > > > > > >> > > > > > > > > Tao, > > > >> > > > > > > > > > > > >> > > > > > > > > In MM people can pass in consumer configs, in which > > > >>people > > > >> > can > > > >> > > > > > specify > > > >> > > > > > > > > consumption topics, either in regular topic list > > format > > > >>or > > > >> > > > > whitelist > > > >> > > > > > / > > > >> > > > > > > > > blacklist. So I think it already does what you need? > > > >> > > > > > > > > > > > >> > > > > > > > > Guozhang > > > >> > > > > > > > > > > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao < > > > >> > > xiaotao...@gmail.com > > > >> > > > > > > > >> > > > > > > wrote: > > > >> > > > > > > > > > > > >> > > > > > > > > > Thank you guys for answering. I think it will be > > good > > > >> that > > > >> > we > > > >> > > > can > > > >> > > > > > > pass > > > >> > > > > > > > > in a > > > >> > > > > > > > > > customised topicCount ( I think this is the > > interface > > > >> > > whitelist > > > >> > > > > and > > > >> > > > > > > > > > backlist implement if I am not mistaken) to MM to > > > >>achieve > > > >> > > > similar > > > >> > > > > > > thing > > > >> > > > > > > > > > > > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < > > > >> > > > wangg...@gmail.com> > > > >> > > > > > > > wrote: > > > >> > > > > > > > > > > > > >> > > > > > > > > > > Hi Tao, > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Unfortunately MM does not support whitelist / > > > >>blacklist > > > >> > at > > > >> > > > the > > > >> > > > > > same > > > >> > > > > > > > > time, > > > >> > > > > > > > > > > and you have to choose either one upon > > > >>initialization. > > > >> As > > > >> > > for > > > >> > > > > > your > > > >> > > > > > > > > case, > > > >> > > > > > > > > > I > > > >> > > > > > > > > > > think it can be captured by some reg-ex to > exclude > > > >> > nothing > > > >> > > > else > > > >> > > > > > but > > > >> > > > > > > > > "10", > > > >> > > > > > > > > > > but I do not know the exact expression. > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Guozhang > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < > > > >> > > > > xiaotao...@gmail.com > > > >> > > > > > > > > > > <javascript:;>> wrote: > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > I actually mean if we can achieve this in > mirror > > > >> maker. > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao < > > > >> > > > > > xiaotao...@gmail.com > > > >> > > > > > > > > > > <javascript:;>> wrote: > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Hi, > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > I have an user case where I need to consume > a > > > >>list > > > >> > > topics > > > >> > > > > > with > > > >> > > > > > > > name > > > >> > > > > > > > > > > that > > > >> > > > > > > > > > > > > matches pattern topic.* except for one that > is > > > >> > > topic.10. > > > >> > > > Is > > > >> > > > > > > > there a > > > >> > > > > > > > > > way > > > >> > > > > > > > > > > > > that I can combine the use of whitelist and > > > >> blacklist > > > >> > > so > > > >> > > > > > that I > > > >> > > > > > > > can > > > >> > > > > > > > > > > > achieve > > > >> > > > > > > > > > > > > something like accept all topics with regex > > > >>topic.* > > > >> > but > > > >> > > > > > exclude > > > >> > > > > > > > > > > topic.10? > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > -- > > > >> > > > > > > > > > > > > Regards, > > > >> > > > > > > > > > > > > Tao > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > -- > > > >> > > > > > > > > > > > Regards, > > > >> > > > > > > > > > > > Tao > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > -- > > > >> > > > > > > > > > > -- Guozhang > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > -- > > > >> > > > > > > > > > Regards, > > > >> > > > > > > > > > Tao > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > -- > > > >> > > > > > > > > -- Guozhang > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > -- > > > >> > > > > > > > Regards, > > > >> > > > > > > > Tao > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > -- > > > >> > > > > > > -- Guozhang > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > -- > > > >> > > > > > Regards, > > > >> > > > > > Tao > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > -- > > > >> > > > > -- Guozhang > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > -- > > > >> > > > Regards, > > > >> > > > Tao > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > -- > > > >> > > -- Guozhang > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > Regards, > > > >> > Tao > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> -- Guozhang > > > >> > > > > > > > > > > > > > > > >-- > > > >Regards, > > > >Tao > > > > > > > > > > > > -- > > Regards, > > Tao > > > > > > -- > -- Guozhang > -- Regards, Tao
