Yes, you are right. a dynamic topicfilter is more appropriate where I can filter topics at runtime via some kind of interface e.g. JMX
On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Tao, > > Based on your description I think the combination of whitelist / blacklist > will not achieve your goal, since it is still static. > > Guozhang > > On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xiaotao...@gmail.com> wrote: > > > Thank you Guozhang for your advice. A dynamic topic filter is what I need > > so that I can stop a topic consumption when I need to at runtime. > > > > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > 1. Dynamic: yeah that is sth. we could think of, this could be useful > > > operationally. > > > 2. Regex: I think in terms of expressiveness it should be sufficient > for > > > almost all subset of topics. In practice usually the rule of thumb is > > that > > > you will create your topics that belongs to the same "group" with some > > > prefix / suffix so that regex expression would not be crazily long. > > > > > > Guozhang > > > > > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xiaotao...@gmail.com> > wrote: > > > > > > > something like dynamic filtering that can be updated at runtime or > deny > > > all > > > > but allow a certain set of topics that cannot be specified easily by > > > regex > > > > > > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > > > > > > > Hmm, what kind of customized filtering do you have in mind? I > thought > > > > with > > > > > "--whitelist" you could already specify regex to do filtering. > > > > > > > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xiaotao...@gmail.com> > > > wrote: > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > I was meant to be topicfilter not topic-count. sorry for the > > > confusion. > > > > > > What I want to achieve is to pass my own customized topicfilter > to > > MM > > > > so > > > > > > that I can filter out topics what ever I like. I know MM doesn't > > > > support > > > > > > this now. I am just thinking if this is a good feature to add in > > > > > > > > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang < > wangg...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > Hi Tao, > > > > > > > > > > > > > > Sorry I was mistaken before, yes in MM you can only directly > > > specify > > > > > > > "--whitelist", "--blacklist", and the number of streams you > want > > > to > > > > > > create > > > > > > > via "--num.streams", but cannot set specific topic-count. This > is > > > > > because > > > > > > > MM is mainly used for cross DC replication, and hence usually > > will > > > > pipe > > > > > > all > > > > > > > topics or a majority of the topics from the source cluster to > the > > > > > > > destination, hence usually you do not care about "some topics > > > should > > > > > get > > > > > > X > > > > > > > streams, while some other topics should get Y streams". > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao < > xiaotao...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > The topic list is not specified in consumer.properties and I > > > don't > > > > > > think > > > > > > > > there is any property in consumer config that allows us to > > > specify > > > > > what > > > > > > > > topics we want to consume. Can you point me to the property > if > > > > there > > > > > is > > > > > > > > any? > > > > > > > > > > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang < > > > > wangg...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Tao, > > > > > > > > > > > > > > > > > > In MM people can pass in consumer configs, in which people > > can > > > > > > specify > > > > > > > > > consumption topics, either in regular topic list format or > > > > > whitelist > > > > > > / > > > > > > > > > blacklist. So I think it already does what you need? > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao < > > > xiaotao...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thank you guys for answering. I think it will be good > that > > we > > > > can > > > > > > > pass > > > > > > > > > in a > > > > > > > > > > customised topicCount ( I think this is the interface > > > whitelist > > > > > and > > > > > > > > > > backlist implement if I am not mistaken) to MM to achieve > > > > similar > > > > > > > thing > > > > > > > > > > > > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang < > > > > wangg...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Tao, > > > > > > > > > > > > > > > > > > > > > > Unfortunately MM does not support whitelist / blacklist > > at > > > > the > > > > > > same > > > > > > > > > time, > > > > > > > > > > > and you have to choose either one upon initialization. > As > > > for > > > > > > your > > > > > > > > > case, > > > > > > > > > > I > > > > > > > > > > > think it can be captured by some reg-ex to exclude > > nothing > > > > else > > > > > > but > > > > > > > > > "10", > > > > > > > > > > > but I do not know the exact expression. > > > > > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < > > > > > xiaotao...@gmail.com > > > > > > > > > > > <javascript:;>> wrote: > > > > > > > > > > > > > > > > > > > > > > > I actually mean if we can achieve this in mirror > maker. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao < > > > > > > xiaotao...@gmail.com > > > > > > > > > > > <javascript:;>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > I have an user case where I need to consume a list > > > topics > > > > > > with > > > > > > > > name > > > > > > > > > > > that > > > > > > > > > > > > > matches pattern topic.* except for one that is > > > topic.10. > > > > Is > > > > > > > > there a > > > > > > > > > > way > > > > > > > > > > > > > that I can combine the use of whitelist and > blacklist > > > so > > > > > > that I > > > > > > > > can > > > > > > > > > > > > achieve > > > > > > > > > > > > > something like accept all topics with regex topic.* > > but > > > > > > exclude > > > > > > > > > > > topic.10? > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Regards, > > > > > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Regards, > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Tao > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > > > -- > > Regards, > > Tao > > > > > > -- > -- Guozhang > -- Regards, Tao
