A little more context about my needs: I have a requirement that I need to start/stop a topic at runtime based on a event sent to MM. at the moment I need to bounce the MM and find a way to exclude the topic from whitelist which is not an easy job with regex. If I can pass in a combination of blacklist and whitelist I can easily achieve this by having something like --whitelist topic.* --blacklist topic.1
On Thu, Mar 12, 2015 at 9:10 PM, tao xiao <xiaotao...@gmail.com> wrote: > something like dynamic filtering that can be updated at runtime or deny > all but allow a certain set of topics that cannot be specified easily by > regex > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wangg...@gmail.com> wrote: > >> Hmm, what kind of customized filtering do you have in mind? I thought with >> "--whitelist" you could already specify regex to do filtering. >> >> On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xiaotao...@gmail.com> wrote: >> >> > Hi Guozhang, >> > >> > I was meant to be topicfilter not topic-count. sorry for the confusion. >> > What I want to achieve is to pass my own customized topicfilter to MM so >> > that I can filter out topics what ever I like. I know MM doesn't support >> > this now. I am just thinking if this is a good feature to add in >> > >> > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wangg...@gmail.com> >> wrote: >> > >> > > Hi Tao, >> > > >> > > Sorry I was mistaken before, yes in MM you can only directly specify >> > > "--whitelist", "--blacklist", and the number of streams you want to >> > create >> > > via "--num.streams", but cannot set specific topic-count. This is >> because >> > > MM is mainly used for cross DC replication, and hence usually will >> pipe >> > all >> > > topics or a majority of the topics from the source cluster to the >> > > destination, hence usually you do not care about "some topics should >> get >> > X >> > > streams, while some other topics should get Y streams". >> > > >> > > Guozhang >> > > >> > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xiaotao...@gmail.com> >> wrote: >> > > >> > > > The topic list is not specified in consumer.properties and I don't >> > think >> > > > there is any property in consumer config that allows us to specify >> what >> > > > topics we want to consume. Can you point me to the property if >> there is >> > > > any? >> > > > >> > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <wangg...@gmail.com >> > >> > > > wrote: >> > > > >> > > > > Tao, >> > > > > >> > > > > In MM people can pass in consumer configs, in which people can >> > specify >> > > > > consumption topics, either in regular topic list format or >> whitelist >> > / >> > > > > blacklist. So I think it already does what you need? >> > > > > >> > > > > Guozhang >> > > > > >> > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xiaotao...@gmail.com> >> > > wrote: >> > > > > >> > > > > > Thank you guys for answering. I think it will be good that we >> can >> > > pass >> > > > > in a >> > > > > > customised topicCount ( I think this is the interface whitelist >> and >> > > > > > backlist implement if I am not mistaken) to MM to achieve >> similar >> > > thing >> > > > > > >> > > > > > On Wednesday, March 11, 2015, Guozhang Wang <wangg...@gmail.com >> > >> > > > wrote: >> > > > > > >> > > > > > > Hi Tao, >> > > > > > > >> > > > > > > Unfortunately MM does not support whitelist / blacklist at the >> > same >> > > > > time, >> > > > > > > and you have to choose either one upon initialization. As for >> > your >> > > > > case, >> > > > > > I >> > > > > > > think it can be captured by some reg-ex to exclude nothing >> else >> > but >> > > > > "10", >> > > > > > > but I do not know the exact expression. >> > > > > > > >> > > > > > > Guozhang >> > > > > > > >> > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao < >> xiaotao...@gmail.com >> > > > > > > <javascript:;>> wrote: >> > > > > > > >> > > > > > > > I actually mean if we can achieve this in mirror maker. >> > > > > > > > >> > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao < >> > xiaotao...@gmail.com >> > > > > > > <javascript:;>> wrote: >> > > > > > > > >> > > > > > > > > Hi, >> > > > > > > > > >> > > > > > > > > I have an user case where I need to consume a list topics >> > with >> > > > name >> > > > > > > that >> > > > > > > > > matches pattern topic.* except for one that is topic.10. >> Is >> > > > there a >> > > > > > way >> > > > > > > > > that I can combine the use of whitelist and blacklist so >> > that I >> > > > can >> > > > > > > > achieve >> > > > > > > > > something like accept all topics with regex topic.* but >> > exclude >> > > > > > > topic.10? >> > > > > > > > > >> > > > > > > > > -- >> > > > > > > > > Regards, >> > > > > > > > > Tao >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > Regards, >> > > > > > > > Tao >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > -- Guozhang >> > > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Regards, >> > > > > > Tao >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > -- Guozhang >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Regards, >> > > > Tao >> > > > >> > > >> > > >> > > >> > > -- >> > > -- Guozhang >> > > >> > >> > >> > >> > -- >> > Regards, >> > Tao >> > >> >> >> >> -- >> -- Guozhang >> > > > > -- > Regards, > Tao > -- Regards, Tao
