[ https://issues.apache.org/jira/browse/KAFKA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217513#comment-13217513 ]
Jun Rao commented on KAFKA-249: ------------------------------- Could we support subscribing to wildcard topics directly in consumerConnector? We could add a new api like the following: createMessageStream(topicRegex: String) : KafkaMessageStream All topics that match topicRegex will be returned in a single message stream, which can then be iterated. > Separate out Kafka mirroring into a stand-alone app > --------------------------------------------------- > > Key: KAFKA-249 > URL: https://issues.apache.org/jira/browse/KAFKA-249 > Project: Kafka > Issue Type: Improvement > Components: core > Reporter: Joel Koshy > Assignee: Joel Koshy > Fix For: 0.7.1 > > Attachments: KAFKA-249.v1.patch > > > I would like to discuss on this jira, the feasibility/benefits of separating > out Kafka's mirroring feature from the broker into a stand-alone app, as it > currently has a couple of limitations and issues. > For example, we recently had to deal with Kafka mirrors that were in fact > idle due to the fact that mirror threads were not created at start-up due to > a rebalancing exception, but the Kafka broker itself did not shutdown. This > has since been fixed, but is indicative of (avoidable) problems in embedding > non-broker specific features in the broker. > Logically, it seems to make sense to separate it out to achieve better > division of labor. Furthermore, enhancements to mirroring may be less > clunky to implement and use with a stand-alone app. For example to support > custom partitioning on the target cluster, or to mirror from multiple > clusters we would probably need to be able to pass in multiple embedded > consumer/embedded producer configs, which would be less ugly if the > mirroring process were a stand-alone app. Also, if we break it out, it > would be convenient to use as a "consumption engine" for the console > consumer which will make it easier to add on features such as wildcards in > topic consumption, since it contains a ZooKeeper topic discovery component. > Any suggestions and/or objections to this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira