[ https://issues.apache.org/jira/browse/KAFKA-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-170: -------------------------------- Priority: Critical (was: Major) Labels: replication (was: ) Since this is proposing an API change, it will be good to think about this as part of 0.8 > Support for non-blocking polling on multiple streams > ---------------------------------------------------- > > Key: KAFKA-170 > URL: https://issues.apache.org/jira/browse/KAFKA-170 > Project: Kafka > Issue Type: New Feature > Components: core > Affects Versions: 0.8 > Reporter: Jay Kreps > Priority: Critical > Labels: replication > > Currently we provide a blocking iterator in the consumer. This is a good > mechanism for consuming data from a single topic, but is limited as a > mechanism for polling multiple streams. > For example if one wants to implement a non-blocking union across multiple > streams this is hard to do because calls may block indefinitely. A similar > situation arrises if trying to implement a streaming join of between two > streams. > I would propose two changes: > 1. Implement a next(timeout) interface on KafkaMessageStream. This will > easily handle some simple cases with minimal change. This handles certain > limited cases nicely and is easy to implement, but doesn't actually cover the > two cases above. > 2. Add an interface to poll streams. > I don't know the best approach for the later api, but it is important to get > it right. One option would be to add a > ConsumerConnector.drainTopics("topic1", "topic2", ...) which blocks until > there is at least one message and then returns a list of triples (topic, > partition, message). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira