Hello. I've implemented a Kafka Consumer Application which consume large number of monitoring data from Kafka Broker and analyze those data accordingly.
I referred to a guide, http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client, since I thought the app needs to implement multi-threading for Kafka Consumer per Topic. Actually, A topic is assigned to each open-source monitoring software, e.g., Nagios, Collectd, etc., in order to distinguish those because each of these uses its own message format, such as JSON, String, and so on. There was, however, an Exception even though my source code for the Kafka Consumer are mostly copied and pasted from the guide; *java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access* First Question. Could the implementation in the guide really prevent the Exception? And second Question is, could the KafkaConsumer support such huge amount of data with one thread? The KafkaConsumer seems not thread-safe, and it can subscribe multi-topics at once. Do I need to change the implementation from the multi-threaded to one-thread and subscribing multi-topics?... I'm just wonder whether a KafkaConsumer is able to stand the bunch of data without performance degradation. Thanks in advance! Best regards KIM
