Hi Cosmin, do you have default server configuration on these new nodes you're setting up? I'd check consumer's socket.timeout.ms, maybe someone set it to 30 instead of 30 000 :) Speaking from my own experience (I had the same symptom and this turned out to be the cause).
Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext <http://sematext.com/> | Contact <http://sematext.com/about/contact.html> On Thu, Jan 7, 2016 at 11:23 PM, Cosmin Marginean <[email protected]> wrote: > Hi > > I have a straightforward piece of code that creates a consumer (Kafka > 0.9.0.0). > > Properties props = new Properties(); > props.put("zookeeper.connect", zookeeperServers); > props.put(org.apache.kafka.clients.consumer.ConsumerConfig.GROUP_ID_CONFIG, > groupId); > log.info("Starting consumer group for topic {} and group ID {}. Zookeeper > servers: {}", topic, groupId, zookeeperServers); > consumer = kafka.consumer.Consumer.createJavaConsumerConnector(new > ConsumerConfig(props)); > log.info("Consumer group started for topic {} and group ID {}", topic, > groupId); > > We’ve run this countless times without any issues, but now we’re deploying a > new environment (AWS, just like the ones before) and it appears that the > client Java process dies entirely (without any logs/crash report/etc). This > happens right after logging the “Starting consumer group..”, so presumably > when it tries to createJavaConsumerConnector > > Agreeably, this might be “environmental”, but even though we triple checked > everything (network setup, kafka logs, zookeeper logs, etc), we couldn’t > identify anything suspicious yet. So what I'd like to know is if there’s a > way to add further Kafka diagnosis/logging. Attached (trace-logging.txt) is > further logging after turning everything to TRACE, and at the top you can see > the message “Starting consumer…”, but with nothing really suspicious as far > as I can tell. > > > As an additional piece of information, Zookeeper does report the following > when this happens > > 2016-01-07 21:58:44,763 [myid:1] - WARN > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of > stream exception > EndOfStreamException: Unable to read additional data from client sessionid > 0x1521e14797c0001, likely client has closed socket > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > at > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) > at java.lang.Thread.run(Thread.java:745) > 2016-01-07 21:58:44,764 [myid:1] - INFO > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed > socket connection for client /10.100.101.159:41613 which had sessionid > 0x1521e14797c0001 > > > Any suggestions would be appreciated. > > Thank you > > Cosmin > > >
