Hi, We're running the Trunk version of Kafka (for its SSL feature) and recently I'm trying to enable the kafka manager with it. After enabling that, I find out some machine's Kafka Server is dead. Looking at the server.log, it has the following logs. java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:19:41,310] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:19:44,766] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:19:48,567] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:19:52,112] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:19:56,307] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space
And then I tried to start Kafka on these dead machines, but find the errors below: [2015-09-12 00:43:44,878] WARN Found an corrupted index file, /datadrive/kafka-logs/userlogs-139/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2015-09-12 00:43:44,883] WARN Found an corrupted index file, /datadrive/kafka-logs/userlogs-166/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2015-09-12 00:43:44,888] WARN Found an corrupted index file, /datadrive/kafka-logs/exoactivitylogs-114/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2015-09-12 00:43:44,895] WARN Found an corrupted index file, /datadrive/kafka-logs/exoactivitylogs-15/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2015-09-12 00:43:44,900] WARN Found an corrupted index file, /datadrive/kafka-logs/exoactivitylogs-178/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2015-09-12 00:43:44,906] WARN Found an corrupted index file, /datadrive/kafka-logs/__consumer_offsets-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2015-09-12 00:43:44,912] WARN Found an corrupted index file, /datadrive/kafka-logs/userlogs-147/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2015-09-12 00:43:44,917] WARN Found an corrupted index file, /datadrive/kafka-logs/exoactivitylogs-95/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) Exception in thread "kafka-log-cleaner-thread-0" java.lang.OutOfMemoryError: Java heap space Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)" [2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server 10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181) (org.apache.zookeeper.ClientCnxn) [2015-09-12 00:45:39,466] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:45:39,467] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:45:39,732] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task 'highwatermark-checkpoint' (kafka.utils.KafkaScheduler) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:45:49,070] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:45:49,702] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:45:49,912] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 00:45:50,140] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space >From zookeeper, I can see that node appear in /brokers/ids, but disappear when the error above happens. Do you know what's the problem? Any hint is very appreciated. Thanks, Qi
