And I tried to clean up the whole kafka-logs folder, and then starts the kafka server again. It will the following errors: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/share/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/share/kafka/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [2015-09-12 00:58:32,203] WARN No meta.properties file under dir /datadrive/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint) Exception in thread "kafka-log-cleaner-thread-0" java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-33" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-51" log4j:ERROR No output stream or file set for the appender named [kafkaAppender]. Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-56" [2015-09-12 00:59:54,546] ERROR Uncaught exception in thread 'kafka-network-thread-3-SSL-33': (org.apache.kafka.common.utils.Utils) [2015-09-12 01:00:34,261] ERROR Uncaught exception in thread 'kafka-network-thread-3-SSL-51': (org.apache.kafka.common.utils.Utils) [2015-09-12 01:00:51,108] ERROR Uncaught exception in thread 'kafka-network-thread-3-SSL-56': (org.apache.kafka.common.utils.Utils) [2015-09-12 01:00:48,867] ERROR Processor got uncaught exception. (kafka.network.Processor) [2015-09-12 01:00:48,867] ERROR Processor got uncaught exception. (kafka.network.Processor) [2015-09-12 01:00:49,121] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 01:00:57,643] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 01:00:54,028] ERROR Uncaught exception in thread 'kafka-network-thread-3-SSL-40': (org.apache.kafka.common.utils.Utils) java.lang.OutOfMemoryError: Java heap space [2015-09-12 01:00:53,656] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 01:00:53,656] ERROR Uncaught exception in thread 'kafka-network-thread-3-SSL-58': (org.apache.kafka.common.utils.Utils) java.lang.OutOfMemoryError: Java heap space Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-46" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-61" [2015-09-12 01:00:53,043] WARN Session 0x24f84cb22a90029 for server 10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2015-09-12 01:01:22,419] ERROR from main-SendThread(10.1.130.12:2181) (org.apache.zookeeper.ClientCnxn) [2015-09-12 01:01:22,051] ERROR Processor got uncaught exception. (kafka.network.Processor) [2015-09-12 01:01:23,489] ERROR Uncaught exception in thread 'kafka-network-thread-3-SSL-46': (org.apache.kafka.common.utils.Utils) [2015-09-12 01:01:21,313] ERROR Processor got uncaught exception. (kafka.network.Processor) [2015-09-12 01:01:24,438] ERROR Uncaught exception in thread 'kafka-network-thread-3-SSL-61': (org.apache.kafka.common.utils.Utils) [2015-09-12 01:01:18,075] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 01:01:02,539] ERROR Uncaught exception in scheduled task 'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler) java.lang.OutOfMemoryError: Java heap space [2015-09-12 01:01:34,023] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space [2015-09-12 01:01:35,676] ERROR Uncaught exception in scheduled task 'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler) And the broker does not appear in zookeeper. Any idea? Thanks, Qi On Fri, Sep 11, 2015 at 5:53 PM, Qi Xu <shkir...@gmail.com> wrote: > Hi, > We're running the Trunk version of Kafka (for its SSL feature) and > recently I'm trying to enable the kafka manager with it. > After enabling that, I find out some machine's Kafka Server is dead. > Looking at the server.log, it has the following logs. > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:19:41,310] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:19:44,766] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:19:48,567] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:19:52,112] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:19:56,307] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > > > And then I tried to start Kafka on these dead machines, but find the > errors below: > [2015-09-12 00:43:44,878] WARN Found an corrupted index file, > /datadrive/kafka-logs/userlogs-139/00000000000000000000.index, deleting and > rebuilding index... (kafka.log.Log) > [2015-09-12 00:43:44,883] WARN Found an corrupted index file, > /datadrive/kafka-logs/userlogs-166/00000000000000000000.index, deleting and > rebuilding index... (kafka.log.Log) > [2015-09-12 00:43:44,888] WARN Found an corrupted index file, > /datadrive/kafka-logs/exoactivitylogs-114/00000000000000000000.index, > deleting and rebuilding index... (kafka.log.Log) > [2015-09-12 00:43:44,895] WARN Found an corrupted index file, > /datadrive/kafka-logs/exoactivitylogs-15/00000000000000000000.index, > deleting and rebuilding index... (kafka.log.Log) > [2015-09-12 00:43:44,900] WARN Found an corrupted index file, > /datadrive/kafka-logs/exoactivitylogs-178/00000000000000000000.index, > deleting and rebuilding index... (kafka.log.Log) > [2015-09-12 00:43:44,906] WARN Found an corrupted index file, > /datadrive/kafka-logs/__consumer_offsets-4/00000000000000000000.index, > deleting and rebuilding index... (kafka.log.Log) > [2015-09-12 00:43:44,912] WARN Found an corrupted index file, > /datadrive/kafka-logs/userlogs-147/00000000000000000000.index, deleting and > rebuilding index... (kafka.log.Log) > [2015-09-12 00:43:44,917] WARN Found an corrupted index file, > /datadrive/kafka-logs/exoactivitylogs-95/00000000000000000000.index, > deleting and rebuilding index... (kafka.log.Log) > Exception in thread "kafka-log-cleaner-thread-0" > java.lang.OutOfMemoryError: Java heap space > > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)" > [2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server > 10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection > and attempting reconnect (org.apache.zookeeper.ClientCnxn) > [2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181) > (org.apache.zookeeper.ClientCnxn) > [2015-09-12 00:45:39,466] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:45:39,467] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:45:39,732] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task > 'highwatermark-checkpoint' (kafka.utils.KafkaScheduler) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:45:49,070] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:45:49,702] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:45:49,912] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > [2015-09-12 00:45:50,140] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.OutOfMemoryError: Java heap space > > From zookeeper, I can see that node appear in /brokers/ids, but disappear > when the error above happens. > > Do you know what's the problem? Any hint is very appreciated. > > Thanks, > Qi > >
