Broker regularly facing memory issues

[Arpit Gogia]

Hello Kafka user group


I am running a Kafka cluster with 3 brokers and have been experiencing
frequent OutOfMemory errors each time with similar error stack trace


java.io.IOException: Map failed

    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938)

    at
kafka.log.AbstractIndex$$anonfun$resize$1.apply$mcZ$sp(AbstractIndex.scala:188)

    at
kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:173)

    at
kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:173)

    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)

    at kafka.log.AbstractIndex.resize(AbstractIndex.scala:173)

    at
kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply$mcZ$sp(AbstractIndex.scala:242)

    at
kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:242)

    at
kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:242)

    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)

    at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:241)

    at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:501)

    at kafka.log.Log$$anonfun$roll$2$$anonfun$apply$32.apply(Log.scala:1635)

    at kafka.log.Log$$anonfun$roll$2$$anonfun$apply$32.apply(Log.scala:1635)

    at scala.Option.foreach(Option.scala:257)

    at kafka.log.Log$$anonfun$roll$2.apply(Log.scala:1635)

    at kafka.log.Log$$anonfun$roll$2.apply(Log.scala:1599)

    at kafka.log.Log.maybeHandleIOException(Log.scala:1996)

    at kafka.log.Log.roll(Log.scala:1599)

    at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1434)

    at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1429)

    at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1429)

    at kafka.log.Log.maybeHandleIOException(Log.scala:1996)

    at kafka.log.Log.deleteSegments(Log.scala:1429)

    at kafka.log.Log.deleteOldSegments(Log.scala:1424)

    at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1501)

    at kafka.log.Log.deleteOldSegments(Log.scala:1492)

    at
kafka.log.LogCleaner$CleanerThread$$anonfun$cleanFilthiestLog$1.apply(LogCleaner.scala:328)

    at
kafka.log.LogCleaner$CleanerThread$$anonfun$cleanFilthiestLog$1.apply(LogCleaner.scala:324)

    at scala.collection.immutable.List.foreach(List.scala:392)

    at
kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:324)

    at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:300)

    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

Caused by: java.lang.OutOfMemoryError: Map failed

    at sun.nio.ch.FileChannelImpl.map0(Native Method)

    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935)

    ... 32 more


Each broker possesses 16 GB of memory out of which 10 GB is allotted to the
JVM as heap. Total partition count on each broker is approximately 2000
with an average partition size of 300 MB.


After looking around, I found out that increasing the OS level memory map
area limit `vm.max_map_count` is a viable solution, since Kafka memory
map’s segment files while rolling over and the above stack trace indicates
a failure in doing that. Since then I have increased this every time a
broker goes down with this error. Currently I am at 250,000 on two brokers
and 200,000 on one, which is very high considering the estimation formula
mentioned at https://kafka.apache.org/documentation/#os. Most recently I
started to monitor the memory map file count (using /proc/<pid>/maps) of
the Kafka process on each broker, below is the graph.


[image: Screenshot 2019-09-27 at 12.02.38 PM.png]


My concern is that this value is on an overall increasing trend, with an
average increase of 27.7K across brokers in the roughly 2 days of
monitoring.


Following are my questions:

   1. Will I have to keep incrementing `vm.max_map_count` till I arrive at
   a stable value?
   2. Could this by any chance indicate a memory leak? Maybe in the
   subroutine that rolls over segment files?
   3. Could the lack of page cache memory be a cause as well? Volume of
   cached memory seems to remain consistent across time so it doesn’t appear
   to be a suspect by I am not ruling it out for now. As a mitigation I will
   be decreasing the JVM heap next time so that more memory is available for
   page cache.

-- 

*Arpit Gogia **|* *Data Engineer*