[
https://issues.apache.org/jira/browse/KAFKA-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15013129#comment-15013129
]
Allen Chan commented on KAFKA-2627:
-----------------------------------
I am also experiencing the same issue.
Allocated 10G to heap and getting the same gc errors in gc.log even though 9GB
of heap is unused.
> Kafka Heap Size increase impact performance badly
> -------------------------------------------------
>
> Key: KAFKA-2627
> URL: https://issues.apache.org/jira/browse/KAFKA-2627
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Environment: CentOS Linux release 7.0.1406 (Core)
> NAME="CentOS Linux"
> VERSION="7 (Core)"
> ID="centos"
> ID_LIKE="rhel fedora"
> VERSION_ID="7"
> PRETTY_NAME="CentOS Linux 7 (Core)"
> ANSI_COLOR="0;31"
> CPE_NAME="cpe:/o:centos:centos:7"
> HOME_URL="https://www.centos.org/";
> BUG_REPORT_URL="https://bugs.centos.org/";
> CentOS Linux release 7.0.1406 (Core)
> CentOS Linux release 7.0.1406 (Core)
> Reporter: Mihir Pandya
>
> Initial Kafka server was configured with
> KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
> As we have high resource to utilize, we changed it to below value
> KAFKA_HEAP_OPTS="-Xmx16G -Xms8G"
> Change highly impacted Kafka & Zookeeper, we started getting various issue at
> both end.
> We were not getting all replica in ISR. And it was an issue with Leader
> Selection which in-turn throwing Socket Connection Error.
> To debug, we checked kafaServer-gc.log, we were getting GC(Allocation
> Failure) though we have lot more Memory is avalable.
> ============== GC Error ===============
> 2015-10-08T09:43:08.796+0000: 4.651: [GC (Allocation Failure) 4.651: [ParNew:
> 272640K->7265K(306688K), 0.0277514 secs] 272640K->7265K(1014528K), 0.0281243
> secs] [Times: user=0.03 sys=0.05, real=0.03 secs]
> 2015-10-08T09:43:11.317+0000: 7.172: [GC (Allocation Failure) 7.172: [ParNew:
> 279905K->3793K(306688K), 0.0157898 secs] 279905K->3793K(1014528K), 0.0159913
> secs] [Times: user=0.03 sys=0.01, real=0.02 secs]
> 2015-10-08T09:43:13.522+0000: 9.377: [GC (Allocation Failure) 9.377: [ParNew:
> 276433K->2827K(306688K), 0.0064236 secs] 276433K->2827K(1014528K), 0.0066834
> secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
> 2015-10-08T09:43:15.518+0000: 11.372: [GC (Allocation Failure) 11.373:
> [ParNew: 275467K->3090K(306688K), 0.0055454 secs] 275467K->3090K(1014528K),
> 0.0057979 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> 2015-10-08T09:43:17.558+0000: 13.412: [GC (Allocation Failure) 13.412:
> [ParNew: 275730K->3346K(306688K), 0.0053757 secs] 275730K->3346K(1014528K),
> 0.0055039 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> ====================================================
> ============= Other Kafka Errors =========================
> [2015-10-01 15:35:19,039] INFO conflict in /brokers/ids/3 data:
> {"jmx_port":-1,"timestamp":"1443709506024","host":"<HOST>","version":1,"port":9092}
> stored data:
> {"jmx_port":-1,"timestamp":"1443702430352","host":"<HOST>","version":1,"port":9092}
> (kafka.utils.ZkUtils$)
> [2015-10-01 15:35:19,042] INFO I wrote this conflicted ephemeral node
> [{"jmx_port":-1,"timestamp":"1443709506024","host":"<HOST>","version":1,"port":9092}]
> at /brokers/ids/3 a while back in a different session, hence I will backoff
> for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
> [2015-10-01 15:23:12,378] INFO Closing socket connection to /172.28.72.162.
> (kafka.network.Processor)
> [2015-10-01 15:23:12,378] INFO Closing socket connection to /172.28.72.162.
> (kafka.network.Processor)
> [2015-10-01 15:21:53,831] ERROR [ReplicaFetcherThread-4-1], Error for
> partition [workorder-topic,1] to broker 1:class
> kafka.common.NotLeaderForPartitionException
> (kafka.server.ReplicaFetcherThread)
> [2015-10-01 15:21:53,834] ERROR [ReplicaFetcherThread-4-1], Error for
> partition [workorder-topic,1] to broker 1:class
> kafka.common.NotLeaderForPartitionException
> (kafka.server.ReplicaFetcherThread)
> [2015-10-01 15:21:53,835] ERROR [ReplicaFetcherThread-4-1], Error for
> partition [workorder-topic,1] to broker 1:class
> kafka.common.NotLeaderForPartitionException
> (kafka.server.ReplicaFetcherThread)
> [2015-10-01 15:21:53,837] ERROR [ReplicaFetcherThread-4-1], Error for
> partition [workorder-topic,1] to broker 1:class
> kafka.common.NotLeaderForPartitionException
> (kafka.server.ReplicaFetcherThread)
> [2015-10-01 15:20:36,210] WARN [ReplicaFetcherThread-0-2], Error in fetch
> Name: FetchRequest; Version: 0; CorrelationId: 9; ClientId:
> ReplicaFetcherThread-0-2; ReplicaId: 3; MaxWait: 500 ms; MinBytes: 1 bytes;
> RequestInfo: [__consumer_offsets,17] ->
> PartitionFetchInfo(0,1048576),[__consumer_offsets,23] ->
> PartitionFetchInfo(0,1048576),[__consumer_offsets,29] ->
> PartitionFetchInfo(0,1048576),[__consumer_offsets,35] ->
> PartitionFetchInfo(0,1048576),[__consumer_offsets,41] ->
> PartitionFetchInfo(0,1048576),[__consumer_offsets,5] ->
> PartitionFetchInfo(0,1048576),[__consumer_offsets,11] ->
> PartitionFetchInfo(0,1048576),[__consumer_offsets,47] ->
> PartitionFetchInfo(0,1048576). Possible cause:
> java.net.SocketTimeoutException (kafka.server.ReplicaFetcherThread)
> [2015-10-01 15:20:36,210] INFO Reconnect due to socket error:
> java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
> [2015-10-01 15:20:38,238] WARN [ReplicaFetcherThread-1-2], Error in fetch
> Name: FetchRequest; Version: 0; CorrelationId: 9; ClientId:
> ReplicaFetcherThread-1-2; ReplicaId: 3; MaxWait: 500 ms; MinBytes: 1 bytes;
> RequestInfo: [tech-topic,1] -> PartitionFetchInfo(6966109,1048576). Possible
> cause: java.net.SocketTimeoutException (kafka.server.ReplicaFetcherThread)
> =======================================================
> I have replaced actual hostname with <HOST>.
> Once we reverted parameter KAFKA_HEAP_OPTS to 1G, all went well.
> Required your assistance for the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
