Hi,
Can anyone help with our kafka servers. Below is the service status.
[root@d-u2-kfk-ms-03 bin]# systemctl status kafka -l
● kafka.service - Apache Kafka server (broker)
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset:
disabled)
Active: active (running) since Sat 2020-08-01 17:50:16 UTC; 21min ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 106439 (java)
CGroup: /system.slice/kafka.service
└─106439 java -Xmx1G -Xms1G -server -XX:+UseG1GC
-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
-XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true
-Xloggc:/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=9990
-Dkafka.logs.dir=/opt/kafka/bin/../logs
-Dlog4j.configuration=file:/opt/kafka/config/log4j.properties -cp
/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b32.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.5.jar:/opt/kafka/bin/../libs/connect-api-1.1.0.jar:/opt/kafka/bin/../libs/connect-file-1.1.0.jar:/opt/kafka/bin/../libs/connect-json-1.1.0.jar:/opt/kafka/bin/../libs/connect-runtime-1.1.0.jar:/opt/kafka/bin/../libs/connect-transforms-1.1.0.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0-b32.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0-b32.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0-b32.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.4.jar:/opt/kafka/bin/../libs/jackson-core-2.9.4.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.4.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.4.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.4.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.4.jar:/opt/kafka/bin/../libs/javassist-3.20.0-GA.jar:/opt/kafka/bin/../libs/javassist-3.21.0-GA.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/bin/../libs/javax.inject-1.jar:/opt/kafka/bin/../libs/javax.inject-2.5.0-b32.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/kafka/bin/../libs/jersey-client-2.25.1.jar:/opt/kafka/bin/../libs/jersey-common-2.25.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.25.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.25.1.jar:/opt/kafka/bin/../libs/jersey-guava-2.25.1.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.25.1.jar:/opt/kafka/bin/../libs/jersey-server-2.25.1.jar:/opt/kafka/bin/../libs/jetty-client-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-continuation-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-http-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-io-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-security-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-server-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-servlet-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-servlets-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-util-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.0.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.0-test-sources.jar:/opt/kafka/bin/../libs/kafka-clients-1.1.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-1.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-1.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-1.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-1.1.0.jar:/opt/kafka/bin/../libs/kafka-tools-1.1.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.4.jar:/opt/kafka/bin/../libs/maven-artifact-3.5.2.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.7.3.jar:/opt/kafka/bin/../libs/scala-library-2.11.12.jar:/opt/kafka/bin/../libs/scala-logging_2.11-3.7.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.1.jar:/opt/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/bin/../libs/zkclient-0.10.jar:/opt/kafka/bin/../libs/zookeeper-3.4.10.jar
kafka.Kafka /opt/kafka/config/server.properties
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
replicas for partition psvc-global-data-0 is [1], below required minimum [2]
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
18:11:52,618] ERROR [ReplicaManager broker=3] Error processing append operation
on partition psvc-global-data-0 (kafka.server.ReplicaManager)
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
replicas for partition psvc-global-data-0 is [1], below required minimum [2]
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
18:11:52,619] ERROR [ReplicaManager broker=3] Error processing append operation
on partition psvc-global-data-0 (kafka.server.ReplicaManager)
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
replicas for partition psvc-global-data-0 is [1], below required minimum [2]
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
18:11:52,619] ERROR [ReplicaManager broker=3] Error processing append operation
on partition psvc-global-data-0 (kafka.server.ReplicaManager)
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
replicas for partition psvc-global-data-0 is [1], below required minimum [2]
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
18:11:52,625] ERROR [ReplicaManager broker=3] Error processing append operation
on partition psvc-global-data-0 (kafka.server.ReplicaManager)
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
replicas for partition psvc-global-data-0 is [1], below required minimum [2]
Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
18:11:52,626] ERROR [ReplicaManager broker=3] Error processing append operation
on partition psvc-global-data-0 (kafka.server.ReplicaManager)
[root@d-u2-kfk-ms-03 bin]#
Your help is much appreciated.
Thanks,
Manoj Chitrala
