HI, On Sat, 16 Nov 2019 at 19:54, Oliver Eckle <[email protected]> wrote:
> Hi, > > yes it is intentional, but just because I don't know better and want to > spare a little resources? > I never understood the benefit of having more brokers than replicas with the intention of saving resources. A lot of people do that, and the Kafka community seems to be okay with it ( i.e. not documentation or caution for NOT doing that). Please make sure you use it to full extent. For your case, I believe your log and index file stored on the affected broker (or rather, the PV attached to it if you have one) may have been corrupted. The best way (rather than debugging and investigating logs endlessly) is to simply delete the pod and let it start again. Also, make sure that it does't refer to the old files (if you have a PV/StatefulSet with it). It's important that upon restart the broker builds all the data files themselves than referring to previously stored files. Try that and see how it goes. Thanks, >From your answer I guess the preferred way is having a replication of 3? > > > -----Ursprüngliche Nachricht----- > Von: M. Manna <[email protected]> > Gesendet: Samstag, 16. November 2019 20:27 > An: [email protected] > Betreff: Re: Kafka Broker do not recover after crash > > Hi, > > On Sat, 16 Nov 2019 at 19:21, Oliver Eckle <[email protected]> wrote: > > > Hello, > > > > > > > > having a Kafka Cluster running in Kubernetes with 3 Brokers and all > > replikations (topic, offsets) set to 2. > > > This sounds strange. You have 3 brokers and replication set to 2. Is this > intentional ? > > > > > > For whatever reason one of the broker crash and restartes. And since > > it circles in some kind of restart/crash loop. > > > > Any idea how to recover? > > > > > > > > Whole Logfile is like that: > > > > > > > > [38;5;6m [38;5;5m19:15:42.58 [0m > > > > [38;5;6m [38;5;5m19:15:42.58 [0m[1mWelcome to the Bitnami kafka > > container[0m > > > > [38;5;6m [38;5;5m19:15:42.58 [0mSubscribe to project updates by > > watching [1mhttps://github.com/bitnami/bitnami-docker-kafka[0m > <http://github.com/bitnami/bitnami-docker-kafka%5B0m> > > <http://github.com/bitnami/bitnami-docker-kafka%5B0m> > > > > [38;5;6m [38;5;5m19:15:42.58 [0mSubmit issues and feature requests at > > [1mhttps://github.com/bitnami/bitnami-docker-kafka/issues[0m > <http://github.com/bitnami/bitnami-docker-kafka/issues%5B0m> > > <http://github.com/bitnami/bitnami-docker-kafka/issues%5B0m> > > > > [38;5;6m [38;5;5m19:15:42.58 [0mSend us your feedback at > > [[email protected][0m > > > > [38;5;6m [38;5;5m19:15:42.59 [0m > > > > [38;5;6m [38;5;5m19:15:42.59 [0m[38;5;2mINFO [0m ==> ** Starting Kafka > > setup > > ** > > > > [38;5;6m [38;5;5m19:15:42.83 [0m[38;5;3mWARN [0m ==> You set the > > environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, > > do not use this flag in a production environment. > > > > [38;5;6m [38;5;5m19:15:42.84 [0m[38;5;2mINFO [0m ==> Initializing > Kafka... > > > > [38;5;6m [38;5;5m19:15:42.84 [0m[38;5;2mINFO [0m ==> No injected > > configuration files found, creating default config files > > > > [38;5;6m [38;5;5m19:15:43.83 [0m[38;5;2mINFO [0m ==> ** Kafka setup > > finished! ** > > > > > > > > [38;5;6m [38;5;5m19:15:43.84 [0m[38;5;2mINFO [0m ==> ** Starting Kafka > > ** > > > > [2019-11-16 19:15:49,625] INFO Registered > > kafka:type=kafka.Log4jController MBean > > (kafka.utils.Log4jControllerRegistration$) > > > > [2019-11-16 19:15:52,933] INFO Registered signal handlers for TERM, > > INT, HUP > > (org.apache.kafka.common.utils.LoggingSignalHandler) > > > > [2019-11-16 19:15:52,934] INFO starting (kafka.server.KafkaServer) > > > > [2019-11-16 19:15:52,935] INFO Connecting to zookeeper on > > kafka-zookeeper > > (kafka.server.KafkaServer) > > > > [2019-11-16 19:15:53,230] INFO [ZooKeeperClient Kafka server] > > Initializing a new session to kafka-zookeeper. > > (kafka.zookeeper.ZooKeeperClient) > > > > [2019-11-16 19:15:53,331] INFO Client > > > > environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255a > > c140bc f, built on 03/06/2019 16:18 GMT > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,331] INFO Client > > environment:host.name=kafka-1.kafka-headless.bd-iot.svc.cluster.local > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,331] INFO Client > > environment:java.version=1.8.0_232 > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,331] INFO Client > > environment:java.vendor=AdoptOpenJDK > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,332] INFO Client > > environment:java.home=/opt/bitnami/java > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,332] INFO Client > > > > > environment:java.class.path=/opt/bitnami/kafka/bin/../libs/activation-1.1.1. > > > > jar:/opt/bitnami/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/o > > pt/bit > > > > nami/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/bitnami/kafka/bin/../ > > libs/a > > > > udience-annotations-0.5.0.jar:/opt/bitnami/kafka/bin/../libs/commons-l > > ang3-3 > > > > .8.1.jar:/opt/bitnami/kafka/bin/../libs/connect-api-2.3.1.jar:/opt/bit > > nami/k > > > > afka/bin/../libs/connect-basic-auth-extension-2.3.1.jar:/opt/bitnami/k > > afka/b > > > > in/../libs/connect-file-2.3.1.jar:/opt/bitnami/kafka/bin/../libs/conne > > ct-jso > > > > n-2.3.1.jar:/opt/bitnami/kafka/bin/../libs/connect-runtime-2.3.1.jar:/ > > opt/bi > > > > tnami/kafka/bin/../libs/connect-transforms-2.3.1.jar:/opt/bitnami/kafk > > a/bin/ > > > > ../libs/guava-20.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-api-2.5.0.ja > > r:/opt > > > > /bitnami/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/bitnami/kafka/bi > > n/../l > > > > ibs/hk2-utils-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/jackson-annotat > > ions-2 > > > > .10.0.jar:/opt/bitnami/kafka/bin/../libs/jackson-core-2.10.0.jar:/opt/ > > bitnam > > > > i/kafka/bin/../libs/jackson-databind-2.10.0.jar:/opt/bitnami/kafka/bin > > /../li > > > > bs/jackson-dataformat-csv-2.10.0.jar:/opt/bitnami/kafka/bin/../libs/ja > > ckson- > > > > datatype-jdk8-2.10.0.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs- > > base-2 > > > > > .10.0.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0. > > > > jar:/opt/bitnami/kafka/bin/../libs/jackson-module-jaxb-annotations-2.1 > > 0.0.ja > > > > r:/opt/bitnami/kafka/bin/../libs/jackson-module-paranamer-2.10.0.jar:/ > > opt/bi > > > > tnami/kafka/bin/../libs/jackson-module-scala_2.11-2.10.0.jar:/opt/bitn > > ami/ka > > > > fka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/bitnami/kafka/bi > > n/../l > > > > > ibs/jakarta.annotation-api-1.3.4.jar:/opt/bitnami/kafka/bin/../libs/jakarta. > > inject-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/jakarta.ws > > .rs-api-2.1.5.jar: > > > > /opt/bitnami/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/bit > > nami/k > > > > afka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/bitnami/kafka/bin/../li > > bs/jav > > > > > ax.servlet-api-3.1.0.jar:/opt/bitnami/kafka/bin/../libs/javax.ws.rs-api-2.1. > > > > 1.jar:/opt/bitnami/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/bitnami/k > > afka/b > > > > in/../libs/jersey-client-2.28.jar:/opt/bitnami/kafka/bin/../libs/jerse > > y-comm > > > > on-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-2. > > 28.jar > > > > :/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar > > :/opt/ > > > > bitnami/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/bitnami/kafka/bin/. > > ./libs > > > > /jersey-media-jaxb-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-serv > > er-2.2 > > > > 8.jar:/opt/bitnami/kafka/bin/../libs/jetty-client-9.4.18.v20190429.jar > > :/opt/ > > > > bitnami/kafka/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt > > /bitna > > > > mi/kafka/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/bitnami/kafk > > a/bin/ > > > > ../libs/jetty-io-9.4.18.v20190429.jar:/opt/bitnami/kafka/bin/../libs/j > > etty-s > > > > ecurity-9.4.18.v20190429.jar:/opt/bitnami/kafka/bin/../libs/jetty-serv > > er-9.4 > > > > .18.v20190429.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlet-9.4.18. > > v20190 > > > > 429.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlets-9.4.18.v20190429 > > .jar:/ > > > > opt/bitnami/kafka/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/bit > > nami/k > > > > afka/bin/../libs/jopt-simple-5.0.4.jar:/opt/bitnami/kafka/bin/../libs/ > > jsr305 > > > > -3.0.2.jar:/opt/bitnami/kafka/bin/../libs/kafka-clients-2.3.1.jar:/opt > > /bitna > > > > > mi/kafka/bin/../libs/kafka-log4j-appender-2.3.1.jar:/opt/bitnami/kafka/bin/. > > > > ./libs/kafka-streams-2.3.1.jar:/opt/bitnami/kafka/bin/../libs/kafka-st > > reams- > > > > examples-2.3.1.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-scala_ > > 2.11-2 > > > > .3.1.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-test-utils-2.3.1 > > .jar:/ > > > > > opt/bitnami/kafka/bin/../libs/kafka-tools-2.3.1.jar:/opt/bitnami/kafka/bin/. > > > > ./libs/kafka_2.11-2.3.1-sources.jar:/opt/bitnami/kafka/bin/../libs/kaf > > ka_2.1 > > > > 1-2.3.1.jar:/opt/bitnami/kafka/bin/../libs/log4j-1.2.17.jar:/opt/bitna > > mi/kaf > > > > ka/bin/../libs/lz4-java-1.6.0.jar:/opt/bitnami/kafka/bin/../libs/maven > > -artif > > > > act-3.6.1.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-2.2.0.jar:/o > > pt/bit > > > > nami/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/bitnami/ka > > fka/bi > > > > > n/../libs/paranamer-2.8.jar:/opt/bitnami/kafka/bin/../libs/plexus-utils-3.2. > > > > 0.jar:/opt/bitnami/kafka/bin/../libs/reflections-0.9.11.jar:/opt/bitna > > mi/kaf > > > > ka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/bitnami/kafka/bin/../libs/sc > > ala-li > > > > brary-2.11.12.jar:/opt/bitnami/kafka/bin/../libs/scala-logging_2.11-3. > > 9.0.ja > > > > r:/opt/bitnami/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/bitnam > > i/kafk > > > > a/bin/../libs/slf4j-api-1.7.26.jar:/opt/bitnami/kafka/bin/../libs/slf4 > > j-log4 > > > > j12-1.7.26.jar:/opt/bitnami/kafka/bin/../libs/snappy-java-1.1.7.3.jar: > > /opt/b > > > > itnami/kafka/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/bitnami/k > > afka/b > > > > in/../libs/validation-api-2.0.1.Final.jar:/opt/bitnami/kafka/bin/../li > > bs/zkc > > > > lient-0.11.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-3.4.14.jar:/op > > t/bitn ami/kafka/bin/../libs/zstd-jni-1.4.0-1.jar > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,333] INFO Client > > > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64: > > /lib64 :/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,333] INFO Client environment:java.io.tmpdir=/tmp > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,334] INFO Client environment:java.compiler=<NA> > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,334] INFO Client environment:os.name=Linux > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,334] INFO Client environment:os.arch=amd64 > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,335] INFO Client > > environment:os.version=4.15.0-1060-azure > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,336] INFO Client environment:user.name=? > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,336] INFO Client environment:user.home=? > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,336] INFO Client environment:user.dir=/ > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,338] INFO Initiating client connection, > > connectString=kafka-zookeeper sessionTimeout=6000 > > watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@31304f > > 14 > > (org.apache.zookeeper.ZooKeeper) > > > > [2019-11-16 19:15:53,528] INFO [ZooKeeperClient Kafka server] Waiting > > until connected. (kafka.zookeeper.ZooKeeperClient) > > > > [2019-11-16 19:15:53,545] INFO Opening socket connection to server > > kafka-zookeeper/10.0.215.214:2181. Will not attempt to authenticate > > using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) > > > > [2019-11-16 19:15:53,552] INFO Socket connection established to > > kafka-zookeeper/10.0.215.214:2181, initiating session > > (org.apache.zookeeper.ClientCnxn) > > > > [2019-11-16 19:15:53,627] INFO Session establishment complete on > > server kafka-zookeeper/10.0.215.214:2181, sessionid = > > 0x10000810b780070, negotiated timeout = 6000 > > (org.apache.zookeeper.ClientCnxn) > > > > [2019-11-16 19:15:53,630] INFO [ZooKeeperClient Kafka server] Connected. > > (kafka.zookeeper.ZooKeeperClient) > > > > [2019-11-16 19:15:55,034] INFO Cluster ID = dvSQ1W2US72rcqGef9tm6w > > (kafka.server.KafkaServer) > > > > [2019-11-16 19:15:55,637] INFO KafkaConfig values: > > > > advertised.host.name = null > > > > advertised.listeners = > > PLAINTEXT://kafka-1.kafka-headless.bd-iot.svc.cluster.local:9092 > > > > advertised.port = null > > > > alter.config.policy.class.name = null > > > > alter.log.dirs.replication.quota.window.num = 11 > > > > alter.log.dirs.replication.quota.window.size.seconds = > > 1 > > > > authorizer.class.name = > > > > auto.create.topics.enable = true > > > > auto.leader.rebalance.enable = true > > > > background.threads = 10 > > > > broker.id = -1 > > > > broker.id.generation.enable = true > > > > broker.rack = null > > > > client.quota.callback.class = null > > > > compression.type = producer > > > > connection.failed.authentication.delay.ms = 100 > > > > connections.max.idle.ms = 600000 > > > > connections.max.reauth.ms = 0 > > > > control.plane.listener.name = null > > > > controlled.shutdown.enable = true > > > > controlled.shutdown.max.retries = 3 > > > > controlled.shutdown.retry.backoff.ms = 5000 > > > > controller.socket.timeout.ms = 30000 > > > > create.topic.policy.class.name = null > > > > default.replication.factor = 2 > > > > delegation.token.expiry.check.interval.ms = 3600000 > > > > delegation.token.expiry.time.ms = 86400000 > > > > delegation.token.master.key = null > > > > delegation.token.max.lifetime.ms = 604800000 > > > > delete.records.purgatory.purge.interval.requests = 1 > > > > delete.topic.enable = true > > > > fetch.purgatory.purge.interval.requests = 1000 > > > > group.initial.rebalance.delay.ms = 0 > > > > group.max.session.timeout.ms = 1800000 > > > > group.max.size = 2147483647 > > > > group.min.session.timeout.ms = 6000 > > > > host.name = > > > > inter.broker.listener.name = null > > > > inter.broker.protocol.version = 2.3-IV1 > > > > kafka.metrics.polling.interval.secs = 10 > > > > kafka.metrics.reporters = [] > > > > leader.imbalance.check.interval.seconds = 300 > > > > leader.imbalance.per.broker.percentage = 10 > > > > listener.security.protocol.map = > > PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SAS > > L_SSL > > > > listeners = PLAINTEXT://:9092 > > > > log.cleaner.backoff.ms = 15000 > > > > log.cleaner.dedupe.buffer.size = 134217728 > > > > log.cleaner.delete.retention.ms = 86400000 > > > > log.cleaner.enable = true > > > > log.cleaner.io.buffer.load.factor = 0.9 > > > > log.cleaner.io.buffer.size = 524288 > > > > log.cleaner.io.max.bytes.per.second = > > 1.7976931348623157E308 > > > > log.cleaner.max.compaction.lag.ms = > > 9223372036854775807 > > > > log.cleaner.min.cleanable.ratio = 0.5 > > > > log.cleaner.min.compaction.lag.ms = 0 > > > > log.cleaner.threads = 1 > > > > log.cleanup.policy = [delete] > > > > log.dir = /tmp/kafka-logs > > > > log.dirs = /bitnami/kafka/data > > > > log.flush.interval.messages = 10000 > > > > log.flush.interval.ms = 1000 > > > > log.flush.offset.checkpoint.interval.ms = 60000 > > > > log.flush.scheduler.interval.ms = 9223372036854775807 > > > > log.flush.start.offset.checkpoint.interval.ms = 60000 > > > > log.index.interval.bytes = 4096 > > > > log.index.size.max.bytes = 10485760 > > > > log.message.downconversion.enable = true > > > > log.message.format.version = 2.3-IV1 > > > > log.message.timestamp.difference.max.ms = > > 9223372036854775807 > > > > log.message.timestamp.type = CreateTime > > > > log.preallocate = false > > > > log.retention.bytes = 1073741824 > > > > log.retention.check.interval.ms = 300000 > > > > log.retention.hours = 168 > > > > log.retention.minutes = null > > > > log.retention.ms = null > > > > log.roll.hours = 168 > > > > log.roll.jitter.hours = 0 > > > > log.roll.jitter.ms = null > > > > log.roll.ms = null > > > > log.segment.bytes = 1073741824 > > > > log.segment.delete.delay.ms = 60000 > > > > max.connections = 2147483647 > > > > max.connections.per.ip = 2147483647 > > > > max.connections.per.ip.overrides = > > > > max.incremental.fetch.session.cache.slots = 1000 > > > > message.max.bytes = 1000012 > > > > metric.reporters = [] > > > > metrics.num.samples = 2 > > > > metrics.recording.level = INFO > > > > metrics.sample.window.ms = 30000 > > > > min.insync.replicas = 1 > > > > num.io.threads = 8 > > > > num.network.threads = 3 > > > > num.partitions = 1 > > > > num.recovery.threads.per.data.dir = 1 > > > > num.replica.alter.log.dirs.threads = null > > > > num.replica.fetchers = 1 > > > > offset.metadata.max.bytes = 4096 > > > > offsets.commit.required.acks = -1 > > > > offsets.commit.timeout.ms = 5000 > > > > offsets.load.buffer.size = 5242880 > > > > offsets.retention.check.interval.ms = 600000 > > > > offsets.retention.minutes = 10080 > > > > offsets.topic.compression.codec = 0 > > > > offsets.topic.num.partitions = 50 > > > > offsets.topic.replication.factor = 2 > > > > offsets.topic.segment.bytes = 104857600 > > > > password.encoder.cipher.algorithm = > > AES/CBC/PKCS5Padding > > > > password.encoder.iterations = 4096 > > > > password.encoder.key.length = 128 > > > > password.encoder.keyfactory.algorithm = null > > > > password.encoder.old.secret = null > > > > password.encoder.secret = null > > > > port = 9092 > > > > principal.builder.class = null > > > > producer.purgatory.purge.interval.requests = 1000 > > > > queued.max.request.bytes = -1 > > > > queued.max.requests = 500 > > > > quota.consumer.default = 9223372036854775807 > > > > quota.producer.default = 9223372036854775807 > > > > quota.window.num = 11 > > > > quota.window.size.seconds = 1 > > > > replica.fetch.backoff.ms = 1000 > > > > replica.fetch.max.bytes = 1048576 > > > > replica.fetch.min.bytes = 1 > > > > replica.fetch.response.max.bytes = 10485760 > > > > replica.fetch.wait.max.ms = 500 > > > > replica.high.watermark.checkpoint.interval.ms = 5000 > > > > replica.lag.time.max.ms = 10000 > > > > replica.socket.receive.buffer.bytes = 65536 > > > > replica.socket.timeout.ms = 30000 > > > > replication.quota.window.num = 11 > > > > replication.quota.window.size.seconds = 1 > > > > request.timeout.ms = 30000 > > > > reserved.broker.max.id = 1000 > > > > sasl.client.callback.handler.class = null > > > > sasl.enabled.mechanisms = [GSSAPI] > > > > sasl.jaas.config = null > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit > > > > sasl.kerberos.min.time.before.relogin = 60000 > > > > sasl.kerberos.principal.to.local.rules = [DEFAULT] > > > > sasl.kerberos.service.name = null > > > > sasl.kerberos.ticket.renew.jitter = 0.05 > > > > sasl.kerberos.ticket.renew.window.factor = 0.8 > > > > sasl.login.callback.handler.class = null > > > > sasl.login.class = null > > > > sasl.login.refresh.buffer.seconds = 300 > > > > sasl.login.refresh.min.period.seconds = 60 > > > > sasl.login.refresh.window.factor = 0.8 > > > > sasl.login.refresh.window.jitter = 0.05 > > > > sasl.mechanism.inter.broker.protocol = GSSAPI > > > > sasl.server.callback.handler.class = null > > > > security.inter.broker.protocol = PLAINTEXT > > > > socket.receive.buffer.bytes = 102400 > > > > socket.request.max.bytes = 104857600 > > > > socket.send.buffer.bytes = 102400 > > > > ssl.cipher.suites = [] > > > > ssl.client.auth = none > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > > > > ssl.endpoint.identification.algorithm = https > > > > ssl.key.password = null > > > > ssl.keymanager.algorithm = SunX509 > > > > ssl.keystore.location = null > > > > ssl.keystore.password = null > > > > ssl.keystore.type = JKS > > > > ssl.principal.mapping.rules = [DEFAULT] > > > > ssl.protocol = TLS > > > > ssl.provider = null > > > > ssl.secure.random.implementation = null > > > > ssl.trustmanager.algorithm = PKIX > > > > ssl.truststore.location = null > > > > ssl.truststore.password = null > > > > ssl.truststore.type = JKS > > > > > > transaction.abort.timed.out.transaction.cleanup.interval.ms > > = 60000 > > > > transaction.max.timeout.ms = 900000 > > > > > > transaction.remove.expired.transaction.cleanup.interval.ms > > = > > 3600000 > > > > transaction.state.log.load.buffer.size = 5242880 > > > > transaction.state.log.min.isr = 2 > > > > transaction.state.log.num.partitions = 50 > > > > transaction.state.log.replication.factor = 2 > > > > transaction.state.log.segment.bytes = 104857600 > > > > transactional.id.expiration.ms = 604800000 > > > > unclean.leader.election.enable = false > > > > zookeeper.connect = kafka-zookeeper > > > > zookeeper.connection.timeout.ms = 6000 > > > > zookeeper.max.in.flight.requests = 10 > > > > zookeeper.session.timeout.ms = 6000 > > > > zookeeper.set.acl = false > > > > zookeeper.sync.time.ms = 2000 > > > > (kafka.server.KafkaConfig) > > > > [2019-11-16 19:15:55,829] INFO KafkaConfig values: > > > > advertised.host.name = null > > > > advertised.listeners = > > PLAINTEXT://kafka-1.kafka-headless.bd-iot.svc.cluster.local:9092 > > > > advertised.port = null > > > > alter.config.policy.class.name = null > > > > alter.log.dirs.replication.quota.window.num = 11 > > > > alter.log.dirs.replication.quota.window.size.seconds = > > 1 > > > > authorizer.class.name = > > > > auto.create.topics.enable = true > > > > auto.leader.rebalance.enable = true > > > > background.threads = 10 > > > > broker.id = -1 > > > > broker.id.generation.enable = true > > > > broker.rack = null > > > > client.quota.callback.class = null > > > > compression.type = producer > > > > connection.failed.authentication.delay.ms = 100 > > > > connections.max.idle.ms = 600000 > > > > connections.max.reauth.ms = 0 > > > > control.plane.listener.name = null > > > > controlled.shutdown.enable = true > > > > controlled.shutdown.max.retries = 3 > > > > controlled.shutdown.retry.backoff.ms = 5000 > > > > controller.socket.timeout.ms = 30000 > > > > create.topic.policy.class.name = null > > > > default.replication.factor = 2 > > > > delegation.token.expiry.check.interval.ms = 3600000 > > > > delegation.token.expiry.time.ms = 86400000 > > > > delegation.token.master.key = null > > > > delegation.token.max.lifetime.ms = 604800000 > > > > delete.records.purgatory.purge.interval.requests = 1 > > > > delete.topic.enable = true > > > > fetch.purgatory.purge.interval.requests = 1000 > > > > group.initial.rebalance.delay.ms = 0 > > > > group.max.session.timeout.ms = 1800000 > > > > group.max.size = 2147483647 > > > > group.min.session.timeout.ms = 6000 > > > > host.name = > > > > inter.broker.listener.name = null > > > > inter.broker.protocol.version = 2.3-IV1 > > > > kafka.metrics.polling.interval.secs = 10 > > > > kafka.metrics.reporters = [] > > > > leader.imbalance.check.interval.seconds = 300 > > > > leader.imbalance.per.broker.percentage = 10 > > > > listener.security.protocol.map = > > PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SAS > > L_SSL > > > > listeners = PLAINTEXT://:9092 > > > > log.cleaner.backoff.ms = 15000 > > > > log.cleaner.dedupe.buffer.size = 134217728 > > > > log.cleaner.delete.retention.ms = 86400000 > > > > log.cleaner.enable = true > > > > log.cleaner.io.buffer.load.factor = 0.9 > > > > log.cleaner.io.buffer.size = 524288 > > > > log.cleaner.io.max.bytes.per.second = > > 1.7976931348623157E308 > > > > log.cleaner.max.compaction.lag.ms = > > 9223372036854775807 > > > > log.cleaner.min.cleanable.ratio = 0.5 > > > > log.cleaner.min.compaction.lag.ms = 0 > > > > log.cleaner.threads = 1 > > > > log.cleanup.policy = [delete] > > > > log.dir = /tmp/kafka-logs > > > > log.dirs = /bitnami/kafka/data > > > > log.flush.interval.messages = 10000 > > > > log.flush.interval.ms = 1000 > > > > log.flush.offset.checkpoint.interval.ms = 60000 > > > > log.flush.scheduler.interval.ms = 9223372036854775807 > > > > log.flush.start.offset.checkpoint.interval.ms = 60000 > > > > log.index.interval.bytes = 4096 > > > > log.index.size.max.bytes = 10485760 > > > > log.message.downconversion.enable = true > > > > log.message.format.version = 2.3-IV1 > > > > log.message.timestamp.difference.max.ms = > > 9223372036854775807 > > > > log.message.timestamp.type = CreateTime > > > > log.preallocate = false > > > > log.retention.bytes = 1073741824 > > > > log.retention.check.interval.ms = 300000 > > > > log.retention.hours = 168 > > > > log.retention.minutes = null > > > > log.retention.ms = null > > > > log.roll.hours = 168 > > > > log.roll.jitter.hours = 0 > > > > log.roll.jitter.ms = null > > > > log.roll.ms = null > > > > log.segment.bytes = 1073741824 > > > > log.segment.delete.delay.ms = 60000 > > > > max.connections = 2147483647 > > > > max.connections.per.ip = 2147483647 > > > > max.connections.per.ip.overrides = > > > > max.incremental.fetch.session.cache.slots = 1000 > > > > message.max.bytes = 1000012 > > > > metric.reporters = [] > > > > metrics.num.samples = 2 > > > > metrics.recording.level = INFO > > > > metrics.sample.window.ms = 30000 > > > > min.insync.replicas = 1 > > > > num.io.threads = 8 > > > > num.network.threads = 3 > > > > num.partitions = 1 > > > > num.recovery.threads.per.data.dir = 1 > > > > num.replica.alter.log.dirs.threads = null > > > > num.replica.fetchers = 1 > > > > offset.metadata.max.bytes = 4096 > > > > offsets.commit.required.acks = -1 > > > > offsets.commit.timeout.ms = 5000 > > > > offsets.load.buffer.size = 5242880 > > > > offsets.retention.check.interval.ms = 600000 > > > > offsets.retention.minutes = 10080 > > > > offsets.topic.compression.codec = 0 > > > > offsets.topic.num.partitions = 50 > > > > offsets.topic.replication.factor = 2 > > > > offsets.topic.segment.bytes = 104857600 > > > > password.encoder.cipher.algorithm = > > AES/CBC/PKCS5Padding > > > > password.encoder.iterations = 4096 > > > > password.encoder.key.length = 128 > > > > password.encoder.keyfactory.algorithm = null > > > > password.encoder.old.secret = null > > > > password.encoder.secret = null > > > > port = 9092 > > > > principal.builder.class = null > > > > producer.purgatory.purge.interval.requests = 1000 > > > > queued.max.request.bytes = -1 > > > > queued.max.requests = 500 > > > > quota.consumer.default = 9223372036854775807 > > > > quota.producer.default = 9223372036854775807 > > > > quota.window.num = 11 > > > > quota.window.size.seconds = 1 > > > > replica.fetch.backoff.ms = 1000 > > > > replica.fetch.max.bytes = 1048576 > > > > replica.fetch.min.bytes = 1 > > > > replica.fetch.response.max.bytes = 10485760 > > > > replica.fetch.wait.max.ms = 500 > > > > replica.high.watermark.checkpoint.interval.ms = 5000 > > > > replica.lag.time.max.ms = 10000 > > > > replica.socket.receive.buffer.bytes = 65536 > > > > replica.socket.timeout.ms = 30000 > > > > replication.quota.window.num = 11 > > > > replication.quota.window.size.seconds = 1 > > > > request.timeout.ms = 30000 > > > > reserved.broker.max.id = 1000 > > > > sasl.client.callback.handler.class = null > > > > sasl.enabled.mechanisms = [GSSAPI] > > > > sasl.jaas.config = null > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit > > > > sasl.kerberos.min.time.before.relogin = 60000 > > > > sasl.kerberos.principal.to.local.rules = [DEFAULT] > > > > sasl.kerberos.service.name = null > > > > sasl.kerberos.ticket.renew.jitter = 0.05 > > > > sasl.kerberos.ticket.renew.window.factor = 0.8 > > > > sasl.login.callback.handler.class = null > > > > sasl.login.class = null > > > > sasl.login.refresh.buffer.seconds = 300 > > > > sasl.login.refresh.min.period.seconds = 60 > > > > sasl.login.refresh.window.factor = 0.8 > > > > sasl.login.refresh.window.jitter = 0.05 > > > > sasl.mechanism.inter.broker.protocol = GSSAPI > > > > sasl.server.callback.handler.class = null > > > > security.inter.broker.protocol = PLAINTEXT > > > > socket.receive.buffer.bytes = 102400 > > > > socket.request.max.bytes = 104857600 > > > > socket.send.buffer.bytes = 102400 > > > > ssl.cipher.suites = [] > > > > ssl.client.auth = none > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > > > > ssl.endpoint.identification.algorithm = https > > > > ssl.key.password = null > > > > ssl.keymanager.algorithm = SunX509 > > > > ssl.keystore.location = null > > > > ssl.keystore.password = null > > > > ssl.keystore.type = JKS > > > > ssl.principal.mapping.rules = [DEFAULT] > > > > ssl.protocol = TLS > > > > ssl.provider = null > > > > ssl.secure.random.implementation = null > > > > ssl.trustmanager.algorithm = PKIX > > > > ssl.truststore.location = null > > > > ssl.truststore.password = null > > > > ssl.truststore.type = JKS > > > > > > transaction.abort.timed.out.transaction.cleanup.interval.ms > > = 60000 > > > > transaction.max.timeout.ms = 900000 > > > > > > transaction.remove.expired.transaction.cleanup.interval.ms > > = > > 3600000 > > > > transaction.state.log.load.buffer.size = 5242880 > > > > transaction.state.log.min.isr = 2 > > > > transaction.state.log.num.partitions = 50 > > > > transaction.state.log.replication.factor = 2 > > > > transaction.state.log.segment.bytes = 104857600 > > > > transactional.id.expiration.ms = 604800000 > > > > unclean.leader.election.enable = false > > > > zookeeper.connect = kafka-zookeeper > > > > zookeeper.connection.timeout.ms = 6000 > > > > zookeeper.max.in.flight.requests = 10 > > > > zookeeper.session.timeout.ms = 6000 > > > > zookeeper.set.acl = false > > > > zookeeper.sync.time.ms = 2000 > > > > (kafka.server.KafkaConfig) > > > > [2019-11-16 19:15:56,039] INFO [ThrottledChannelReaper-Fetch]: > > Starting > > (kafka.server.ClientQuotaManager$ThrottledChannelReaper) > > > > [2019-11-16 19:15:56,044] INFO [ThrottledChannelReaper-Produce]: > > Starting > > (kafka.server.ClientQuotaManager$ThrottledChannelReaper) > > > > [2019-11-16 19:15:56,046] INFO [ThrottledChannelReaper-Request]: > > Starting > > (kafka.server.ClientQuotaManager$ThrottledChannelReaper) > > > > [2019-11-16 19:15:56,335] INFO Loading logs. (kafka.log.LogManager) > > > > [2019-11-16 19:15:56,638] INFO [Log partition=__consumer_offsets-4, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:56,727] INFO [Log partition=__consumer_offsets-4, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:56,931] INFO [Log partition=__consumer_offsets-4, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:56,933] INFO [Log partition=__consumer_offsets-4, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 399 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,029] INFO [Log partition=__consumer_offsets-22, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,029] INFO [Log partition=__consumer_offsets-22, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,031] INFO [Log partition=__consumer_offsets-22, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,032] INFO [Log partition=__consumer_offsets-22, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 6 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,147] INFO [Log partition=__consumer_offsets-32, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,148] INFO [Log partition=__consumer_offsets-32, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,150] INFO [Log partition=__consumer_offsets-32, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,226] INFO [Log partition=__consumer_offsets-32, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 189 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,330] INFO [Log partition=__consumer_offsets-39, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,330] INFO [Log partition=__consumer_offsets-39, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,333] INFO [Log partition=__consumer_offsets-39, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,334] INFO [Log partition=__consumer_offsets-39, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 6 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,429] INFO [Log partition=__consumer_offsets-26, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,429] INFO [Log partition=__consumer_offsets-26, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,431] INFO [Log partition=__consumer_offsets-26, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,432] INFO [Log partition=__consumer_offsets-26, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 5 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,527] INFO [Log partition=__consumer_offsets-44, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,529] INFO [Log partition=__consumer_offsets-44, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,533] INFO [Log partition=__consumer_offsets-44, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,534] INFO [Log partition=__consumer_offsets-44, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 8 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,634] INFO [Log partition=__consumer_offsets-25, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,635] INFO [Log partition=__consumer_offsets-25, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,637] INFO [Log partition=__consumer_offsets-25, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,638] INFO [Log partition=__consumer_offsets-25, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 7 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,730] INFO [Log partition=__consumer_offsets-8, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,730] INFO [Log partition=__consumer_offsets-8, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,733] INFO [Log partition=__consumer_offsets-8, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,733] INFO [Log partition=__consumer_offsets-8, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 5 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,741] INFO [Log partition=batch.alarm-0, > > dir=/bitnami/kafka/data] Recovering unflushed segment 0 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,826] INFO [Log partition=batch.alarm-0, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,830] INFO [Log partition=batch.alarm-0, > > dir=/bitnami/kafka/data] Loading producer state till offset 0 with > > message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,833] INFO [Log partition=batch.alarm-0, > > dir=/bitnami/kafka/data] Completed load of log with 1 segments, log > > start offset 0 and log end offset 0 in 94 ms (kafka.log.Log) > > > > [2019-11-16 19:15:57,936] INFO [Log partition=__consumer_offsets-38, > > dir=/bitnami/kafka/data] Recovering unflushed segment 33982499 > > (kafka.log.Log) > > > > [2019-11-16 19:15:57,937] INFO [Log partition=__consumer_offsets-38, > > dir=/bitnami/kafka/data] Loading producer state till offset 33982499 > > with message format version 2 (kafka.log.Log) > > > > [2019-11-16 19:15:57,941] INFO [ProducerStateManager > > partition=__consumer_offsets-38] Loading producer state from snapshot > > file > '/bitnami/kafka/data/__consumer_offsets-38/00000000000033982499.snapshot' > > (kafka.log.ProducerStateManager) > > > > [2019-11-16 19:16:10,208] INFO Terminating process due to signal > > SIGTERM > > (org.apache.kafka.common.utils.LoggingSignalHandler) > > > > [2019-11-16 19:16:10,217] INFO [KafkaServer id=1012] shutting down > > (kafka.server.KafkaServer) > > > > [2019-11-16 19:16:10,226] ERROR [KafkaServer id=1012] Fatal error > > during KafkaServer shutdown. (kafka.server.KafkaServer) > > > > java.lang.IllegalStateException: Kafka server is still starting up, > > cannot shut down! > > > > at > > kafka.server.KafkaServer.shutdown(KafkaServer.scala:584) > > > > at > > kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala: > > 48) > > > > at kafka.Kafka$$anon$1.run(Kafka.scala:81) > > > > [2019-11-16 19:16:10,233] ERROR Halting Kafka. > > (kafka.server.KafkaServerStartable) > > > > > > > > Kind Regards > > > > Oliver > > > > > > >
