[jira] [Commented] (KAFKA-6541) StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread

2018-02-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366412#comment-16366412
 ] 

ASF GitHub Bot commented on KAFKA-6541:
---

hachikuji closed pull request #4548: KAFKA-6541: Fixed a StackOverflow bug in 
kafka-coordinator-heartbeat-thread
URL: https://github.com/apache/kafka/pull/4548
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 
b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
index 0747e8db146..a0d4ed8f57c 100644
--- 
a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
+++ 
b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
@@ -57,6 +57,7 @@
 // flag and the request completion queue below).
 private final Logger log;
 private final KafkaClient client;
+private boolean disconnecting = false;
 private final UnsentRequests unsent = new UnsentRequests();
 private final Metadata metadata;
 private final Time time;
@@ -387,9 +388,13 @@ private void checkDisconnects(long now) {
 }
 
 public void disconnect(Node node) {
-failUnsentRequests(node, DisconnectException.INSTANCE);
-
 synchronized (this) {
+if(disconnecting){
+return;
+}
+disconnecting=true;
+failUnsentRequests(node, DisconnectException.INSTANCE);
+
 client.disconnect(node.idString());
 }
 
@@ -483,6 +488,7 @@ public boolean connectionFailed(Node node) {
 public void tryConnect(Node node) {
 synchronized (this) {
 client.ready(node, time.milliseconds());
+disconnecting = false;
 }
 }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread
> --
>
> Key: KAFKA-6541
> URL: https://issues.apache.org/jira/browse/KAFKA-6541
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
> Environment: Linux
>Reporter: Anh Le
>Priority: Major
>
> There's something wrong with our client library when sending heart beats. 
> This bug seems to be identical to this one: 
> [http://mail-archives.apache.org/mod_mbox/kafka-users/201712.mbox/%3CCALte62w6=pJObC+i36BkoqbOLTKsQ=nrddv6dm8abfwb5ps...@mail.gmail.com%3E]
>  
> Here's the log:
>  
> {quote}2018-02-08 13:55:01,102 ERROR 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread
>  Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | 
> default-group':
> java.lang.StackOverflowError: null
>  at java.lang.StringBuilder.append(StringBuilder.java:136)
>  at 
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:302)
>  at 
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.getFormattedMessage(LoggingEvent.java:293)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.prepareForDeferredProcessing(LoggingEvent.java:206)
>  at 
> ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:223)
>  at 
> ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
>  at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
>  at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
>  at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
>  at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
>  at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
>  at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)
>  at ch.qos.logback.classic.Logger.info(Logger.java:583)
>  at 
> org.apache.kafka.common.utils.LogContext$KafkaLogger.info(LogContext.java:341)
>  at 
>

[jira] [Commented] (KAFKA-6541) StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread

2018-02-08 Thread Jason Gustafson (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357279#comment-16357279
 ] 

Jason Gustafson commented on KAFKA-6541:


[~anhldbk] This was fixed in KAFKA-6366 which will be included in 1.0.1. The 
current RC is available here: 
[http://home.apache.org/~ewencp/kafka-1.0.1-rc0/]. If you have time to confirm 
the fix and close this JIRA, we would appreciate it.

> StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread
> --
>
> Key: KAFKA-6541
> URL: https://issues.apache.org/jira/browse/KAFKA-6541
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
> Environment: Linux
>Reporter: Anh Le
>Priority: Major
>
> There's something wrong with our client library when sending heart beats. 
> This bug seems to be identical to this one: 
> [http://mail-archives.apache.org/mod_mbox/kafka-users/201712.mbox/%3CCALte62w6=pJObC+i36BkoqbOLTKsQ=nrddv6dm8abfwb5ps...@mail.gmail.com%3E]
>  
> Here's the log:
>  
> {quote}2018-02-08 13:55:01,102 ERROR 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread
>  Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | 
> default-group':
> java.lang.StackOverflowError: null
>  at java.lang.StringBuilder.append(StringBuilder.java:136)
>  at 
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:302)
>  at 
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.getFormattedMessage(LoggingEvent.java:293)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.prepareForDeferredProcessing(LoggingEvent.java:206)
>  at 
> ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:223)
>  at 
> ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
>  at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
>  at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
>  at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
>  at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
>  at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
>  at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)
>  at ch.qos.logback.classic.Logger.info(Logger.java:583)
>  at 
> org.apache.kafka.common.utils.LogContext$KafkaLogger.info(LogContext.java:341)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:649)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.failUnsentRequests(ConsumerNetworkClient.java:416)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disconnect(ConsumerNetworkClient.java:388)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:653)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.failUnsentRequests(ConsumerNetworkClient.java:416)
>  at 
>

[jira] [Commented] (KAFKA-6541) StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread

2018-02-08 Thread Anh Le (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356819#comment-16356819
 ] 

Anh Le commented on KAFKA-6541:
---

[~huxi_2b] I created [a PR|https://github.com/apache/kafka/pull/4548]  for this 
issue. Please have a look at it.

> StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread
> --
>
> Key: KAFKA-6541
> URL: https://issues.apache.org/jira/browse/KAFKA-6541
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
> Environment: Linux
>Reporter: Anh Le
>Priority: Major
>
> There's something wrong with our client library when sending heart beats. 
> This bug seems to be identical to this one: 
> [http://mail-archives.apache.org/mod_mbox/kafka-users/201712.mbox/%3CCALte62w6=pJObC+i36BkoqbOLTKsQ=nrddv6dm8abfwb5ps...@mail.gmail.com%3E]
>  
> Here's the log:
>  
> {quote}2018-02-08 13:55:01,102 ERROR 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread
>  Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | 
> default-group':
> java.lang.StackOverflowError: null
>  at java.lang.StringBuilder.append(StringBuilder.java:136)
>  at 
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:302)
>  at 
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.getFormattedMessage(LoggingEvent.java:293)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.prepareForDeferredProcessing(LoggingEvent.java:206)
>  at 
> ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:223)
>  at 
> ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
>  at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
>  at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
>  at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
>  at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
>  at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
>  at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)
>  at ch.qos.logback.classic.Logger.info(Logger.java:583)
>  at 
> org.apache.kafka.common.utils.LogContext$KafkaLogger.info(LogContext.java:341)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:649)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.failUnsentRequests(ConsumerNetworkClient.java:416)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disconnect(ConsumerNetworkClient.java:388)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:653)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.failUnsentRequests(ConsumerNetworkClient.java:416)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disconnect(ConsumerNetworkClient.java:388)
>  at 
>

[jira] [Commented] (KAFKA-6541) StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread

2018-02-08 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356818#comment-16356818
 ] 

ASF GitHub Bot commented on KAFKA-6541:
---

anhldbk opened a new pull request #4548: KAFKA-6541: Fix a StackOverflow bug in 
kafka-coordinator-heartbeat-thread
URL: https://github.com/apache/kafka/pull/4548
 
 
   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   - The bug is caused by infinite recursive calls to 
`ConsumerNetworkClient.disconnect()` 
   
   - In class 
`org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient`, I 
introduced a boolean field named `disconnecting` to prevent such calls happen
   
   ```java
   public void disconnect(Node node) {
   synchronized (this) {
   if(disconnecting){
   return;
   }
   disconnecting=true;
   failUnsentRequests(node, DisconnectException.INSTANCE);  
  
   
   client.disconnect(node.idString());
   }
   
   // We need to poll to ensure callbacks from in-flight requests on 
the disconnected socket are fired
   pollNoWakeup();
   }
   ```
   
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   No test to implement (I think this is a trivial fix)
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread
> --
>
> Key: KAFKA-6541
> URL: https://issues.apache.org/jira/browse/KAFKA-6541
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
> Environment: Linux
>Reporter: Anh Le
>Priority: Major
>
> There's something wrong with our client library when sending heart beats. 
> This bug seems to be identical to this one: 
> [http://mail-archives.apache.org/mod_mbox/kafka-users/201712.mbox/%3CCALte62w6=pJObC+i36BkoqbOLTKsQ=nrddv6dm8abfwb5ps...@mail.gmail.com%3E]
>  
> Here's the log:
>  
> {quote}2018-02-08 13:55:01,102 ERROR 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread
>  Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | 
> default-group':
> java.lang.StackOverflowError: null
>  at java.lang.StringBuilder.append(StringBuilder.java:136)
>  at 
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:302)
>  at 
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>  at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.getFormattedMessage(LoggingEvent.java:293)
>  at 
> ch.qos.logback.classic.spi.LoggingEvent.prepareForDeferredProcessing(LoggingEvent.java:206)
>  at 
> ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:223)
>  at 
> ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
>  at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
>  at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
>  at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
>  at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
>  at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
>  at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)
>  at ch.qos.logback.classic.Logger.info(Logger.java:583)
>  at 
> org.apache.kafka.common.utils.LogContext$KafkaLogger.info(LogContext.java:341)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:649)
>  at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)
>  at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
>  at 
>

[jira] [Commented] (KAFKA-6541) StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread

2018-02-08 Thread huxihx (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356756#comment-16356756
 ] 

huxihx commented on KAFKA-6541:
---

Seems `coordinatorDead` was recursively called  which could be proved by the 
repeating pattern of line numbers (797 -> 653 -> 388 -> 416 -> ... -> 797 -> 
653 -> ...)

> StackOverflow exceptions in thread 'kafka-coordinator-heartbeat-thread
> --
>
> Key: KAFKA-6541
> URL: https://issues.apache.org/jira/browse/KAFKA-6541
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
> Environment: Linux
>Reporter: Anh Le
>Priority: Major
>
> There's something wrong with our client library when sending heart beats. 
> This bug seems to be identical to this one: 
> [http://mail-archives.apache.org/mod_mbox/kafka-users/201712.mbox/%3CCALte62w6=pJObC+i36BkoqbOLTKsQ=nrddv6dm8abfwb5ps...@mail.gmail.com%3E]
>  
> Here's the log:
>  
> {{2018-02-08 13:55:01,102 ERROR 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread
>  Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | 
> default-group':}}
> {{java.lang.StackOverflowError: null}}
> {{ at java.lang.StringBuilder.append(StringBuilder.java:136)}}
> {{ at 
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:302)}}
> {{ at 
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)}}
> {{ at 
> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)}}
> {{ at 
> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)}}
> {{ at 
> ch.qos.logback.classic.spi.LoggingEvent.getFormattedMessage(LoggingEvent.java:293)}}
> {{ at 
> ch.qos.logback.classic.spi.LoggingEvent.prepareForDeferredProcessing(LoggingEvent.java:206)}}
> {{ at 
> ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:223)}}
> {{ at 
> ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)}}
> {{ at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)}}
> {{ at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)}}
> {{ at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)}}
> {{ at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)}}
> {{ at 
> ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)}}
> {{ at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)}}
> {{ at ch.qos.logback.classic.Logger.info(Logger.java:583)}}
> {{ at 
> org.apache.kafka.common.utils.LogContext$KafkaLogger.info(LogContext.java:341)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:649)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.failUnsentRequests(ConsumerNetworkClient.java:416)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disconnect(ConsumerNetworkClient.java:388)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead(AbstractCoordinator.java:653)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onFailure(AbstractCoordinator.java:797)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:496)}}
> {{ at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:353)}}
> {{ at 
>