[
https://issues.apache.org/jira/browse/KAFKA-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477981#comment-17477981
]
Willian Dallastella edited comment on KAFKA-13597 at 1/18/22, 3:38 PM:
-----------------------------------------------------------------------
Hey [~showuon], so I changed the service again to use *kafka-clients*
{color:#de350b}*3.0.0*{color} (without any other change), I saw this exception
that was only happening once during startup:
{code:java}
java.lang.NoSuchMethodError: 'long
org.apache.kafka.clients.producer.RecordMetadata.checksum()'
at
io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor.onAcknowledgement(MonitoringProducerInterceptor.java:98)
at
org.apache.kafka.clients.producer.internals.ProducerInterceptors.onAcknowledgement(ProducerInterceptors.java:88)
at
org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1367)
at
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:270)
{code}
We were using {*}io.confluent:monitoring-interceptors
{color:#de350b}6.2.2{color}{*}, updating to *{color:#00875a}7.0.1{color}*
{color:#172b4d}fixed the above exception and also the memory leak.{color}
{color:#172b4d}The properties were not changed, I'm still using the one from
the description above.{color}
{color:#172b4d}After startup, we didn't have any other exceptions.{color}
{color:#172b4d}Considering that kafka-clients 3.0.0 should be ok and the issue
was some incompatibility between it and io.confluent:monitoring-interceptors.
It was hard to get the exception only happened during startup. {color}
{color:#172b4d}Shall I close this ticket?{color}
was (Author: JIRAUSER283709):
Hey [~showuon], so I changed the service again to use *kafka-clients*
{color:#de350b}*3.0.0*{color} (without any other change), I saw this exception
that was only happening once during startup:
{code:java}
java.lang.NoSuchMethodError: 'long
org.apache.kafka.clients.producer.RecordMetadata.checksum()'
at
io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor.onAcknowledgement(MonitoringProducerInterceptor.java:98)
at
org.apache.kafka.clients.producer.internals.ProducerInterceptors.onAcknowledgement(ProducerInterceptors.java:88)
at
org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1367)
at
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:270)
{code}
We were using {*}io.confluent:monitoring-interceptors
{color:#de350b}6.2.2{color}{*}, updating to *{color:#00875a}7.0.1{color}*
{color:#172b4d}fixed the above exception and also the memory leak.{color}
{color:#172b4d}The properties were not changed, I'm still using the one from
the description above.{color}
{color:#172b4d}After startup, we didn't have any other exceptions.{color}
{color:#172b4d}Considering that kafka-clients 3.0.0 should be ok and the issue
was some incompatibility between it and io.confluent:monitoring-interceptors.
It was hard to get the exception only happened during startup. Shall I close
this ticket?{color}
> Memory leak with kafka-clients 3.0.0
> ------------------------------------
>
> Key: KAFKA-13597
> URL: https://issues.apache.org/jira/browse/KAFKA-13597
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 3.0.0
> Reporter: Willian Dallastella
> Priority: Major
> Fix For: 3.0.1
>
> Attachments: image-2022-01-17-12-59-43-411.png,
> image-2022-01-17-13-01-43-418.png
>
>
> I'm having this issue reported here:
> [https://github.com/spring-projects/spring-kafka/issues/2056]
>
> It is a Spring Boot 2.5.7 application that started failing (out of memory)
> when updated to Spring Boot 2.6.1.
> This updated *kafka-clients {color:#00875a}2.7.1{color}* to
> *{color:#ff0000}3.0.0{color}* and started causing the memory leak.
>
> The service is using a standard spring *katkaTemplate* to send messages.
> {code:java}
> kafkaTemplate.send(
> topic,
> id,
> data
> ).addCallback(kafkaLoggerCallback) {code}
> This producer is quite heavy (~2.5k messages/s).
> My kafka config:
> {code:java}
> spring:
> kafka:
> bootstrap-servers:
> - my_kafka_host:9092
> properties:
> client.id: my_client_id
> ssl.endpoint.identification.algorithm: https
> request.timeout.ms: 20000
> retry.backoff.ms: 500
> sasl.jaas.config:
> org.apache.kafka.common.security.plain.PlainLoginModule required
> username="my_user" password="my_password";
> security.protocol: SASL_SSL
> sasl.mechanism: PLAIN
> interceptor.classes:
> io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
> confluent.monitoring.interceptor.sasl.mechanism: PLAIN
> confluent.monitoring.interceptor.security.protocol: SASL_SSL
> confluent.monitoring.interceptor.sasl.jaas.config:
> org.apache.kafka.common.security.plain.PlainLoginModule required
> username="my_user" password="my_password";
> schema.registry.basic.auth.user.info: sr_ccloud_key:sr_ccloud_key
> schema.registry.url: https://my_schema_registry
> ack: 1
> producer:
> key-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
> value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
> {code}
>
> *Downgrading* just the lib *kafka-clients* to *{color:#00875a}2.7.1{color}*
> or *{color:#00875a}2.8.1{color}* solves the issue.
> h2. Memory leak sample
> * The green line around 16:40 is the service using kafka-clients 2.7.1
> * The pink line around 16:55 is without downgrading, that means
> kafka-clients 3.0.0
> !image-2022-01-17-12-59-43-411.png|width=1017,height=303!
>
> h2. Profiling sample
>
> !image-2022-01-17-13-01-43-418.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
