Occasional Warnings in logs
Hi, I've a spring boot kafka consumer which consumes messages from a kafka cluster which also produces a message to another cluster as part of consuming the message. I get the below errors periodically. I have auto retries enabled and it looks like these messages are retried successfully but can't really find anything definitive online if there's anything I can do to prevent / or reduce these errors. (Once an hour maybe less) Is this just the producer connection timing out due to inactivity? Log Entries 2023-08-27T14:10:17.899Z WARN 25650 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-1] Got error produce response with correlation id 343740 on topic-partition MDM_CSTMR_LYLTY_DTLS-9, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 119 2023-08-27T14:10:17.899Z WARN 25650 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-1] Received invalid metadata error in produce request on partition MDM_CSTMR_LYLTY_DTLS-9 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 119. Going to request metadata update now Cheers Lee C
Re: Consumer has high network bandwidth when consuming no messages
Hi Kirk, Points 1 and 2: yes we have both logging and metrics. I'll collate Point 3 concurrency refers to the spring boot kafka concurrent container. 2 threads will be made available to the consumer group to consume 30 partitions making up a single topic. point 4: consumer- Kafka Consumer Configurations for Confluent Platform | Confluent Documentation <https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#fetch-max-wait-ms> On Thu, 13 Apr 2023 at 16:03, Kirk True wrote: > Hi Lee, > > Some questions: > > 1. Can you enable metrics for your consumer(s)? See [1] and [2] for links > on consumer-level and fetch-level metrics, respectively. > 2. Can you enable more detailed logging? > 3. What do you mean specifically by “concurrency of 2?” > 4. Can you verify the name of the fetch.wait.max.ms configuration? I > don’t see that configuration option. Is that a broker configuration or a > consumer configuration? > > Thanks, > Kirk > > [1] https://kafka.apache.org/documentation/#consumer_group_monitoring > [2] https://kafka.apache.org/documentation/#consumer_fetch_monitoring > > > On Apr 13, 2023, at 4:52 AM, Lee Carroll > > > wrote: > > > > Hi All, > > I've a spring boot kafka consumer group, consuming a topic with 30 > > partitions with a concurrency of 2 (staging set up). > > > > The network traffic in and out was very high >200k per second even when > > consuming no messages. > > > > I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than > > the default which has had a significant impact but the network is still > > seeing ~30k a second. > > > > Is this to be expected? > > > > Cheers Lee C > >
Consumer has high network bandwidth when consuming no messages
Hi All, I've a spring boot kafka consumer group, consuming a topic with 30 partitions with a concurrency of 2 (staging set up). The network traffic in and out was very high >200k per second even when consuming no messages. I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than the default which has had a significant impact but the network is still seeing ~30k a second. Is this to be expected? Cheers Lee C
kafka producer as an aws lambda function vulnerable to dropped messages
Hi All,
I've been experimenting with a Kafka producer implemented as an AWS lambda
function and intial tests have shown potential issues with this approach.
I'm new to Kafka so I may have made some basics errors.
I've searched across the mailing list but nothing related is coming up so
hopefully this question is not an obvious duplicate.
In short is running kafka as a lambda producer a good idea? Or more
generally running a producer as a FAAS a good idea. I'm thinking possibly
not given the below experience.
My set up
I have a producer p which has a dependency of a configured producer.
public class WebEventProducer implements
RequestHandler {
private Producer producer ;
These dependencies are configured using example set up during construction with
bootstrap.servers = one-of-my-brokers:9093
acks = all
retries = 0
batch.size = 16384
linger.ms = 1
timeout.ms = 1000
request.timeout.ms = 1000
metadata.fetch.timeout.ms = 2000
buffer.memory = 33554432
key.serializer = org.apache.kafka.common.serialization.StringSerializer
value.serializer = org.apache.kafka.common.serialization.StringSerializer
I async send messages and I do very little in the callback. Producer
#close or producer #flush
are not called.
What I observe:
Under load the producer sends messages very nicely thank you very
much. I'm interpreting this as the function container is persisted and
the handler
function is reused. The state held in the producer is reused and
nothing is lost. All good.
However if load stops I observe two things.
1. a message can be held in the producer buffer without being sent. In
order for the message to be sent further messages must be processed.
2. If subsequent messages are not sent the functional container gets
tore down after x amount of time and the message is lost.
*Now I know I'm cheating a little by having state in my
**dependencies** but in the main it works very well. However are the
above edge cases fatal. I'm thinking*
*that maybe I can get round this issue by calling #flush in the call
back but is this good practice? Also I simply don't have any ability
to call #close*
*as I don't appear to have a decent hook. Does this matter? Is kafka
tolerant of hung up producers, will they be cleaned up?*
*Lastly under very heavy load the FAAS container could spin up a 1000
containers all running an instance of my producer (that is extreme and
we'd never get their*
*but it is in theory possible). Is it okay to have hundreds of
producers all producing the same topic?*
*Really sorry for the long question and the rambling information...*
*Lee C*
