Re: Customers are getting same emails for roughly 30-40 times

Hi, You can check Consumer Api https://kafka.apache.org/10/javadoc/?org/apache/kafka/clients/consumer/KafkaConsumer.html . Refer : Manual Offset Control --Senthil On Sat, May 25, 2019, 9:53 AM ASHOK MACHERLA wrote: > Dear Hans > > Thanks for you reply > > As you said we are getting same

Re: Customers are getting same emails for roughly 30-40 times

Dear Hans Thanks for you reply As you said we are getting same issue, our consumers some times goes to rebalance mode, during this time customer getting duplicate emails. So, How to set manual commit offsets?? Is there any parameters to add for that. Please reply this email Sent from Outlook

Re: Customers are getting same emails for roughly 30-40 times

Its not just the config, you need to change your code. kafka.auto.commit.interval.ms=3000 means that consumers only commit offsets every 3 seconds so if there is any failure or rebalance they will reconsume up to 3 seconds of data per partition. That could be many hundreds or thousands of

I have lots of WARN in log every day

I have lots of WARN in log every day.like these,what does means?[2019-05-24 11:05:06,784] WARN Attempting to send response via channel for which there is no open connection, connection id 2 (kafka.network.Processor) [2019-05-24 11:05:10,884] WARN Attempting to send response via channel for

Re: Stream application :: State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED continuously

Hello Nayanjyoti, Regarding the KIP-328, on-disk buffer is indeed being implemented but it has not been completed and unfortunately has to slip to the next release. Now about the "PARTITIONS_REVOKED to PARTITIONS_ASSIGNED" issue, it is possible that if you are restoring tons of data from the

Re: Streams configuration for a stream with long varied processing times

Hi Raman, Since you are using `transformation` already which is a lower-level API in DSL, you can basically do arbitrary logic like threading pool to process the records within your `process()` or `transform()` function. Note that, like consumer docs mentioned `Typically, you must disable

RE: Customers are getting same emails for roughly 30-40 times

Dear Team First of all thanks for reply on this issue. Right now we are using these configurations at consumer side kafka.max.poll.records=20 max.push.batch.size=100 enable.auto.commit=true auto.offset.reset=latest kafka.auto.commit.interval.ms=3000 kafka.session.timeout.ms=1

Re: Customers are getting same emails for roughly 30-40 times

It also seems you are using "at least one" strategy (maybe with auto-commit, or commiting after sending the email) Maybe a "at most once" could be a valid business strategy here ? - at least once (you will deliver all the emails, but you could deliver duplicates) consumeMessages sendEmails

Re: Customers are getting same emails for roughly 30-40 times

Consumers will rebalance if you add partitions, add consumers to the group or if a consumer leaves the group. Consumers will leave the group after not communicating with the server for a period set by session.timeout.ms. This is usually due to an exception in the code polling with the consumer,

Re: Restart of kafka-streams with more than one partition on topic is reprocessing the data from beginning of the topic

Hello Kalyani, try testing the RC kafka-2.2.1-rc1, for what you describe seems to be a problem that has been solved in the version 2.2.1 ( https://issues.apache.org/jira/browse/KAFKA-7895) (which is under voting right now 2.2.1-RC1

Customers are getting same emails for roughly 30-40 times

Dear Team Member Currently we are using Kafka 0.10.1, zookeeper 3.4.6 versions. In our project we have to send bulk emails to customers for this purpose we are using Kafka cluster setup. But customers are getting same emails for roughly 30-40 times. This is very worst thing. In this situation