Hello Pieter, If you only have one record every few seconds that may be too small given you have at least 25 partitions (as I saw you have a xxx--repartition-24 partition), which means that for a single partition, it may not see any records for a long time, and in this case you may need to override it to very large values. On the other hand, if you can reduce your num.partitions that may also help increasing the traffic per partition.
Also could you show me how did you override the configs in the repartition topics? Guozhang On Tue, Jun 4, 2019 at 2:10 AM Pieter Hameete <pieter.hame...@blockbax.com> wrote: > Hello, > > Our Kafka streams applications are showing the following warning every few > seconds (on each of our 3 brokers, and on each of the 2 instances of the > streams application): > > > [Producer > clientId=event-rule-engine-dd71ae9b-523c-425d-a7c0-c62993315b30-StreamThread-1-1_24-producer, > transactionalId=event-rule-engine-1_24] Resetting sequence number of batch > with current sequence 1 for partition > event-rule-engine-KSTREAM-REDUCE-STATE-STORE-0000000015-repartition-24 to 0 > > > > Followed by: > > > > [Producer > clientId=event-rule-engine-dd71ae9b-523c-425d-a7c0-c62993315b30-StreamThread-1-1_24-producer, > transactionalId=event-rule-engine-1_24] Got error produce response with > correlation id 5902 on topic-partition > event-rule-engine-KSTREAM-REDUCE-STATE-STORE-0000000015-repartition-24, > retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID > > The brokers are showing errors that look related: > > > Error processing append operation on partition > event-rule-engine-KSTREAM-REDUCE-STATE-STORE-0000000015-repartition-24 > (kafka.server.ReplicaManager) > > org.apache.kafka.common.errors.UnknownProducerIdException: Found no record > of producerId=72 on the broker. It is possible that the last message with > the producerId=72 has been removed due to hitting the retention limit. > > > > We would expect the UNKNOWN_PRODUCER_ID error to occur once. After a retry > the record would be published on the partition and the PRODUCER_ID would be > known. However, this error keeps occurring every few seconds. This is > roughly at the same rate at which records are produced on the input topics > partitions, so it seems like it occurs for (nearly) every input record. > > > > The following JIRA issue: https://issues.apache.org/jira/browse/KAFKA-7190 > looks related. Except the Jira issue mentions ‘little traffic’, and I am > not sure if a message per every few seconds is regarded as little traffic. > Matthias mentions in the issue that a workaround seems to be to increase > topic configs `segment.bytes`, `segment.index.bytes`, and `segment.ms` > for the corresponding repartition topics. We’ve tried manually overriding > these configs for a relevant topic to the config values in the linked pull > request (https://github.com/apache/kafka/pull/6511) but this did not > result in the errors disappearing. > > > > Could anyone help us to figure out what is happening here, and why the > proposed fix for the above JIRA issue is not working in this case? > > > > Best, > > > > Pieter > > -- -- Guozhang
