[jira] [Commented] (KAFKA-6817) UnknownProducerIdException when writing messages with old timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521948#comment-16521948
]
Wouter Bancken commented on KAFKA-6817:
---
Is there any news on a solution/workaround for clients not using Kafka Streams?
We have tried setting transactional.id.expiration.ms to its maximum value (24
days) as mentioned earlier in the comments and this greatly reduces the number
of occurrences but it does not eliminate the error.
In our use cases it is possible that a producer will remain active for a long
period without writing to a certain topic.
> UnknownProducerIdException when writing messages with old timestamps
>
>
> Key: KAFKA-6817
> URL: https://issues.apache.org/jira/browse/KAFKA-6817
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 1.1.0
>Reporter: Odin Standal
>Priority: Major
>
> We are seeing the following exception in our Kafka application:
> {code:java}
> ERROR o.a.k.s.p.internals.StreamTask - task [0_0] Failed to close producer
> due to the following error: org.apache.kafka.streams.errors.StreamsException:
> task [0_0] Abort sending since an error caught with a previous record (key
> 22 value some-value timestamp 1519200902670) to topic
> exactly-once-test-topic- v2 due to This exception is raised by the broker if
> it could not locate the producer metadata associated with the producerId in
> question. This could happen if, for instance, the producer's records were
> deleted because their retention time had elapsed. Once the last records of
> the producerId are removed, the producer's metadata is removed from the
> broker, and future appends by the producer will return this exception. at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:125)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.access$500(RecordCollectorImpl.java:48)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:180)
> at
> org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1199)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596)
> at
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557)
> at
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481)
> at
> org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
> at
> org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692)
> at
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
> at
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474) at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at
> java.lang.Thread.run(Thread.java:748) Caused by:
> org.apache.kafka.common.errors.UnknownProducerIdException
> {code}
> We discovered this error when we had the need to reprocess old messages. See
> more details on
> [Stackoverflow|https://stackoverflow.com/questions/49872827/unknownproduceridexception-in-kafka-streams-when-enabling-exactly-once?noredirect=1#comment86901917_49872827]
> We have reproduced the error with a smaller example application. The error
> occurs after 10 minutes of producing messages that have old timestamps (type
> 1 year old). The topic we are writing to has a retention.ms set to 1 year so
> we are expecting the messages to stay there.
> After digging through the ProducerStateManager-code in the Kafka source code
> we have a theory of what might be wrong.
> The ProducerStateManager.removeExpiredProducers() seems to remove producers
> from memory erroneously when processing records which are older than the
> maxProducerIdExpirationMs (coming from the `transactional.id.expiration.ms`
> configuration), which is set by default to 7 days.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6817) UnknownProducerIdException when writing messages with old timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460142#comment-16460142
]
Wouter Bancken edited comment on KAFKA-6817 at 5/1/18 8:56 PM:
---
We seem to be running into a similar issue.
We are using one Kafka Producer which is a long-running bean with idempotence
enabled.
The exception that we are getting is
{code}
kafka-producer-network-thread | producer-1] ERROR
apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1]
The broker returned org.apache.kafka.common.errors.UnknownProducerIdException:
This exception is raised by the broker if it could not locate the producer
metadata associated with the producerId in question. This could happen if, for
instance, the producer's records were deleted because their retention time had
elapsed. Once the last records of the producerId are removed, the producer's
metadata is removed from the broker, and future appends by the producer will
return this exception. for topic-partition company.company-alias-0 at offset
-1. This indicates data loss on the broker, and should be investigated.
Caused by: org.apache.kafka.common.errors.UnknownProducerIdException: This
exception is raised by the broker if it could not locate the producer metadata
associated with the producerId in question. This could happen if, for instance,
the producer's records were deleted because their retention time had elapsed.
Once the last records of the producerId are removed, the producer's metadata is
removed from the broker, and future appends by the producer will return this
exception.
{code}
We have no knowledge of issues on the broker that might have caused any data
loss. We are using Kafka 1.0.0
was (Author: wouterbanckenaca):
We seem to be running into a similar issue.
We are using one Kafka Producer which is a long-running bean with idempotence
enabled.
The exception that we are getting is
{code}
kafka-producer-network-thread | producer-1] ERROR
apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1]
The broker returned org.apache.kafka.common.errors.UnknownProducerIdException:
This exception is raised by the broker if it could not locate the producer
metadata associated with the producerId in question. This could happen if, for
instance, the producer's records were deleted because their retention time had
elapsed. Once the last records of the producerId are removed, the producer's
metadata is removed from the broker, and future appends by the producer will
return this exception. for topic-partition company.company-alias-0 at offset
-1. This indicates data loss on the broker, and should be investigated.
Caused by: org.apache.kafka.common.errors.UnknownProducerIdException: This
exception is raised by the broker if it could not locate the producer metadata
associated with the producerId in question. This could happen if, for instance,
the producer's records were deleted because their retention time had elapsed.
Once the last records of the producerId are removed, the producer's metadata is
removed from the broker, and future appends by the producer will return this
exception.
{code}
We have no knowledge of issues on the broker that might have caused any data
loss.
> UnknownProducerIdException when writing messages with old timestamps
>
>
> Key: KAFKA-6817
> URL: https://issues.apache.org/jira/browse/KAFKA-6817
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 1.1.0
>Reporter: Odin Standal
>Priority: Major
>
> We are seeing the following exception in our Kafka application:
> {code:java}
> ERROR o.a.k.s.p.internals.StreamTask - task [0_0] Failed to close producer
> due to the following error: org.apache.kafka.streams.errors.StreamsException:
> task [0_0] Abort sending since an error caught with a previous record (key
> 22 value some-value timestamp 1519200902670) to topic
> exactly-once-test-topic- v2 due to This exception is raised by the broker if
> it could not locate the producer metadata associated with the producerId in
> question. This could happen if, for instance, the producer's records were
> deleted because their retention time had elapsed. Once the last records of
> the producerId are removed, the producer's metadata is removed from the
> broker, and future appends by the producer will return this exception. at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:125)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.access$500(RecordCollectorImpl.java:48)
> at
>
[jira] [Commented] (KAFKA-6817) UnknownProducerIdException when writing messages with old timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460142#comment-16460142
]
Wouter Bancken commented on KAFKA-6817:
---
We seem to be running into a similar issue.
We are using one Kafka Producer which is a long-running bean with idempotence
enabled.
The exception that we are getting is
{code}
kafka-producer-network-thread | producer-1] ERROR
apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1]
The broker returned org.apache.kafka.common.errors.UnknownProducerIdException:
This exception is raised by the broker if it could not locate the producer
metadata associated with the producerId in question. This could happen if, for
instance, the producer's records were deleted because their retention time had
elapsed. Once the last records of the producerId are removed, the producer's
metadata is removed from the broker, and future appends by the producer will
return this exception. for topic-partition company.company-alias-0 at offset
-1. This indicates data loss on the broker, and should be investigated.
Caused by: org.apache.kafka.common.errors.UnknownProducerIdException: This
exception is raised by the broker if it could not locate the producer metadata
associated with the producerId in question. This could happen if, for instance,
the producer's records were deleted because their retention time had elapsed.
Once the last records of the producerId are removed, the producer's metadata is
removed from the broker, and future appends by the producer will return this
exception.
{code}
We have no knowledge of issues on the broker that might have caused any data
loss.
> UnknownProducerIdException when writing messages with old timestamps
>
>
> Key: KAFKA-6817
> URL: https://issues.apache.org/jira/browse/KAFKA-6817
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Affects Versions: 1.1.0
>Reporter: Odin Standal
>Priority: Major
>
> We are seeing the following exception in our Kafka application:
> {code:java}
> ERROR o.a.k.s.p.internals.StreamTask - task [0_0] Failed to close producer
> due to the following error: org.apache.kafka.streams.errors.StreamsException:
> task [0_0] Abort sending since an error caught with a previous record (key
> 22 value some-value timestamp 1519200902670) to topic
> exactly-once-test-topic- v2 due to This exception is raised by the broker if
> it could not locate the producer metadata associated with the producerId in
> question. This could happen if, for instance, the producer's records were
> deleted because their retention time had elapsed. Once the last records of
> the producerId are removed, the producer's metadata is removed from the
> broker, and future appends by the producer will return this exception. at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:125)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.access$500(RecordCollectorImpl.java:48)
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:180)
> at
> org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1199)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204)
> at
> org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627)
> at
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596)
> at
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557)
> at
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481)
> at
> org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
> at
> org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692)
> at
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
> at
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474) at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at
> java.lang.Thread.run(Thread.java:748) Caused by:
> org.apache.kafka.common.errors.UnknownProducerIdException
> {code}
> We discovered this error when we had the need to reprocess old messages. See
> more details on
>
[jira] [Updated] (KAFKA-6557) Kafka Consumer Offsets are unavailable during rebalancing
[ https://issues.apache.org/jira/browse/KAFKA-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wouter Bancken updated KAFKA-6557: -- Description: Using the kafka-consumer-groups tool to fetch the consumer offsets, the offsets seem to be unavailable during rebalancing. During the rebalancing, the consumer group has no partition assignments. *Scenario sketch* # I have a consumer group 'X' subscribed to topics 'A' and 'B' with partition assignments and lag information. Consumer group 'X' is "Stable". # a. Topic 'C' is (being) created. b. During this creation, I do not have a partition assignment for consumer group 'X' for topics 'A' and 'B' but the consumer group is still "Stable". # A second later: I have a partition assignment for consumer group 'X' for topics 'A' and 'B' again and the consumer group is still "Stable". I at least expected the state of consumer group 'X' during step 2b to be "PreparingRebalance" or "AwaitingSync". Otherwise there is no way to detect that the consumer is rebalancing. See the [discussion on the 'users' mailinglist|https://mail-archives.apache.org/mod_mbox/kafka-users/201801.mbox/%3CCAL5nZgS1im_AVVCdknLjLCFK4CkCEaqH9%3DqctgZJZKB%3Dt3hxwg%40mail.gmail.com%3E] for additional information. was: Using the kafka-consumer-groups tool to fetch the consumer offsets, the offsets seem to be unavailable during rebalancing. During the rebalancing, the consumer group has no partition assignments. *Scenario sketch* 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with partition assignments and lag information. Consumer group 'X' is "Stable". 2a. Topic 'C' is (being) created.2b. During this creation, I do not have a partition assignment for consumer group 'X' for topics 'A' and 'B' but the consumer group is still "Stable". 3. A second later: I have a partition assignment for consumer group 'X' for topics 'A' and 'B' again and the consumer group is still "Stable". I at least expected the state of consumer group 'X' during step 2b to be "PreparingRebalance" or "AwaitingSync". Otherwise there is no way to detect that the consumer is rebalancing. See the [discussion on the 'users' mailinglist|https://mail-archives.apache.org/mod_mbox/kafka-users/201801.mbox/%3CCAL5nZgS1im_AVVCdknLjLCFK4CkCEaqH9%3DqctgZJZKB%3Dt3hxwg%40mail.gmail.com%3E] for additional information. > Kafka Consumer Offsets are unavailable during rebalancing > - > > Key: KAFKA-6557 > URL: https://issues.apache.org/jira/browse/KAFKA-6557 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Wouter Bancken >Priority: Major > > Using the kafka-consumer-groups tool to fetch the consumer offsets, the > offsets seem to be unavailable during rebalancing. During the rebalancing, > the consumer group has no partition assignments. > > *Scenario sketch* > # I have a consumer group 'X' subscribed to topics 'A' and 'B' with > partition assignments and lag information. Consumer group 'X' is "Stable". > # a. Topic 'C' is (being) created. > b. During this creation, I do not have a partition assignment for consumer > group 'X' for topics 'A' and 'B' but the consumer group is still "Stable". > # A second later: I have a partition assignment for consumer group 'X' for > topics 'A' and 'B' again and the consumer group is still "Stable". > > I at least expected the state of consumer group 'X' during step 2b to be > "PreparingRebalance" or "AwaitingSync". Otherwise there is no way to detect > that the consumer is rebalancing. > See the [discussion on the 'users' > mailinglist|https://mail-archives.apache.org/mod_mbox/kafka-users/201801.mbox/%3CCAL5nZgS1im_AVVCdknLjLCFK4CkCEaqH9%3DqctgZJZKB%3Dt3hxwg%40mail.gmail.com%3E] > for additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6557) Kafka Consumer Offsets are unavailable during rebalancing
Wouter Bancken created KAFKA-6557: - Summary: Kafka Consumer Offsets are unavailable during rebalancing Key: KAFKA-6557 URL: https://issues.apache.org/jira/browse/KAFKA-6557 Project: Kafka Issue Type: Bug Affects Versions: 1.0.0 Reporter: Wouter Bancken Using the kafka-consumer-groups tool to fetch the consumer offsets, the offsets seem to be unavailable during rebalancing. During the rebalancing, the consumer group has no partition assignments. *Scenario sketch* 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with partition assignments and lag information. Consumer group 'X' is "Stable". 2a. Topic 'C' is (being) created.2b. During this creation, I do not have a partition assignment for consumer group 'X' for topics 'A' and 'B' but the consumer group is still "Stable". 3. A second later: I have a partition assignment for consumer group 'X' for topics 'A' and 'B' again and the consumer group is still "Stable". I at least expected the state of consumer group 'X' during step 2b to be "PreparingRebalance" or "AwaitingSync". Otherwise there is no way to detect that the consumer is rebalancing. See the [discussion on the 'users' mailinglist|https://mail-archives.apache.org/mod_mbox/kafka-users/201801.mbox/%3CCAL5nZgS1im_AVVCdknLjLCFK4CkCEaqH9%3DqctgZJZKB%3Dt3hxwg%40mail.gmail.com%3E] for additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
