[
https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586886#comment-16586886
]
lambdaliu commented on KAFKA-7190:
----------------------------------
Hi [~hachikuji], sorry for the later reply.
I have thought about the solution you suggested last time, and found it's not
easy to reset sequence to 0 for each transaction. Because the broker may
receive a new transactional produce request before the last complete
transaction's EndTxnMaker request. So we would better cache all PID in memory
until it expired. For the idea of soft delete records which beyond the LSO,
it's easy to implement. But the deletion of retention time and retention size
breach may still delete the segment which contain active transaction. For this
case, maybe we can use snapshot file to save the active PID. With this change
we can always recover PID which have active transaction from log.
For the KIP you posted seems great and I am glad to work on it. Thanks.
> Under low traffic conditions purging repartition topics cause WARN statements
> about UNKNOWN_PRODUCER_ID
> ---------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7190
> URL: https://issues.apache.org/jira/browse/KAFKA-7190
> Project: Kafka
> Issue Type: Improvement
> Components: core, streams
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Bill Bejeck
> Assignee: lambdaliu
> Priority: Major
>
> When a streams application has little traffic, then it is possible that
> consumer purging would delete
> even the last message sent by a producer (i.e., all the messages sent by
> this producer have been consumed and committed), and as a result, the broker
> would delete that producer's ID. The next time when this producer tries to
> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case,
> this error is retriable: the producer would just get a new producer id and
> retries, and then this time it will succeed.
>
> Possible fixes could be on the broker side, i.e., delaying the deletion of
> the produderIDs for a more extended period or on the streams side developing
> a more conservative approach to deleting offsets from repartition topics
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
