[
https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556042#comment-16556042
]
Matthias J. Sax commented on KAFKA-7190:
----------------------------------------
> The main reason we need to prevent client from deleting beyond HW is because
>otherwise follower may receive OutOfRangeException
Isn't `OutOfRangeException` only a symptom? It seems that the actual issue is
potential data loss if uncommitted data get's deleted... Ie, the main reason to
not delete data is to avoid data loss?
I understand your second point that the `OutOfRangeException` should never be
triggered if the API is used correctly. However, I still think the change is
justified and it would be an additional "safety net" – of course, we should
document that uncommitted data cannot be deleted via purge data request. As an
enhancement, we should also introduce a "force" option to purge data to allow
deleting uncommitted data (not sure if we need this): for this case, we should
document that it might result in `OutOfRangeException` for downstream consumers
and should be used with care?
[~guozhang] [~lambdaliu] For keeping PID. Why do we need a very short "delay
time" until the PID can be purged? Why would 7 days be an issue? Might be
better to avoid additional configs if we can.
> Under low traffic conditions purging repartition topics cause WARN statements
> about UNKNOWN_PRODUCER_ID
> ---------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7190
> URL: https://issues.apache.org/jira/browse/KAFKA-7190
> Project: Kafka
> Issue Type: Improvement
> Components: core, streams
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Bill Bejeck
> Assignee: lambdaliu
> Priority: Major
>
> When a streams application has little traffic, then it is possible that
> consumer purging would delete
> even the last message sent by a producer (i.e., all the messages sent by
> this producer have been consumed and committed), and as a result, the broker
> would delete that producer's ID. The next time when this producer tries to
> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case,
> this error is retriable: the producer would just get a new producer id and
> retries, and then this time it will succeed.
>
> Possible fixes could be on the broker side, i.e., delaying the deletion of
> the produderIDs for a more extended period or on the streams side developing
> a more conservative approach to deleting offsets from repartition topics
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
