GitHub user apurvam opened a pull request:
https://github.com/apache/kafka/pull/2735
KAFKA-4815 : Add idempotent producer semantics
This is from the KIP-98 proposal.
The main points of discussion surround the correctness logic, particularly
the Log class where incoming entries are validated and duplicates are dropped,
and also the producer error handling to ensure that the semantics are sound
from the users point of view.
There is some subtlety in the idempotent producer semantics. This patch
only guarantees idempotent production upto the point where an error has to be
returned to the user. Once we hit a such a non-recoverable error, we can no
longer guarantee message ordering nor idempotence without additional logic at
the application level.
In particular, if an application wants guaranteed message order without
duplicates, then it needs to do the following on the error callback:
# Close the producer so that no queued batches are sent. This is important
for guaranteeing ordering.
# Read the tail of the log to inspect the last message committed. This is
important for avoiding duplicates.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/confluentinc/kafka
exactly-once-idempotent-producer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2735.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2735
----
commit 83f5b2937d368efa768be6d7df1f57d99425a252
Author: fpj <[email protected]>
Date: 2016-10-10T16:19:52Z
Add broker producer id mapping support
commit 37eac83364a67eebf0e1d96552d9fe864baa3d76
Author: Guozhang Wang <[email protected]>
Date: 2017-02-14T19:12:58Z
KEOS: idempotent producer pid generation (#126)
Add the transaction coordinator for pid generation and management.
commit 6d918d0b44302aa81627f3f2748756356614f228
Author: hachikuji <[email protected]>
Date: 2017-02-23T18:59:46Z
Minor fixes for test failures and consistency (#133)
commit d1868602be8b3e81d98644d673a73b9f730ad586
Author: Jason Gustafson <[email protected]>
Date: 2017-03-01T22:40:30Z
Fix headers, new checkstyle rules, and some breakage in KafkaApis
commit f2c01a71508cd5c1eaf8c03a372f7dc308af3cf2
Author: hachikuji <[email protected]>
Date: 2017-03-07T00:23:11Z
Avoid removal of non-expired PIDs when log cleaning (#138)
commit 31ff08eaf704b9e4fe6ee492277d981b47e8016c
Author: Apurva Mehta <[email protected]>
Date: 2017-03-09T01:50:01Z
Client side implementation of the idempotent producer. (#129)
commit 439f284fd4efa56449d43f2fb5e14614689c7b80
Author: Apurva Mehta <[email protected]>
Date: 2017-03-09T21:23:55Z
Fix build errors due to rebase
commit 0a81a40258da29ed62caf745e7c1895ca7eead4c
Author: hachikuji <[email protected]>
Date: 2017-03-10T17:50:01Z
PIDs should be expired according to the transactional id expiration setting
(#139)
commit 2643e5b6bb6ad6b3d1cd6b15073a92396be5693b
Author: Apurva Mehta <[email protected]>
Date: 2017-03-15T18:45:07Z
Fix build errors due to rebase
commit 848137a28b02cd085605e92726d0457b0153e1c4
Author: Apurva Mehta <[email protected]>
Date: 2017-03-16T18:43:29Z
Remove depependence of MemoryRecordsBuilder on TransactionState
commit 15d23b65166cff6dbd7be0e925ff9ec811d7baac
Author: hachikuji <[email protected]>
Date: 2017-03-23T22:30:39Z
A few minor improvements (#150)
commit fafffa8f2445e2a559f24e5286c948c616287f9a
Author: Apurva Mehta <[email protected]>
Date: 2017-03-24T20:02:48Z
Reset transaction state on all irrecoverable exceptions (#153)
commit 637f864887e341c51f4fd7192016d2afc45bcf1e
Author: Apurva Mehta <[email protected]>
Date: 2017-03-24T21:08:29Z
Fix build errors due to rebase
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
