[
https://issues.apache.org/jira/browse/KAFKA-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16756554#comment-16756554
]
Ask Solem commented on KAFKA-6278:
----------------------------------
@Jason can you use multiple transactional ids over the same network socket? I
tried sending multiple InitProducerIDRequests on the same socket and it doesn't
complain, but would this be safe to do?
I'm trying to implement exactly-once in our kafka streams port, and reading the
Kafka Streams source code it seems they are actually using one producer per
TaskId+partition?! That would mean you could end up having hundreds of
producers, not just one per thread, so now investigating using multiple
transactional ids in the same producer.
> Allow multiple concurrent transactions on a single producer
> -----------------------------------------------------------
>
> Key: KAFKA-6278
> URL: https://issues.apache.org/jira/browse/KAFKA-6278
> Project: Kafka
> Issue Type: New Feature
> Components: producer
> Reporter: Tim Cuthbertson
> Priority: Major
>
> It's recommended to share a producer between threads, because it's likely
> faster / cheaper.
> However with the transactional API there's a big caveat. If you're using
> transactions, every message sent to a given producer instance will be
> considered part of the "active transaction" regardless of what thread it came
> from. Furthermore, if two threads want to use transactions on a shared
> producer instance, it (probably) won't work.
> Possible fix: add an API which exposes the transaction ID to the user,
> instead of making it internal state of the producer. e.g.:
> {noformat}
> Transaction tx = producer.beginTransaction()
> producer.send(tx, message)
> producer.commitTransaction(tx)
> {noformat}
> That way, it's explicit which transaction a message will be part of, rather
> than the current state which is "the open transaction, which may have been
> opened by an unrelated thread".
> See also initial discussion on slack:
> https://confluentcommunity.slack.com/archives/C488525JT/p1511739734000012
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
