jolshan commented on code in PR #17454:
URL: https://github.com/apache/kafka/pull/17454#discussion_r1795963676
##########
docs/design.html:
##########
@@ -290,24 +290,56 @@ <h3 class="anchor-heading"><a id="semantics"
class="anchor-link"></a><a href="#s
messages have a primary key and so the updates are idempotent (receiving
the same message twice just overwrites a record with another copy of itself).
</ol>
<p>
- So what about exactly once semantics (i.e. the thing you actually want)?
When consuming from a Kafka topic and producing to another topic (as in a <a
href="https://kafka.apache.org/documentation/streams";>Kafka Streams</a>
- application), we can leverage the new transactional producer capabilities
in 0.11.0.0 that were mentioned above. The consumer's position is stored as a
message in a topic, so we can write the offset to Kafka in the
- same transaction as the output topics receiving the processed data. If the
transaction is aborted, the consumer's position will revert to its old value
and the produced data on the output topics will not be visible
- to other consumers, depending on their "isolation level." In the default
"read_uncommitted" isolation level, all messages are visible to consumers even
if they were part of an aborted transaction,
- but in "read_committed," the consumer will only return messages from
transactions which were committed (and any messages which were not part of a
transaction).
+ So what about exactly-once semantics? When consuming from a Kafka topic
and producing to another topic (as in a <a
href="https://kafka.apache.org/documentation/streams";>Kafka Streams</a>
application), we can
+ leverage the new transactional producer capabilities in 0.11.0.0 that were
mentioned above. The consumer's position is stored as a message in an internal
topic, so we can write the offset to Kafka in the
+ same transaction as the output topics receiving the processed data. If the
transaction is aborted, the consumer's stored position will revert to its old
value and the produced data on the output topics will not
Review Comment:
Did we want to change this line because it is misleading -- the stored
positition doesn't revert unless the offsets are fetched again.
##########
docs/design.html:
##########
@@ -290,24 +290,56 @@ <h3 class="anchor-heading"><a id="semantics"
class="anchor-link"></a><a href="#s
messages have a primary key and so the updates are idempotent (receiving
the same message twice just overwrites a record with another copy of itself).
</ol>
<p>
- So what about exactly once semantics (i.e. the thing you actually want)?
When consuming from a Kafka topic and producing to another topic (as in a <a
href="https://kafka.apache.org/documentation/streams";>Kafka Streams</a>
- application), we can leverage the new transactional producer capabilities
in 0.11.0.0 that were mentioned above. The consumer's position is stored as a
message in a topic, so we can write the offset to Kafka in the
- same transaction as the output topics receiving the processed data. If the
transaction is aborted, the consumer's position will revert to its old value
and the produced data on the output topics will not be visible
- to other consumers, depending on their "isolation level." In the default
"read_uncommitted" isolation level, all messages are visible to consumers even
if they were part of an aborted transaction,
- but in "read_committed," the consumer will only return messages from
transactions which were committed (and any messages which were not part of a
transaction).
+ So what about exactly-once semantics? When consuming from a Kafka topic
and producing to another topic (as in a <a
href="https://kafka.apache.org/documentation/streams";>Kafka Streams</a>
application), we can
+ leverage the new transactional producer capabilities in 0.11.0.0 that were
mentioned above. The consumer's position is stored as a message in an internal
topic, so we can write the offset to Kafka in the
+ same transaction as the output topics receiving the processed data. If the
transaction is aborted, the consumer's stored position will revert to its old
value and the produced data on the output topics will not
Review Comment:
Did we want to change this line because it is misleading -- the stored
position doesn't revert unless the offsets are fetched again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
