[
https://issues.apache.org/jira/browse/KAFKA-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-13523:
---------------------------------
Priority: Blocker (was: Major)
> Implement IQv2 support in global stores
> ---------------------------------------
>
> Key: KAFKA-13523
> URL: https://issues.apache.org/jira/browse/KAFKA-13523
> Project: Kafka
> Issue Type: Sub-task
> Reporter: John Roesler
> Priority: Blocker
>
> Global stores pose one significant problem for IQv2: when they start up, they
> skip the regular ingest pipeline and instead use the "restoration" pipeline
> to read up until the current end offset. Then, they switch over to the
> regular ingest pipeline.
> IQv2 position tracking expects to track the position of each record from the
> input topic through the ingest pipeline and then get the position headers
> through the restoration pipeline via the changelog topic. The fact that
> global stores "restore" the input topic instead of ingesting it violates our
> expectations.
> It has also caused other problems, so we may want to consider switching the
> global store processing to use the normal paradigm rather than adding
> special-case logic to track positions in global stores.
>
> Note: there are two primary reasons that global stores behave this way:
> # We can write in batches during restoration, so the I/O may be more
> efficient
> # The global thread does not transition to RUNNING state until it reaches
> the (current) end of the input topic, which blocks other threads from joining
> against it, thereby improving the time synchronization of global KTable joins.
> If we want to propose changing the bootstrapping pipeline for global threads,
> we should have some kind of answer to these concerns.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
