Hi Greg, > I don't think I understand what you mean here. Are you suggesting an alternative to the Admin API? An external project could certainly build such a component with the Admin API.
So I was thinking of something more complex than the Admin API, an external service that can instruct clusters (using the Admin API) to start or stop replication for certain partitions, list their replication flows, throttle them and so on. With this you won't be able to control just one cluster but you could register your clusters and control them in a centralized fashion. Similar to what RestServer is to Connect, though with this you don't create connectors but replication flows. I've updated the KIP with my remote log storage thoughts (in data semantics). Best, Viktor On Sat, Oct 7, 2023 at 7:22 PM Greg Harris <greg.har...@aiven.io.invalid> wrote: > Hello hudeqi, > > I apologize if the KIP and discussion have diverged, as I've been > trying to add detail rather than propose changes. > > > Why can't we use the follower fetch protocol? > > What you've described sounds like a very reasonable implementation of > CCR. I purposely have not specified any implementation details so far > and have been focusing only on the user-facing semantics of the > feature. You are of course welcome to add details of how the > follower-fetch implementation would work to the KIP. > > I think maybe this wording in the KIP was ambiguous: "Are not eligible > for fetch-from-follower on the source cluster". I clarified the > justification for this earlier in Tom's point D3. But to make the > statement itself more clear: > > Consumers on the source cluster will not see a cross-cluster leader or > followers as valid replicas to fetch from for load-sharing or latency > optimization. Consumers on the target cluster will see the > cross-cluster followers as valid replicas to fetch from, as they will > appear as normal replicas. This was only meant to describe the > relationship of the remote replicas with Consumers, not with each > other. > > I hope this is more clear. > Greg > > On Sat, Oct 7, 2023 at 1:38 AM hudeqi <16120...@bjtu.edu.cn> wrote: > > > > Hi, Greg. > > > > After reading this KIP and your discussion, I feel that it is very > divergent. I can only start from one of them: > > Why can't we use the follower fetch protocol? The leader of the target > cluster topic partition can be treated as the follower of the source > cluster topic partition leader, and the fetched data is directly appended > to the local log (the remote fetch thread is inherited to the follower > fetch thread, thereby retaining the offset of the log), so that consumer/ > producer client can be omitted. Of course, this is just data replication. I > may have to think more about group offset/acl/config replication. > > > > best, > > hudeqi >
