Omer, Thanks for the email. This is an interesting thing to consider. Conceptually, there is no reason why the controllers couldn't manage the metadata for multiple brokers. The main counter-argument I can think of is essentially the same as the motivation -- less isolation. With a shared controller, one "noisy" broker cluster that put a lot of load on the controller could affect metadata availability/latency for other broker clusters. Related to this, having multiple broker clusters share one controller cluster means a larger blast radius for controller failures.
The "noisy neighbor" problem could be mitigated with a good implementation, but the failure coupling cannot. In the containerized world, resources are abstracted away, so there is not so much overhead to run a set of dedicated controller nodes. Even with bare-metal hardware, controller processes can be run on the same nodes as broker processes if needed. The 2+1 data center example seems a bit tangential to me. > This way metadata and data would have different level of availability and it enable enterprises to design a more cost effective solution by separating metadata and data service layer Is the idea here to have a multi-region controller quorum and then single region broker clusters? Could you achieve the same thing with one large Kafka cluster spread across regions but with topics having assignments that kept them region local? Is the "cost effectiveness" you're after just inter-broker networking costs? Maybe you could expand on this scenario and help motivate it a bit more? -David
