gosonzhang opened a new issue #1524: URL: https://github.com/apache/incubator-inlong/issues/1524
In our environment, there is a situation where the number of concurrent filter consumption of Topic in some clusters has reached 100, and it is likely to exceed 100 to 200, or even 300, but the number of concurrent consumption of our single node is limited. So, we want to add a replica solution to TubeMQ to support over 100 concurrent filtered consumption. Considering that the current TubeMQ mainly supports high-performance scenarios, we consider the asynchronous replication copy scheme, which is similar to Kafka's replica scheme, but is slightly different:  1. Cluster's Broker nodes that support the replication scheme are organized according to Group, and the nodes in the Group are assigned as one Leader and more Follower nodes; Follower nodes pulls data asynchronously from Leader node and saves it locally; 2. The Leader node in the Group undertakes production and a small amount of consumption; the Follower nodes only do pure consumption processing. When the Leader node fails, the Follower nodes can be switched to the Leader node and perform related change processing; 3. The producer and consumer have no perception of the replica, and still maintain the existing realization for production and consumption -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
