[ ] +1 approve it would make Consumer Client more simple to use. and do u have a work flow to show how POP mode work ?
heng du <[email protected]> 于2021年1月18日周一 下午2:37写道： > Hi RocketMQ Community, > > This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming. > > In order to better implement a lightweight client, @ayanamist proposes a > new consumption model, and at the same time transfers the load balancing > logic of the original client to the broker, which not only solves the > original queue occupancy problem but also It can also avoid the consumption > delay caused by a certain consumer hangs. > > The vote will be open for at least 72 hours or until a necessary number of > votes are reached. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > > > > Best Regards! > Henry > > ayanamist <[email protected]> 于2021年1月8日周五 上午11:25写道： > > > # RIP-19 RocketMQ Pop Consuming > > > > # Status > > > > - Current State: Proposed > > - Authors: [ayanamist]([ > > https://github.com/ayanamist/](https://github.com/ayanamist/)) > > - Shepherds: [duhengforever]([ > > https://github.com/duhenglucky/](https://github.com/duhengforever/)) > > - Mailing List discussion: [email protected]; > > [email protected] > > - Pull Request: RIP-19 > > - Released: - > > > > # Background & Motivation > > > > ### What do we need to do > > > > - Will we add a new module? > > > > No. > > > > - Will we add new APIs? > > > > Yes. > > > > - Will we add new feature? > > > > Yes. > > > > ### Why should we do that > > > > - Are there any problems of our current project? > > > > The current subscription load balancing strategy is based on the > > dimension of message queue. All behaviors are owned by the client side. > > There are three main steps: > > > > 1. Each consumer regularly obtains the total number of topic message > > queues and all consumers. > > 2. Using a general algorithm to sort the queues by consumer ip and > > queue index to calculate which message queue is allocated to which > > consumer. > > 3. Each consumer pulls messages using allocated orders described > above. > > > > According to this allocation method, if an abnormality occurs in a > > consumer (the application itself is abnormal, or a broker is upgrading) > so > > that it causes slow subscription, messages will be accumulated, but this > > queue will not be re-allocated to another consumer, so the accumulation > > will become more and more serious. > > > > > > Chinese version: > > > > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步： > > > > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数 > > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列 > > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费 > > > > > > > > > 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。 > > > > - What can we benefit proposed changes? > > > > The accumulated messages will be subscribed by other consumers if one > > consumer behaves abnormally. > > > > Chinese version: > > > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。 > > > > # Goals > > > > - What problem is this proposal designed to solve? > > > > The accumulated messages will be subscribed by other consumers if one > > consumer behaves abnormally. > > > > Chinese version: > > > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。 > > > > - To what degree should we solve the problem? > > > > This RIP must guarantee below point: > > > > 1. High availablity: Subscription of one message queue will not be > > affected by single consumer failure. > > 2. High performance: This implementation affects latency and > throughput > > less than 10%. > > > > > > Chinese version: > > > > 新方案需要保证两点： > > > > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响 > > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内 > > > > # Non-Goals > > > > - What problem is this proposal NOT designed to solve? > > > > Improve client-side load balancing. > > > > - Are there any limits of this proposal? > > > > Nothing specific. > > > > # Changes > > > > ## Architecture > > > > Current "Pull mode": > >  > > > > Proposed "Pop mode": > >  > > > > Move inter-queue balance of one topic from client side to server side. > > Clients make pull request without specified queues to broker, and broker > > fetch messages from queues internally and returns, which ensures one > queue > > will be consumed by multiple clients. The whole behavior is like a queue > > pop process. > > > > It will add a new request command querying queue assignments in broker, > and > > add pop-feature-support flag to pull request which makes broker use pop > > mode. > > > > ## Interface Design/Change > > > > - Method signature changes > > > > Nothing specific. > > > > - Method behavior changes > > > > Nothing specific. > > > > - CLI command changes > > > > Add `setConsumeMode` for admin to switch between old pull mode and > new > > pop mode for one subscription. > > > > - Log format or content changes > > > > Nothing specific. > > > > ## Compatibility, Deprecation, and Migration Plan > > > > - Are backward and forward compatibility taken into consideration? > > > > New RequestCode between client and broker are added, so there are 2 > > compatibility situations: > > > > 1. old client+new broker: old clients won't make request with > > pop-feature-support flag, so broker will not enable pop mode, which keep > > all things as before. > > 2. new client+old broker: new clients will detect whether broker > > support the new request command querying queue assignments, if not, it > will > > fallback to use old pull mode. > > > > - Are there deprecated APIs? > > > > Nothing specific. > > > > - How do we do migration? > > > > Nothing specific. > > > > ## Implementation Outline > > > > We will implement the proposed changes by **2** phases. > > > > ## Phase 1 > > > > 1. Implement server-side balance capability in broker > > 2. Implement client-side request using new pop-mode > > > > ## Phase 2 > > > > 1. Implement new sdk compatibility with old broker. > > 2. Implement feature detection in broker and client. > > > > # Rejected Alternatives > > > > ## How does alternatives solve the issue you proposed? > > > > Improve client rebalance logic? I don't get a quite good idea. > > > > ## Pros and Cons of alternatives > > > > Client rebalance logic will become quite complicated. > > > > ## Why should we reject above alternatives > > >
