+1 Nice. LogicQueue is a great feature and will bring new capabilities for RocketMQ.
Xu16 Zhang 张旭 <zhangx...@xiaomi.com.invalid> 于2021年6月1日周二 下午12:04写道： > +1 > > > > -------- > 张旭 > 人工智能与云平台-云技术-计算平台组 > 电话: 15011056041 > > ________________________________________ > 发件人: zhibo li <osgooli...@gmail.com> > 发送时间: 2021年5月31日 15:00 > 收件人: dev@rocketmq.apache.org > 主题: [External Mail]Re: [VOTE] RIP-21 RocketMQ Logical Queue > > *外部邮件，谨慎处理 | This message originated from outside of XIAOMI. Please treat > this email with caution* > > > +1 > > heng du <duhengfore...@apache.org> 于2021年5月29日周六 上午9:55写道： > > > Hi RocketMQ Community, > > > > This is the vote for the kickoff of RIP-21 RocketMQ logic queue. > > > > In order to better meet the needs of stream computing and scaling, the > > concept of logical queues is proposed in this RIP, which enables RocketMQ > > to have the ability to scale in seconds without changing the number of > > queues. > > > > The vote will be open for at least 72 hours or until a necessary number > > of votes are reached. > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > > > > > ayanamist <ayanam...@gmail.com> 于2021年5月18日周二 上午10:14写道： > > > > > Below is Markdown text with some GFM syntax. > > > > > > # Status > > > - Current State: Proposed > > > - Authors: [ayanamist](https://github.com/ayanamist) > > > - Shepherds: duhengfore...@apache.org > > > - Mailing List discussion: dev@rocketmq.apache.org > > > - Pull Request: #PR_NUMBER > > > - Released: <relased_version> > > > # Background & Motivation > > > What do we need to do > > > - Will we add a new module? > > > No. > > > - Will we add new APIs? > > > It will add a new constant to mock broker names for logical queues. > > > - Will we add a new feature? > > > Yes. > > > > > > Why should we do that > > > - Are there any problems with our current project? > > > Currently, the MessageQueue of RocketMQ is coupled with broker name, > > which > > > results that the queue number will change if broker number increases or > > > decreases, which causes all queues to rebalance, which may cause > service > > > disruption like flink job restarts in minutes. > > > - What can we benefit from proposed changes? > > > The number of logical queues is not related with the number of brokers: > > We > > > can increase broker number without changing logical queue number, > > moreover, > > > we can increase logical queue number without deploying a new broker. > > > # Goals > > > - What problem is this proposal designed to solve? > > > Provide an abstraction, logical queue, to decouple between queue number > > and > > > broker number. > > > - To what degree should we solve the problem? > > > We should not hurt availability or performance in the implementation. > > > # Non-Goals > > > - What problem is this proposal NOT designed to solve? > > > We will not improve the mechanism of queues rebalance. > > > - Are there any limits of this proposal? > > > Only newer clients with changes in this proposal will benefit. > > > # Changes > > > ## Architecture > > > > > > We use one or more MessageQueue to make one LogicalQueue; One > > LogicalQueue > > > is unique in one topic, and one MessageQueue belongs to one and only > one > > > LogicalQueue. > > > > > > | brokerName | MessageQueue | LogicalQueue | > > > |------------|--------------|--------------| > > > | broker1 | 0 | 0 | > > > | broker1 | 1 | 1 | > > > | broker2 | 0 | 2 | > > > | broker2 | 1 | 3 | > > > > > > After one LogicalQueue migrated from broker1 to broker2, there will be > > two > > > MessageQueues for one LogicalQueue. We only migrate mapping but not > > actual > > > data, so broker1 is still serving for old data consuming but not data > > > producing. > > > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > > |------------|--------------|--------------|--------------------| > > > | **broker1** | **0** | **0(0-100)** | > **ReadOnly** > > > | > > > | broker1 | 1 | 1 | Normal | > > > | broker2 | 0 | 2 | Normal | > > > | broker2 | 1 | 3 | Normal | > > > | **broker2** | **2** | **0(101-)** | **Normal** > > > | > > > > > > After broker1 cleans all data from the commit log and consume queue, > > > QueueStatus becomes Expired(neither readable nor writable). > > > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > > |------------|--------------|--------------|-------------| > > > | **broker1** | **0** | **0(-)** | **Expired** > > | > > > | broker1 | 1 | 1 | Normal | > > > | broker2 | 0 | 2 | Normal | > > > | broker2 | 1 | 3 | Normal | > > > | broker2 | 2 | 0(101-) | Normal | > > > > > > If this LogicalQueue is migrated back to broker1, it will reuse this > > > expired MessageQueue > > > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > > |------------|--------------|--------------|-------------| > > > | **broker1** | **0** | **0(201-)** | **Normal** > > | > > > | broker1 | 1 | 1 | Normal | > > > | broker2 | 0 | 2 | Normal | > > > | broker2 | 1 | 3 | Normal | > > > | **broker2** | **2** | **0(101-200)** | **ReadOnly** > > | > > > > > > If this LogicalQueue is migrated back to broker1 while MessageQueue not > > > expired, it will create a new MessageQueue > > > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > > |------------|--------------|--------------|-------------| > > > | **broker1** | **0** | **0(0-100)** | **ReadOnly** > > | > > > | broker1 | 1 | 1 | Normal | > > > | **broker1** | **2** | **0(201-)** | **Normal** > > | > > > | broker2 | 0 | 2 | Normal | > > > | broker2 | 1 | 3 | Normal | > > > | **broker2** | **2** | **0(101-200)** | **ReadOnly** > > | > > > > > > If broker2 is offlined, all LogicalQueue in this broker should be > > migrated > > > away. > > > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > > |------------|--------------|--------------|-------------| > > > | broker1 | 0 | 0 | Normal | > > > | broker1 | 1 | 1 | Normal | > > > | **broker1** | **2** | **2(101-)** | **Normal** > > | > > > | **broker1** | **3** | **3(101-)** | **Normal** > > | > > > | **broker2** | **0** | **2(0-100)** | **ReadOnly** > > | > > > | **broker2** | **1** | **3(0-100)** | **ReadOnly** > > | > > > > > > When all data including commit log and consume queue in broker2 are > > > cleaned, broker2 can be removed. > > > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > > |------------|--------------|--------------|-------------| > > > | broker1 | 0 | 0 | Normal | > > > | broker1 | 1 | 1 | Normal | > > > | **broker1** | **2** | **2(101-)** | **Normal** > | > > > | **broker1** | **3** | **3(101-)** | **Normal** > > | > > > > > > When a new broker is deployed, we can migrate some LogicalQueues to > this > > > broker to spare some producing traffic. > > > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > > |------------|--------------|--------------|-------------| > > > | broker1 | 0 | 0 | Normal | > > > | broker1 | 1 | 1 | Normal | > > > | **broker1** | **2** | **2(101-200)** | **ReadOnly** > > | > > > | **broker1** | **3** | **3(101-200)** | **ReadOnly** > > | > > > | **broker3** | **0** | **2(201-)** | **Normal** > > | > > > | **broker3** | **1** | **3(201-)** | **Normal** > > | > > > > > > All mapping data are stored separately in brokers, and via > registerBroker > > > to be gathered in namesrv; Each broker only stores its own logical > queue > > > mapping but not other one's. All management operations should be > invoked > > by > > > CLI or direct rpc command, automatic operation support is missing now, > > and > > > require [RIP-18]( > > > > > > > > > https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade > > > ) > > > to be implemented first. > > > > > > ## Interface Design/Change > > > - Method signature changes > > > No. > > > - Method behavior changes > > > When a topic is enabled LogicalQueue support, broker name of > MessageQueue > > > result returned by some methods like `fetchSubscribeMessageQueues` or > > > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does > > not > > > have broker name concept. > > > - CLI command changes > > > Add some operation command for LogicalQueue, like > > > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping` > > > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` > `MigrateLogicalQueue`. > > > Also `updateTopic` adds a new parameter to enable LogicalQueue support > > > immediately after the topic is created. > > > - Log format or content changes > > > No. > > > ## Compatibility, Deprecation, and Migration Plan > > > - Are backward and forward compatibility taken into consideration? > > > Yes. > > > - Everything will work well if no topic is enabled LogicalQueue > support, > > > whether on old/new broker/namesrv/client. > > > - LogicalQueue support will work only under new broker+namesrv+client, > > > otherwise, everything will work like no LogicalQueue support. > > > - Are there deprecated APIs? > > > No. > > > - How do we do migration? > > > No need to migrate, this is a feature which needs to be enabled > manually. > > > ## Implementation Outline > > > We will implement the proposed changes by 2 phases. > > > ### Phase 1 basic function > > > 1. Implement LogicalQueue routes query and merge mechanism in namesrv. > > > 2. Implement LogicalQueue mapping report from broker to namesrv. > > > 3. Implement LogicalQueue support in the client. > > > 4. Implement LogicalQueue migration in broker. > > > 5. Implement LogicalQueue admin processor and CLI. > > > ### Phase 2 compatibility > > > 1. Implement proxy request+response for old client in broker. > > > 2. Implement intermediate state(support is partly enabled in some > brokers > > > but not all) protection in namesrv. > > > # Rejected Alternatives > > > - How does alternatives solve the issue you proposed? > > > Implement a whole new Logical Queue architecture from scratch, this > > > absolutely will solve the problem. > > > - Pros and Cons of alternatives > > > Pros: from-scratch way does not bring anything good. > > > Cons: from-scratch way will break existent concept, add much more > > > complexity to code and not user-friendly. > > > - Why should we reject the above alternatives > > > It does no good. > > > > > > #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ > This e-mail and its attachments contain confidential information from > XIAOMI, which is intended only for the person or entity whose address is > listed above. Any use of the information contained herein in any way > (including, but not limited to, total or partial disclosure, reproduction, > or dissemination) by persons other than the intended recipient(s) is > prohibited. If you receive this e-mail in error, please notify the sender > by phone or email immediately and delete it!******/# >
