RIP-19 RocketMQ DelayMessage Status
* Current State: Proposed * Authors: * Shepherds: * Mailing List discussion: [RocetMQ Developers]([email protected]) * Pull Request: #PR_NUMBER * Released: <relased_version> Background & Motivation At present, RocketMQ only supports delay messages with fixed delay levels(1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h), which is not flexible enough, and the current delay message design is realized by using new separate Topic(SCHEDULE_TOPIC_XXXX) and Queue, if we want to achieve any time delay of seconds within 40 days, based on the existing solution, 40 * 24 * 60 * 60 = 345600 Queues are needed, and the cost is very high. Also, the message cannot be delayed too long because the CommitLog will be deleted after a certain amount of time (default 3 days) In order to meet the requirements of flexible time delay messages, this proposal solves this problem by redesigning the architecture of delay messages and introducing TimeWheel. Goals Delay message can be delivered in any time delay of seconds within 40 days(or other time range, this should be configured by deployer). Non-Goals * Does not support millisecond delay * Delay message delivery over 40 days will fail * If the consumer currently has a message accumulation, the delay message will be ranked after the accumulation messages, and will not be delivered strictly according to the configured time Changes Architectures Adopt multi-level TimeWheel design, coarse-grained TimeWheel on disk, fine-grained TimeWheel in RAM. DelayMessageLog stores the delay message sending progress. The normal process is as follows: 1. Message sequence appended to CommitLog 2. The ReputMessageService thread continuously obtains newly added messages, and the Dispatcher determines whether it is a delayed message: * Normal Message: ConsumeQueue is constructed normally, consumers consume normally * Delay Message: re-stored to DelayMessageStorage, stored according to the coarser time granularity of the multi-level TimeWheel, such as hour; if the delivery time is in the range of the TimeWheel in the RAM, the message is also added to the TimeWheel in the RAM. 3. At the beginning of the coarse-grained time, the TimeWheel in RAM is constructed according to DelayMessageStorage. 4. Convert the delay message to normal message and append it to CommitLog. If the append is successful, update the delay message delivery record DelayMessageLog. Implementation Outline We will implement the proposed changes by 3 phases. Phase 1 In the master-slave cluster deployment mode, only the master can provide delivery of delay messages Phase 2 In the master-slave cluster deployment mode, both master and slave can provide delivery of delay messages Phase 3 In the cluster deployment mode based on Dleger, delay messages can be delivered normally. HeChengyang
