Hello RocketMQ Community,
This is the discussion for supporting 'Chunked Message' in RocketMQ. Like all messaging systems, Apache RocketMQ imposes a size limit on each message sent to the broker. This prevents the load per message from exceeding the maxMessageSize set in the Broker, which defaults to 4 MB. However, in application scenarios such as image processing and audio processing, many users need the RocketMQ client to send large messages to the broker. We can achieve this by adjusting maxMessageSize. However, this approach can cause many problems. For example, if a client publishes a 100 MB message, and the Broker allows the message to be stored to disk, the Broker will spend a lot of IO resources processing the message. This will affect the performance of broker flushing and cause a backlog of messages. A more common solution is to store large messages (such as audio and video) in object storage such as sso, and then use MQ for message notification and distribution. However, in some scenarios, this may not be applicable: - The message itself is not very large, e.g. only 16 MB, and the user does not want to store it in an external sso, which introduces additional overhead and makes the process cumbersome. - Many companies only want to use one system to store messages. Therefore, when RocketMQ sends large messages, in addition to the above solutions, we can also support the 'message chunking' feature. Through message chunking, the producer can split a large message into multiple blocks according to maxMessageSize, and send each block to the Broker as a normal message. Consumers assemble these chunked messages back into the original message when consuming. 'message chunking' is suitable for messages larger than 4mb, but not very large, for example: - Recommendation system scene AI model (about 16MB) - Flink tasks for big data scenarios (about 12MB) - The messages of some business teams often exceed 4MB, which cannot be separated and stored separately in sso - In my team, there are often business requirements where messages exceed the 4mb limit. It is worth mentioning that another message engine, Apache / Pulsar, also supports the 'message chunking' feature. So I think that although there are relatively few large message scenarios, and there is already a solution to store large messages in 'sso', 'message chunking' should also be used as an alternative solution, so that RocketMQ can support large messages like Pulsar Just as much friendlier.We shouldn't ignore this issue just because there are relatively few large message scenarios, That's not the right attitude to have when it comes to solving problems. Welcome to leave your views and discuss this Feature with us by replying to the email.
