Hi Pulsar Community,
I discovered a bug that chunk messages producing fails if topic level
maxMessageSize is set [1]. The root cause of this issue is because chunk
message is using broker level maxMessageSize as chunk size. And topic level
maxMessageSize is always <= broker level maxMessageSize. So once it is set, the
on-going chunk message producing fails.
## Proposed changes
I would like to fix this by just skipping topic level maxMessageSize check in
org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
level maxMessageSize is introduced in [2], for the purpose of "easier to plan
resource quotas for client allocation". And IMO this change will not bring
further complex into this.
## Alternative
Add a client side topic level maxMessageSize and keep it synced with broker.
Required changes:
- [client] Add a new field
org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this client
side topic level maxMessageSize.
- [PulsarApi.proto] Add a new field maxMessageSize in the
CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
- [PulsarApi.proto] Add a new Command like
CommandUpdateClientPolicy{producerId, maxMessageSize} to update
ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
Further more, some other data consistency issues need be handled very carefully
when maxMessageSize is updated.
This alternative is complex but can also solve other topic level maxMessageSize
issue [3] when batching is enabled (non-batching case is solved with PR [4]).
Any suggestions or other use cases of topic level maxMessageSize will be
appreciated.
Thanks,
Haiting Jiang
[1] https://github.com/apache/pulsar/issues/13360
[2] https://github.com/apache/pulsar/pull/8732
[3] https://github.com/apache/pulsar/issues/12958
[4] https://github.com/apache/pulsar/pull/13147
