Hi zhimin, >> 1. About the length of Topic, 128 can meet most scenarios. Even if the >> storage module supports store long topic data, >> I still recommend that the length of this non-system topic‘s >> length should not exceed the 128 limit. >> For system topics, such as the retry topic name of pop consumption >> can increase the upper limit.
Storage layer, IMO, is not the best place to apply such limit. Broker module might be a better option; Normally, we should store a logical, fixed width number in the storage system and let the upper layer to interpret the actual name. >> For the system properties in the message, there should be its own >> namespace to distinguish it from the user's properties and protect it >> from being modified by the user. Yes! This is what this RIP is going to do. >> Is it necessary to store the server IP in every message? This >> design will waste more storage space. Yes, this is indeed an issue. Actually, two questions need to be answered: 1, how does store IP help; 2, How to define store IP? The first node that saves the message or the node that serves the message pull request. >> The "topic remapping" needs to consider the compatibility with the >> existing design, It is recommended to discuss it as another >> improvement. As pointed out in the previous section, most storage system stores logical, fixed width number and let the business layer interpret the actual topic name. This matters in this RIP as it decides what to store in the storage tier. Logical fixed with number or string name or both during migration. >> We should consider compression and columnar storage in the new >> storage format. Compression can be made transparent, similar to what RocsDB/LevelDB does for SST files at the bottom most level. On Mon, Oct 10, 2022 at 2:50 PM zhimin li <lizhim...@gmail.com> wrote: > This is a great improvement plan, I have the following thoughts: > > 1. About the length of Topic, 128 can meet most scenarios. Even if the > storage module supports store long topic data, > I still recommend that the length of this non-system topic‘s > length should not exceed the 128 limit. > For system topics, such as the retry topic name of pop consumption > can increase the upper limit. > > 2. For the system properties in the message, there should be its own > namespace to distinguish it from the user's properties and protect it > from being modified by the user. > > 3. Is it necessary to store the server IP in every message? This > design will waste more storage space. > > 4. The "topic remapping" needs to consider the compatibility with the > existing design, It is recommended to discuss it as another > improvement. > > 5. We should consider compression and columnar storage in the new > storage format. >
