Great RIP, one question: we thought about zero copy when consumers pull messages,if we support v2,zero copy can't support.
Zhanhui Li <lizhan...@gmail.com> 于2022年10月10日周一 15:57写道： > Hi zhimin, > > >> 1. About the length of Topic, 128 can meet most scenarios. Even if the > >> storage module supports store long topic data, > >> I still recommend that the length of this non-system topic‘s > >> length should not exceed the 128 limit. > >> For system topics, such as the retry topic name of pop consumption > >> can increase the upper limit. > > Storage layer, IMO, is not the best place to apply such limit. Broker > module might be a better option; Normally, we should store a logical, fixed > width number in the storage system and let the upper layer to interpret the > actual name. > > > >> For the system properties in the message, there should be its own > >> namespace to distinguish it from the user's properties and protect it > >> from being modified by the user. > > Yes! This is what this RIP is going to do. > > >> Is it necessary to store the server IP in every message? This > >> design will waste more storage space. > > Yes, this is indeed an issue. Actually, two questions need to be answered: > 1, how does store IP help; 2, How to define store IP? The first node that > saves the message or the node that serves the message pull request. > > >> The "topic remapping" needs to consider the compatibility with the > >> existing design, It is recommended to discuss it as another > >> improvement. > > As pointed out in the previous section, most storage system stores logical, > fixed width number and let the business layer interpret the actual topic > name. This matters in this RIP as it decides what to store in the storage > tier. Logical fixed with number or string name or both during migration. > > >> We should consider compression and columnar storage in the new > >> storage format. > > Compression can be made transparent, similar to what RocsDB/LevelDB does > for SST files at the bottom most level. > > > On Mon, Oct 10, 2022 at 2:50 PM zhimin li <lizhim...@gmail.com> wrote: > > > This is a great improvement plan, I have the following thoughts: > > > > 1. About the length of Topic, 128 can meet most scenarios. Even if the > > storage module supports store long topic data, > > I still recommend that the length of this non-system topic‘s > > length should not exceed the 128 limit. > > For system topics, such as the retry topic name of pop consumption > > can increase the upper limit. > > > > 2. For the system properties in the message, there should be its own > > namespace to distinguish it from the user's properties and protect it > > from being modified by the user. > > > > 3. Is it necessary to store the server IP in every message? This > > design will waste more storage space. > > > > 4. The "topic remapping" needs to consider the compatibility with the > > existing design, It is recommended to discuss it as another > > improvement. > > > > 5. We should consider compression and columnar storage in the new > > storage format. > > > -- ============================================= fuyou001 Best Regards
