Hi fuyou, Thanks for your feedback. Basically, if both broker and clients support v2 serialization, we can minimize the cost of serialization and deserialization a lot. If clients remained v1 and brokers upgraded, extra overhead would be minimal, within an acceptable boundary.
Here are a few blogs discussing pros and cos of this style of serde, among implementations. https://engineering.fb.com/2015/07/31/android/improving-facebook-s-performance-on-android-with-flatbuffers/ https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html Zhanhui Li On Tue, Oct 11, 2022 at 10:02 AM fuyou <[email protected]> wrote: > Great RIP, > one question: > we thought about zero copy when consumers pull messages,if we support > v2,zero copy can't support. > > Zhanhui Li <[email protected]> 于2022年10月10日周一 15:57写道： > > > Hi zhimin, > > > > >> 1. About the length of Topic, 128 can meet most scenarios. Even if the > > >> storage module supports store long topic data, > > >> I still recommend that the length of this non-system topic‘s > > >> length should not exceed the 128 limit. > > >> For system topics, such as the retry topic name of pop consumption > > >> can increase the upper limit. > > > > Storage layer, IMO, is not the best place to apply such limit. Broker > > module might be a better option; Normally, we should store a logical, > fixed > > width number in the storage system and let the upper layer to interpret > the > > actual name. > > > > > > >> For the system properties in the message, there should be its own > > >> namespace to distinguish it from the user's properties and protect it > > >> from being modified by the user. > > > > Yes! This is what this RIP is going to do. > > > > >> Is it necessary to store the server IP in every message? This > > >> design will waste more storage space. > > > > Yes, this is indeed an issue. Actually, two questions need to be > answered: > > 1, how does store IP help; 2, How to define store IP? The first node > that > > saves the message or the node that serves the message pull request. > > > > >> The "topic remapping" needs to consider the compatibility with the > > >> existing design, It is recommended to discuss it as another > > >> improvement. > > > > As pointed out in the previous section, most storage system stores > logical, > > fixed width number and let the business layer interpret the actual topic > > name. This matters in this RIP as it decides what to store in the storage > > tier. Logical fixed with number or string name or both during migration. > > > > >> We should consider compression and columnar storage in the new > > >> storage format. > > > > Compression can be made transparent, similar to what RocsDB/LevelDB does > > for SST files at the bottom most level. > > > > > > On Mon, Oct 10, 2022 at 2:50 PM zhimin li <[email protected]> wrote: > > > > > This is a great improvement plan, I have the following thoughts: > > > > > > 1. About the length of Topic, 128 can meet most scenarios. Even if the > > > storage module supports store long topic data, > > > I still recommend that the length of this non-system topic‘s > > > length should not exceed the 128 limit. > > > For system topics, such as the retry topic name of pop consumption > > > can increase the upper limit. > > > > > > 2. For the system properties in the message, there should be its own > > > namespace to distinguish it from the user's properties and protect it > > > from being modified by the user. > > > > > > 3. Is it necessary to store the server IP in every message? This > > > design will waste more storage space. > > > > > > 4. The "topic remapping" needs to consider the compatibility with the > > > existing design, It is recommended to discuss it as another > > > improvement. > > > > > > 5. We should consider compression and columnar storage in the new > > > storage format. > > > > > > > > -- > ============================================= > > fuyou001 > Best Regards >
