Re: About replacing byteBuffer
In fact, we can do that. There's no copy if bytebuf passed to the compositeByteBuf are wrapped by ByteBuffer. The problem is that we might not force every module to do this(may be checked only by review code). If anyone uses Netty's byteBuf directly and does not manage memory properly(and we don't have good surveillance to find this problem), this could result in a memory leak. Maybe, how to make better use of Netty, provide more monitoring capabilities, and constrain the development standards of all modules, this may need further consideration. BR, --- Sijia Li -邮件原件- 发件人: Xiangdong Huang 发送时间: 2022年5月17日 8:57 收件人: dev 主题:Re: About replacing byteBuffer If we introduce Netty, data copy when scaling a bytebuf is not what we want. Can we use compositeByteBuf to replace it and meanwhile enjoy the benefit of pooledByteBuf? --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2022年5月16日周一 12:35写道: > Hi, > > The serialization interface needs to be refactored afterward. > > Before that, using ByteArrayOutputStream is easier. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > > 李思佳 于2022年5月16日周一 11:44写道: > > > Hi all, > > > > When I was developing the snapshot interface for the configNode > > module, I noticed that the parameters received by the serialization > > interface were all defined as ByteBuffer, which seemed to have some > > problems. For > example, > > the external main process has no way of knowing how big the buffer > > will > be. > > We can only estimate a large value to allocate memory. > > > > Then I looked at the serialization interfaces of other modules, and > > it seemed that most modules did the same thing. This could be a > > problem once the actual size of the buffer exceeds our estimate. So > > I did a quick > survey > > of Netty's byteBuf last week, and here's the Chinese version of the > results< > > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>. > > > > At the same time, we found that the consensus module also has some > > ByteBuf requirements. But byteBuf doesn't seem to be enough to give > > us precise control over the size of the memory pool, and we may need > > to wrap it if we decide to use it. > > > > Finally, we decided to use stream type instead of byteBuffer in > > configNode for the time being. I will start this work to see if this > > is > the > > better way this week. If any idea, please let me know. > > > > By the way, Netty’s ByteBuf provides powerful tool operations that > > we will not discard outright, but rather as an option. > > > > BR, > > --- > > Sijia Li > > > > >
Re: About replacing byteBuffer
If we introduce Netty, data copy when scaling a bytebuf is not what we want. Can we use compositeByteBuf to replace it and meanwhile enjoy the benefit of pooledByteBuf? --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2022年5月16日周一 12:35写道: > Hi, > > The serialization interface needs to be refactored afterward. > > Before that, using ByteArrayOutputStream is easier. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > > 李思佳 于2022年5月16日周一 11:44写道: > > > Hi all, > > > > When I was developing the snapshot interface for the configNode module, I > > noticed that the parameters received by the serialization interface were > > all defined as ByteBuffer, which seemed to have some problems. For > example, > > the external main process has no way of knowing how big the buffer will > be. > > We can only estimate a large value to allocate memory. > > > > Then I looked at the serialization interfaces of other modules, and it > > seemed that most modules did the same thing. This could be a problem once > > the actual size of the buffer exceeds our estimate. So I did a quick > survey > > of Netty's byteBuf last week, and here's the Chinese version of the > results< > > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>. > > > > At the same time, we found that the consensus module also has some > > ByteBuf requirements. But byteBuf doesn't seem to be enough to give us > > precise control over the size of the memory pool, and we may need to wrap > > it if we decide to use it. > > > > Finally, we decided to use stream type instead of byteBuffer in > > configNode for the time being. I will start this work to see if this is > the > > better way this week. If any idea, please let me know. > > > > By the way, Netty’s ByteBuf provides powerful tool operations that we > > will not discard outright, but rather as an option. > > > > BR, > > --- > > Sijia Li > > > > >
Re: About replacing byteBuffer
Hi, The serialization interface needs to be refactored afterward. Before that, using ByteArrayOutputStream is easier. Thanks, — Jialin Qiao Apache IoTDB PMC 李思佳 于2022年5月16日周一 11:44写道: > Hi all, > > When I was developing the snapshot interface for the configNode module, I > noticed that the parameters received by the serialization interface were > all defined as ByteBuffer, which seemed to have some problems. For example, > the external main process has no way of knowing how big the buffer will be. > We can only estimate a large value to allocate memory. > > Then I looked at the serialization interfaces of other modules, and it > seemed that most modules did the same thing. This could be a problem once > the actual size of the buffer exceeds our estimate. So I did a quick survey > of Netty's byteBuf last week, and here's the Chinese version of the results< > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>. > > At the same time, we found that the consensus module also has some > ByteBuf requirements. But byteBuf doesn't seem to be enough to give us > precise control over the size of the memory pool, and we may need to wrap > it if we decide to use it. > > Finally, we decided to use stream type instead of byteBuffer in > configNode for the time being. I will start this work to see if this is the > better way this week. If any idea, please let me know. > > By the way, Netty’s ByteBuf provides powerful tool operations that we > will not discard outright, but rather as an option. > > BR, > --- > Sijia Li > >