Hi, lanyu and hongying I am very pleased to see that you have made breakthroughs in memory management, and I am so eager to learn memory control techniques from you!
Best ——————— Xinyu Tan > 2024年5月23日 15:24，Yuan Tian <jackietie...@gmail.com> 写道： > > Hi hongying, > > sure, feel free to discuss anything about any modules of IoTDB in mail list. > > On Thu, May 23, 2024 at 2:42 PM 张洪胤 <sprico...@163.com> wrote: > >> hi, yuan tian, I am investigating the memory management mechanisms of >> Oracle, PostgreSQL, and other databases, and sorting out the current memory >> management status of Apache IoTDB. I believe that a unified memory >> management framework cannot be separated from the discussion and >> participation of the entire community, as well as the participation of open >> source workers who specialize in different modules. >> >> >> >> >> Best regards, >> >> Hongyin Zhang >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> At 2024-05-23 14:25:01, "Yuan Tian" <jackietie...@gmail.com> wrote: >>> Hi hongying, >>> >>> looking forward to your unified memory management framework for IoTDB. >>> >>> On Thu, May 23, 2024 at 12:11 PM Hongyin Zhang <sprico...@apache.org> >> wrote: >>> >>>> +1, brilliant job! This work allowed me to learn a lot, which laid a >> solid >>>> foundation for subsequent system memory control and improved the >> system's >>>> robustness in query scenarios. >>>> >>>> Best regards, >>>> Hongyin Zhang >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> At 2024-05-23 11:08:35, "Liao Lanyu" <1435078...@qq.com.INVALID> wrote: >>>>> Hi, >>>>> Currently, the IoTDB query engine does not implement memory control at >>>> the FE (Frontend) stage. In scenarios with massive series queries (e.g., >>>> select * from root.**), the query plans generated at the FE stage can >>>> become excessively large. Roughly estimating, the size of a single >>>> SeriesScanNode is about 1/2 KB, which means that two million series >>>> corresponding to two million SeriesScanNodes would occupy 1 GB, posing a >>>> potential risk of Out-Of-Memory (OOM). In high concurrency scenarios, >> even >>>> if a single query plan is not large, the total memory occupied by >> multiple >>>> query plans can still lead to OOM. >>>>> Therefore, it is now desired to introduce memory size control for FE >>>> query plans within the query engine. >>>>> The PR is: https://github.com/apache/iotdb/pull/12573 >>>>> >>>>> >>>>> >>>>> >>>>> 1435078631 >>>>> 1435078...@qq.com >>>>> >>>>> >>>>> >>>>> >>>> >>
