hi, yuan tian, I am investigating the memory management mechanisms of Oracle, PostgreSQL, and other databases, and sorting out the current memory management status of Apache IoTDB. I believe that a unified memory management framework cannot be separated from the discussion and participation of the entire community, as well as the participation of open source workers who specialize in different modules.
Best regards, Hongyin Zhang At 2024-05-23 14:25:01, "Yuan Tian" <jackietie...@gmail.com> wrote: >Hi hongying, > >looking forward to your unified memory management framework for IoTDB. > >On Thu, May 23, 2024 at 12:11 PM Hongyin Zhang <sprico...@apache.org> wrote: > >> +1, brilliant job! This work allowed me to learn a lot, which laid a solid >> foundation for subsequent system memory control and improved the system's >> robustness in query scenarios. >> >> Best regards, >> Hongyin Zhang >> >> >> >> >> >> >> >> >> >> >> >> At 2024-05-23 11:08:35, "Liao Lanyu" <1435078...@qq.com.INVALID> wrote: >> >Hi, >> >Currently, the IoTDB query engine does not implement memory control at >> the FE (Frontend) stage. In scenarios with massive series queries (e.g., >> select * from root.**), the query plans generated at the FE stage can >> become excessively large. Roughly estimating, the size of a single >> SeriesScanNode is about 1/2 KB, which means that two million series >> corresponding to two million SeriesScanNodes would occupy 1 GB, posing a >> potential risk of Out-Of-Memory (OOM). In high concurrency scenarios, even >> if a single query plan is not large, the total memory occupied by multiple >> query plans can still lead to OOM. >> >Therefore, it is now desired to introduce memory size control for FE >> query plans within the query engine. >> >The PR is: https://github.com/apache/iotdb/pull/12573 >> > >> > >> > >> > >> >1435078631 >> >1435078...@qq.com >> > >> > >> > >> > >>
