Timed flush memtable and timed close TsFileProcessor
Hi, If we open a TsFileProcessor and then never write to it, this TsFileProcessor will become a zombie because nothing can trigger a flush or close operation. So, i am implementing timed flush memtable [1] and timed close TsFileProcessor [2]. Timed flush memtable is used to flush inactive memtables periodically according to their created time. Timed close TsFileProcessor is used to close inactive TsFileProcessors periodically according to how long they don't process write operations. I add 9 params to control above two functions, and these 9 params all can be hot loaded by 'load configuration' command. 1. enable_timed_flush_seq_memtable: whether to timed flush sequence tsfiles' memtables. 2. seq_memtable_flush_interval_in_ms: when a sequence memTable's created time is older than current time minus this, the memtable is flushed to disk. 3. seq_memtable_flush_check_interval_in_ms: the interval to check whether the sequence memtables need flushing. 4. enable_timed_flush_unseq_memtable: whether to timed flush unsequence tsfiles' memtables. 5. unseq_memtable_flush_interval_in_ms: when an unsequence memTable's created time is older than current time minus this, the memtable is flushed to disk. 6. unseq_memtable_flush_check_interval_in_ms: the interval to check whether the unsequence memtables need flushing. 7. enable_timed_close_tsfile: whether to timed close TsfileProcessors. 8. close_tsfile_interval_after_flushing_in_ms: when TsfileProcessor's last working memtable flush time is older than current time minus this and its working memtable is null, this TsfileProcessor will be closed. 9. close_tsfile_check_interval_in_ms: the interval to check whether the TsfileProcessors need closing. What do you think? [1] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?? Haiming Zhu
Re: Timed flush memtable and timed close TsFileProcessor
Hi, I have already implemented both issues [1][2]. Timed flush memtable [3] is used to flush inactive memtables periodically according to their created time. For example, if we set seq_memtable_flush_interval_in_ms to 1000, memtable A, created at time 2000, can be flushed after time 3000. Timed close TsFileProcessor [4] is used to close inactive TsFileProcessors periodically according to how long they don't process write operations. For example, if we set close_tsfile_interval_after_flushing_in_ms to 1000, TsFileProcessor A, whose last workMemtable was flush at time 2000 and current workMemtable is null (i.e., no more wirite operations after last flush), can be closed after time 3000. But, i am confused whether to open timed flush seq/unseq memtable and timed close TsFileProcessor by default, because they can save memory in some user cases while they just waste cpu in other user cases. Look forward to your reply. [1] https://github.com/apache/iotdb/pull/3776 [2] https://github.com/apache/iotdb/pull/3777 [3] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [4] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?? Haiming Zhu -- -- ??: "dev" https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?? Haiming Zhu
Re: Timed flush memtable and timed close TsFileProcessor
Hi, > I am confused whether to open timed flush seq/unseq memtable and timed close TsFileProcessor by default, because they can save memory in some user cases while they just waste cpu in other user cases. It could be enabled by default, both seq and unseq. The 12 hours for memtable flush interval and close file interval is fine. If it really kept 12 hours in memory, the CPU load should be very low, no need to worry. Thanks, — Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 Haiming Zhu 于2021年8月19日周四 下午4:37写道： > Hi, > > > I have already implemented both issues [1][2]. > > > Timed flush memtable [3] is used to flush inactive memtables periodically > according to their created time. For example, if we set > seq_memtable_flush_interval_in_ms to 1000, memtable A, created at time > 2000, can be flushed after time 3000. > > > Timed close TsFileProcessor [4] is used to close inactive TsFileProcessors > periodically according to how long they don't process write operations. For > example, if we set close_tsfile_interval_after_flushing_in_ms to 1000, > TsFileProcessor A, whose last workMemtable was flush at time 2000 and > current workMemtable is null (i.e., no more wirite operations after last > flush), can be closed after time 3000. > > > But, i am confused whether to open timed flush seq/unseq memtable and > timed close TsFileProcessor by default, because they can save memory in > some user cases while they just waste cpu in other user cases. > > > Look forward to your reply. > > > [1] https://github.com/apache/iotdb/pull/3776 > [2] https://github.com/apache/iotdb/pull/3777 > > [3] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 > [4] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 > > > Thanks, > — > Haiming Zhu > > > > > -- 原始邮件 ------ > 发件人: > "dev" > > 发送时间: 2021年8月16日(星期一) 下午5:18 > 收件人: "IoTDB邮件列表" > 主题: Timed flush memtable and timed close TsFileProcessor > > > > Hi, > > > If we open a TsFileProcessor and then never write to it, this > TsFileProcessor will become a zombie because nothing can trigger a flush or > close operation. So, i am implementing timed flush memtable [1] and timed > close TsFileProcessor [2]. Timed flush memtable is used to flush inactive > memtables periodically according to their created time. Timed close > TsFileProcessor is used to close inactive > TsFileProcessors periodically according to how long they don't > process write operations. > > > I add 9 params to control above two functions, and these 9 params all can > be hot loaded by 'load configuration' command. > > > 1. enable_timed_flush_seq_memtable: whether to timed flush sequence > tsfiles' memtables. > 2. seq_memtable_flush_interval_in_ms: when a sequence memTable's created > time is older than current time minus this, the memtable is flushed to disk. > 3. seq_memtable_flush_check_interval_in_ms: the interval to check whether > the sequence memtables need flushing. > 4. enable_timed_flush_unseq_memtable: whether to timed flush unsequence > tsfiles' memtables. > 5. unseq_memtable_flush_interval_in_ms: when an unsequence memTable's > created time is older than current time minus this, the memtable is flushed > to disk. > 6. unseq_memtable_flush_check_interval_in_ms: the interval to check > whether the unsequence memtables need flushing. > 7. enable_timed_close_tsfile: whether to timed close TsfileProcessors. > 8. close_tsfile_interval_after_flushing_in_ms: when > TsfileProcessor's last working memtable flush time is older than current > time minus this and its working memtable is null, this TsfileProcessor will > be closed. > 9. close_tsfile_check_interval_in_ms: the interval to check > whether the TsfileProcessors need closing. > > > What do you think? > > > [1] > https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 > [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 > > > Thanks, > — > Haiming Zhu
Re: Timed flush memtable and timed close TsFileProcessor
Hi, I think the param number 3 and number 6 can be merged into one parameter. The checking interval can be the same value for both sequence and unsequence memtables. I don’t think any users will try to set different value for them. How do you think? Thanks, Haonan Hou > On Aug 19, 2021, at 6:04 PM, Jialin Qiao wrote: > > Hi, > >> I am confused whether to open timed flush seq/unseq memtable and timed > close TsFileProcessor by default, because they can save memory in some user > cases while they just waste cpu in other user cases. > > It could be enabled by default, both seq and unseq. > The 12 hours for memtable flush interval and close file interval is fine. > If it really kept 12 hours in memory, the CPU load should be very low, no > need to worry. > > Thanks, > — > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > Haiming Zhu 于2021年8月19日周四 下午4:37写道： > >> Hi, >> >> >> I have already implemented both issues [1][2]. >> >> >> Timed flush memtable [3] is used to flush inactive memtables periodically >> according to their created time. For example, if we set >> seq_memtable_flush_interval_in_ms to 1000, memtable A, created at time >> 2000, can be flushed after time 3000. >> >> >> Timed close TsFileProcessor [4] is used to close inactive TsFileProcessors >> periodically according to how long they don't process write operations. For >> example, if we set close_tsfile_interval_after_flushing_in_ms to 1000, >> TsFileProcessor A, whose last workMemtable was flush at time 2000 and >> current workMemtable is null (i.e., no more wirite operations after last >> flush), can be closed after time 3000. >> >> >> But, i am confused whether to open timed flush seq/unseq memtable and >> timed close TsFileProcessor by default, because they can save memory in >> some user cases while they just waste cpu in other user cases. >> >> >> Look forward to your reply. >> >> >> [1] https://github.com/apache/iotdb/pull/3776 >> [2] https://github.com/apache/iotdb/pull/3777 >> >> [3] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 >> [4] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 >> >> >> Thanks, >> — >> Haiming Zhu >> >> >> >> >> -- 原始邮件 -- >> 发件人: >> "dev" >> >> > 发送时间: 2021年8月16日(星期一) 下午5:18 >> 收件人: "IoTDB邮件列表"> >> 主题: Timed flush memtable and timed close TsFileProcessor >> >> >> >> Hi, >> >> >> If we open a TsFileProcessor and then never write to it, this >> TsFileProcessor will become a zombie because nothing can trigger a flush or >> close operation. So, i am implementing timed flush memtable [1] and timed >> close TsFileProcessor [2]. Timed flush memtable is used to flush inactive >> memtables periodically according to their created time. Timed close >> TsFileProcessor is used to close inactive >> TsFileProcessors periodically according to how long they don't >> process write operations. >> >> >> I add 9 params to control above two functions, and these 9 params all can >> be hot loaded by 'load configuration' command. >> >> >> 1. enable_timed_flush_seq_memtable: whether to timed flush sequence >> tsfiles' memtables. >> 2. seq_memtable_flush_interval_in_ms: when a sequence memTable's created >> time is older than current time minus this, the memtable is flushed to disk. >> 3. seq_memtable_flush_check_interval_in_ms: the interval to check whether >> the sequence memtables need flushing. >> 4. enable_timed_flush_unseq_memtable: whether to timed flush unsequence >> tsfiles' memtables. >> 5. unseq_memtable_flush_interval_in_ms: when an unsequence memTable's >> created time is older than current time minus this, the memtable is flushed >> to disk. >> 6. unseq_memtable_flush_check_interval_in_ms: the interval to check >> whether the unsequence memtables need flushing. >> 7. enable_timed_close_tsfile: whether to timed close TsfileProcessors. >> 8. close_tsfile_interval_after_flushing_in_ms: when >> TsfileProcessor's last working memtable flush time is older than current >> time minus this and its working memtable is null, this TsfileProcessor will >> be closed. >> 9. close_tsfile_check_interval_in_ms: the interval to check >> whether the TsfileProcessors need closing. >> >> >> What do you think? >> >> >> [1] >> https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 >> [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 >> >> >> Thanks, >> — >> Haiming Zhu >
Re: Timed flush memtable and timed close TsFileProcessor
Hi, Param seq_memtable_flush_check_interval_in_ms and unseq_memtable_flush_check_interval_in_ms aim to make timed flush memtable configuration more flexible. If users don't have this requirement, i will merge both params into one param memtable_flush_check_interval_in_ms. Thanks, ?? Haiming Zhu -- -- ??: "dev" https://github.com/apache/iotdb/pull/3776 >> [2] https://github.com/apache/iotdb/pull/3777 >> >> [3] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 >> [4] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 >> >> >> Thanks, >> ?? >> Haiming Zhu >> >> >> >> >> -- -- >> ??: >> "dev" >> >> https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 >> [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 >> >> >> Thanks, >> ?? >> Haiming Zhu >
