[jira] [Commented] (IOTDB-4022) Adjust the storage_group_report_threshold according to memory for writing
[ https://issues.apache.org/jira/browse/IOTDB-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616878#comment-17616878 ] yanze chen commented on IOTDB-4022: --- 更新测试： 吞吐(point/s)，写入内存分配=8980481489 || ||aligned_seq_config.properties||seq_config.properties||unseq_config.properties||aligned_unseq_config.properties|| |storage_group_report_threshold=16777216|8912483.19|17433312.42/ 16564385.56|3345125.65 |半个小时只跑了10%| |write_memory_vriation_report_proportion=0.001|13742844.78|16521936.57 /13984814.52|5082498.77 | | 吞吐(point/s)，总内存分配=100M || ||seq_config.properties|| |storage_group_report_threshold=16777216|写入一部分后OOM| |write_memory_vriation_report_proportion=0.001|写入失败，reject| > Adjust the storage_group_report_threshold according to memory for writing > - > > Key: IOTDB-4022 > URL: https://issues.apache.org/jira/browse/IOTDB-4022 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: yanze chen >Priority: Major > Labels: pull-request-available > Attachments: 完整实验结果.md > > > Currently, we have a parameter called storage_group_report_threshold, which > means when a storage group report its memory usage, it should accumulate a > memory change of this number. > If this parameter is too small, may report frequently, which may decrease the > insertion performance. > If this parameter is too large, may report not in time, which may cause OOM. > > We need to change this parameter according to the memory allocated for > writing. > > (1) remove storage_group_report_threshold > (2) Add a new parameter write_memory_vriation_report_proportion > > To rel/0.13 and master -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4022) Adjust the storage_group_report_threshold according to memory for writing
[ https://issues.apache.org/jira/browse/IOTDB-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577697#comment-17577697 ] yanze chen commented on IOTDB-4022: --- h2. 测试环境 IOTDB：私有云 16核32G 1T的硬盘，默认配置，根据日志看到写入内存分配=2993055334 benchmark：周测配置aligned_seq_config.properties、unseq_config.properties h2. 测试结果 || ||aligned_seq_config.properties||unseq_config.properties|| |storage_group_report_threshold=16777216|2209.05 second|1598.57 second| |write_memory_vriation_report_proportion=0.001|2361.42 second|1632.44 second| |write_memory_vriation_report_proportion=0.01|2190.96 second| | > Adjust the storage_group_report_threshold according to memory for writing > - > > Key: IOTDB-4022 > URL: https://issues.apache.org/jira/browse/IOTDB-4022 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: yanze chen >Priority: Major > Labels: pull-request-available > > Currently, we have a parameter called storage_group_report_threshold, which > means when a storage group report its memory usage, it should accumulate a > memory change of this number. > If this parameter is too small, may report frequently, which may decrease the > insertion performance. > If this parameter is too large, may report not in time, which may cause OOM. > > We need to change this parameter according to the memory allocated for > writing. > > (1) remove storage_group_report_threshold > (2) Add a new parameter write_memory_vriation_report_proportion > > To rel/0.13 and master -- This message was sent by Atlassian Jira (v8.20.10#820010)
