Re: Change default time partition interval from 1 day to 1 week
Dear Qingxin Feng, The benefit is that time partition can help us isolate data by time. We change it back to 7 days because we find some problems when using 1 day. To conclude it, "too small is bad, how big depends on user needs". Best, -------- Haiming Zhu 冯 庆新 于2022年10月10日周一 21:23写道： > Dear Haiming Zhu, > > If only refer to the explanation below mailing list, "No time partition" > seems better. > So why is it 7 days?not 10 days. What experiments and analyses have you > done to support this change. > > Best Regards. > Qingxin Feng > > 发件人: Haiming Zhu<mailto:heimin...@gmail.com> > 发送时间: 2022年10月10日 20:03 > 收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org> > 主题: Re: Change default time partition interval from 1 day to 1 week > > Three reasons to do this: > >1. 1 day is too small to enable time partition, there're too few tsfiles >in one partition, which is bad for compaction. >2. Avoid wal accumulation caused by too many time partitions. >3. Avoid too many partition table entries and then trigger confignode >snapshot frequently. > > > > Best, > > Haiming Zhu > > > 冯 庆新 于2022年10月10日周一 17:46写道： > > > Hi, > > But “enable_partition” is still “false”,right? According to this it > > doesn't affect anyone who using default value in > > ‘iotdb-datanode.properties’ to using standalone version apache-iotdb. > > If someone uses the distributed version, it will affect newly compiled > and > > started instances,because it works by default in > > “iotdb-confignode.properties”. > > What is the expected impact? Degraded write performance? > > > > Best Regards. > > Thanks! > > > > 发件人: Haiming Zhu<mailto:heimin...@gmail.com> > > 发送时间: 2022年10月10日 15:06 > > 收件人: iotdb<mailto:dev@iotdb.apache.org> > > 主题: Change default time partition interval from 1 day to 1 week > > > > Hi, > > > > We want to change default time partition interval from 1 day to 1 week, > > including > > time_partition_interval_for_routing in the confignode and > > time_partition_interval_for_storage in the datanode. > > > > Best, > > > > Haiming Zhu > > > > > >
Re: Change default time partition interval from 1 day to 1 week
Three reasons to do this: 1. 1 day is too small to enable time partition, there're too few tsfiles in one partition, which is bad for compaction. 2. Avoid wal accumulation caused by too many time partitions. 3. Avoid too many partition table entries and then trigger confignode snapshot frequently. Best, Haiming Zhu 冯 庆新 于2022年10月10日周一 17:46写道： > Hi, > But “enable_partition” is still “false”,right? According to this it > doesn't affect anyone who using default value in > ‘iotdb-datanode.properties’ to using standalone version apache-iotdb. > If someone uses the distributed version, it will affect newly compiled and > started instances,because it works by default in > “iotdb-confignode.properties”. > What is the expected impact? Degraded write performance? > > Best Regards. > Thanks! > > 发件人: Haiming Zhu<mailto:heimin...@gmail.com> > 发送时间: 2022年10月10日 15:06 > 收件人: iotdb<mailto:dev@iotdb.apache.org> > 主题: Change default time partition interval from 1 day to 1 week > > Hi, > > We want to change default time partition interval from 1 day to 1 week, > including > time_partition_interval_for_routing in the confignode and > time_partition_interval_for_storage in the datanode. > > Best, > > Haiming Zhu > >
Change default time partition interval from 1 day to 1 week
Hi, We want to change default time partition interval from 1 day to 1 week, including time_partition_interval_for_routing in the confignode and time_partition_interval_for_storage in the datanode. Best, Haiming Zhu
Modify set system status command syntax
Hi, The syntax of set system status will change from > set system to readonly set system to writeable to > set system to readonly set system to running set system to error In cluster mode, we provide commands to set the local node status and set the cluster status, valid for the entire cluster by default. > set system to readonly on local set system to readonly on cluster Best, ---- Haiming Zhu School of Software, Tsinghua University
Re: [Discuss] enlarge default value of avg_series_point_number_threhold
I think both avg_series_point_number_threshold and memtable_size_threshold parameters can be calculated by IoTDB itself. On one hand, the time series number of one memtable should be considered for parameter avg_series_point_number_threshold. On the other hand, the memory size allocated for write and the number of storage groups should be considered for parameter memtable_size_threshold. Besides, it is an interesting and challenging issue to manage high frequency time series and low frequency time series, hot data and cold data. Best, Haiming Zhu Haonan Hou 于2022年5月17日周二 14:46写道： > Hi, > > As we know, we have a parameter avg_series_point_number_threshold=1, > which controls the maximum point number of one series in memtable can not > exceed the threshold. > The main purpose of this configuration is solve the problem that if the > number of point in one TVList is too large, it will take too much sorting > time on query and flush. > > However, we recently got many feedback about when import historical data > series by series, each time we insert a series with 10,000 points, then the > 10,000 points will form a tsfile. > As a result, there would be a lot of small tsfiles after the data imported > and the query efficiency is not as good as we expected. > > Therefore, I would like to change the default value of this parameter to > 100,000. Two reasons about it. One, change the threshold larger can improve > the efficiency of the scenario > above. Besides, it may not have any side effect to enlarge the number on > other application scenario. Someone may remember, this configuration was > actually introduced to IoTDB > before the memory control mechanism added. After we have the memory > control, this threshold almost cannot be reached except the importing > historical data scenario. > > This is the first step to solve the problem. We may need to find a more > smart strategy to replace this parameter in the future. > > How do you think? > > BR, > Haonan Hou > > >
Change semantics of TsFile filename
Hi, everyone
Currently, the filename format of each tsfile is
{file_created_time}-{version_id}-{inner_space_merge_num}-{cross_space_merge_num}.tsfile.
In one time partition, the order of tsfiles is guaranteed by the
version_id, for example, 1651825804093-2-0-0.tsfile is after
1651825804092-1-0-0.tsfile
The problem is that filename conflict may occur in the cross space
compaction and load scenes. In the cross space compaction, assuming there
exists 3-2-0-0.tsfile, 4-3-0-0.tsfile and 5-5-0-0.tsfile in the sequence
folder, if file 4-3-0-0.tsfile is selected, compaction cannot generate 3 or
more target files because only 2 version_id are left between 2 and 5, so
some big target files may be generated. In the load, assuming there exists
3-2-0-0.tsfile, 3-3-0-0.tsfile and 3-3-0-0.tsfile in the sequence folder,
no more sequence files cannot be loaded between 3-2-0-0.tsfile and
3-3-0-0.tsfile, they can only be loaded into the unsequence folder.
In response to these problems, the format won't be changed, but the meaning
of file_created_time and version_id will be different. Instead of
version_id, we use file_created_time to guarantee the order of tsfiles, and
if two tsfiles have the same file_created_time, then we use version_id to
guarantee the order. This semantics change may afftect query, compaction
and load module.
Hope for some suggestions.
Best,
Haiming Zhu
School of Software, Tsinghua University
朱海铭
清华大学 软件学院
New WAL Design
Hi everyone, I have finished the design of new WAL, here is the doc : https://cwiki.apache.org/confluence/x/2YqqCw. If you have any questions please let me know. Best, Haiming Zhu School of Software, Tsinghua University 朱海铭 清华大学 软件学院
Re: Enable wiki in Github
Hi, +1 :) Best, Haiming Zhu School of Software, Tsinghua University 朱海铭 清华大学 软件学院 钟文豪 于2021年9月23日周四 上午9:45写道： > Hi, > +1 > Thanks.zhongwenhao > > > > > > At 2021-09-21 21:26:21, "Xiangdong Huang" wrote: > >Hi, > > > >Using Github Wiki can indeed make it easy to update documents. > > > >> If we move the wiki from confluence to Github wiki, we still need to > >> sync the wiki documents in some place of ASF. > > > >As Github wiki is also a git codebase > >(https://github.com/YOUR_USERNAME/YOUR_REPOSITORY.wiki.git), > >if we enable it for iotdb (i.e., github.com/apache/iotdb.wiki.git), > >will ASF have a corresponding git codebase automatically? > > > >Best, > >--- > >Xiangdong Huang > >School of Software, Tsinghua University > > > > 黄向东 > >清华大学 软件学院 > > > >Willem Jiang 于2021年9月20日周一 上午10:12写道： > >> > >> Here is one concern we need to address. > >> Although we are using Github as our daily development platform, ASF > >> is still syncing the data to ASF infrastructure through email > >> forwarding or git sync. > >> In this way we still have a chance to commit the code through ASF infra. > >> If we move the wiki from confluence to Github wiki, we still need to > >> sync the wiki documents in some place of ASF. > >> > >> Willem Jiang > >> > >> Twitter: willemjiang > >> Weibo: 姜宁willem > >> > >> On Sun, Sep 19, 2021 at 4:55 PM Jialin Qiao > wrote: > >> > > >> > Hi, > >> > > >> > We have tried a lot about how to manage our documents (UserGuide, > >> > SystemDesign, ContributeGuide...). > >> > > >> > First, we manage these docs as many .md files in our repo, however, > update > >> > it is troublesome, need to submit a pr and the website update is not > in > >> > time. > >> > > >> > Next, we try the confluence, However, the editor of confluence is not > >> > friendly. md is good. > >> > > >> > Therefore, we try to use Github issues to manage these things. E.g., > join > >> > the community https://github.com/apache/iotdb/issues/1995. > >> > > >> > Finally, we realize that GitHub wiki may be the best place to manage > such > >> > documents, it is the nearest to our code, users do not need to go to > >> > another place (all in Github). As for the website, we could read the > .md > >> > files from GitHub wiki. > >> > > >> > What do you think? > >> > > >> > If everyone recognized it, I will submit an issue to the INFRA. > >> > > >> > Thanks, > >> > — > >> > Jialin Qiao > >> > School of Software, Tsinghua University > >> > > >> > 乔嘉林 > >> > 清华大学 软件学院 >
Discuss how to rename MOVE operator
Hi, MOVE operator is confusing, i thought it just moves some TsFile to the target directory, but it also unloads the TsFile from IoTDB and the data i moved cannot be searched any more. So, it think MOVE operator should be renamed and maybe UNLOAD is more suitable. What do you think? Best, Haiming Zhu School of Software, Tsinghua University 朱海铭 清华大学 软件学院
Discuss delete storage group SQL statement
Hi, After reviewing the code of delete storage group SQL statement, i have some questions about the implementation. First, when deleting several storage groups in one statement, the metadata in MManger will be deleted together after all storage groups are deleted. But, if there are concurrent write operations, the write operations will use old metadata, after the 'delete storage group' statement is executed successfully and the metadata is deleted from MManger, the data written in this period cannot be queried any more. Secondly, processorMap in StorageEngine class may need a ReentrantReadWriteLock. For example, if we execute delete storage group statement in cli and there are concurrent write operations to these storage groups. Current code first remove storage group from processorMap and then delete it, so other write operations will create a new VirtualStorageGroupManager in processorMap and the new VSGManager's recover method will conflict with the delete procedure, because tsfile may be truncated in recover and delete needs flushing memtable to the same tsfile. However, if we first delete a storage group and then remove it from processorMap, other write operations may create a new TsFileProcessor in the interval, making this delete statement incorrect. So, we need a ReentrantReadWriteLock to lock processorMap. Best, Haiming Zhu School of Software, Tsinghua University 朱海铭 清华大学 软件学院
Discuss delete storage group SQL statement
Hi, After reviewing the code of delete storage group SQL statement, i have some questions about the implementation. First, when deleting several storage groups in one statement, the metadata in MManger will be deleted together after all storage groups are deleted. But, if there are concurrent write operations, the write operations will use old metadata, after the 'delete storage group' statement is executed successfully and the metadata is deleted from MManger, the data written in this period cannot be queried any more. Secondly, processorMap in StorageEngine class may need a ReentrantReadWriteLock. For example, if we execute delete storage group statement in cli and there are concurrent write operations to these storage groups. Current code first remove storage group from processorMap and then delete it, so other write operations will create a new VirtualStorageGroupManager in processorMap and the new VSGManager's recover method will conflict with the delete procedure, because tsfile may be truncated in recover and delete needs flushing memtable to the same tsfile. However, if we first delete a storage group and then remove it from processorMap, other write operations may create a new TsFileProcessor in the interval, making this delete statement incorrect. So, we need a ReentrantReadWriteLock to lock processorMap. Best, Haiming Zhu School of Software, Tsinghua University 朱海铭 清华大学 软件学院
Re: Timed flush memtable and timed close TsFileProcessor
Hi, Param seq_memtable_flush_check_interval_in_ms and unseq_memtable_flush_check_interval_in_ms aim to make timed flush memtable configuration more flexible. If users don't have this requirement, i will merge both params into one param memtable_flush_check_interval_in_ms. Thanks, ?? Haiming Zhu ---- ??: "dev" https://github.com/apache/iotdb/pull/3776 [2]nbsp;https://github.com/apache/iotdb/pull/3777 [3]nbsp;https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [4]nbsp;https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?????? Haiming Zhu --nbsp;nbsp;-- ??: "dev" https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?????????? Haiming Zhu
Re: Timed flush memtable and timed close TsFileProcessor
Hi, I have already implemented both issues [1][2]. Timed flush memtable [3] is used to flush inactive memtables periodically according to their created time. For example, if we set seq_memtable_flush_interval_in_ms to 1000, memtable A, created at time 2000, can be flushed after time 3000. Timed close TsFileProcessor [4] is used to close inactive TsFileProcessors periodically according to how long they don't process write operations. For example, if we set close_tsfile_interval_after_flushing_in_ms to 1000, TsFileProcessor A, whose last workMemtable was flush at time 2000 and current workMemtable is null (i.e., no more wirite operations after last flush), can be closed after time 3000. But, i am confused whether to open timed flush seq/unseq memtable and timed close TsFileProcessor by default, because they can save memory in some user cases while they just waste cpu in other user cases. Look forward to your reply. [1]https://github.com/apache/iotdb/pull/3776 [2]https://github.com/apache/iotdb/pull/3777 [3]https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [4]https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?? Haiming Zhu ---- ??: "dev" https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?????? Haiming Zhu
Timed flush memtable and timed close TsFileProcessor
Hi, If we open a TsFileProcessor and then never write to it, this TsFileProcessor will become a zombie because nothing can trigger a flush or close operation. So, i am implementing timed flush memtable [1] and timed close TsFileProcessor [2]. Timed flush memtable is used to flush inactive memtables periodically according to their created time. Timed close TsFileProcessor is used to close inactive TsFileProcessorsperiodically according to how long they don't process write operations. I add 9 params to control above two functions, and these 9 params all can be hot loaded by 'load configuration' command. 1. enable_timed_flush_seq_memtable: whether to timed flush sequence tsfiles' memtables. 2. seq_memtable_flush_interval_in_ms: when a sequence memTable's created time is older than current time minus this, the memtable is flushed to disk. 3. seq_memtable_flush_check_interval_in_ms: the interval to check whether the sequencememtables need flushing. 4. enable_timed_flush_unseq_memtable: whether to timed flush unsequence tsfiles' memtables. 5. unseq_memtable_flush_interval_in_ms: when an unsequence memTable's created time is older than current time minus this, the memtable is flushed to disk. 6. unseq_memtable_flush_check_interval_in_ms: the interval to check whether the unsequencememtables need flushing. 7. enable_timed_close_tsfile: whether to timed close TsfileProcessors. 8.close_tsfile_interval_after_flushing_in_ms: when TsfileProcessor's last working memtable flush time is older than current time minus this and its working memtable is null, this TsfileProcessor will be closed. 9.close_tsfile_check_interval_in_ms: the interval to check whether the TsfileProcessorsneed closing. What do you think? [1]https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1496 [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1569 Thanks, ?? Haiming Zhu
