[jira] [Created] (IOTDB-5265) [fast compaction] One tenth of the devices are "ALL_DELETED"，fast compaction is slow
刘珍 created IOTDB-5265: - Summary: [fast compaction] One tenth of the devices are "ALL_DELETED"，fast compaction is slow Key: IOTDB-5265 URL: https://issues.apache.org/jira/browse/IOTDB-5265 Project: Apache IoTDB Issue Type: Improvement Components: mpp-cluster Reporter: 刘珍 Assignee: 周沛辰 Attachments: image-2022-12-22-10-05-22-860.png master 1219 4a59a2c 100 设备 * 6 序列共600序列，2亿点/序列，顺序和乱序各50%。 对比2个场景的合并性能： delete 10个设备的数据 没有delete操作 有如下2个问题， 问题1： cross_performer=fast配置：没有delete数据的场景，合并耗时最长，比默认都长。 问题2： cross_performer=fast inner_unseq_performer=fast inner_seq_performer=fast 配置 没有delete数据的场景，反而合并的快。 !image-2022-12-22-10-05-22-860.png! 测试环境： 1. 192.168.10.72 48CPU384GB 关合并， time_partition_interval=6048000 生成测试数据。 配置文件见附件，1.conf运行完，cli flush，再运行2.conf，cli flush。 2.对比合并性能 compaction_write_throughput_mb_per_sec=1048576 （合并不限流） time_partition_interval=6048000 query_timeout_threshold=3600 delete 10个设备的数据： cli -e "delete from root.test.g_0.d_0.*;" cli -e "delete from root.test.g_0.d_10.*;" cli -e "delete from root.test.g_0.d_20.*;" cli -e "delete from root.test.g_0.d_30.*;" cli -e "delete from root.test.g_0.d_40.*;" cli -e "delete from root.test.g_0.d_50.*;" cli -e "delete from root.test.g_0.d_60.*;" cli -e "delete from root.test.g_0.d_70.*;" cli -e "delete from root.test.g_0.d_80.*;" cli -e "delete from root.test.g_0.d_90.*;" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5264) NumpyTablet support insert with None
Haonan Hou created IOTDB-5264: - Summary: NumpyTablet support insert with None Key: IOTDB-5264 URL: https://issues.apache.org/jira/browse/IOTDB-5264 Project: Apache IoTDB Issue Type: New Feature Reporter: Haonan Hou -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5264) NumpyTablet support insert with None
[ https://issues.apache.org/jira/browse/IOTDB-5264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haonan Hou reassigned IOTDB-5264: - Assignee: Haonan Hou > NumpyTablet support insert with None > > > Key: IOTDB-5264 > URL: https://issues.apache.org/jira/browse/IOTDB-5264 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Haonan Hou >Assignee: Haonan Hou >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5261) The number of clients in client pool of AsyncIoTConsensusServiceClient can not be changed through dn_max_connection_for_internal_service
[ https://issues.apache.org/jira/browse/IOTDB-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Houliang Qi reassigned IOTDB-5261: -- Assignee: Houliang Qi > The number of clients in client pool of AsyncIoTConsensusServiceClient can > not be changed through dn_max_connection_for_internal_service > > > Key: IOTDB-5261 > URL: https://issues.apache.org/jira/browse/IOTDB-5261 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Server >Affects Versions: 1.0.0 >Reporter: Houliang Qi >Assignee: Houliang Qi >Priority: Major > Attachments: image-2022-12-21-17-16-13-037.png > > > The number of clients in the client pool of AsyncIoTConsensusServiceClient > can not be changed through _dn_max_connection_for_internal_service_ > we change the _dn_max_connection_for_internal_service_ to 500, however it not > take effect. > _!image-2022-12-21-17-16-13-037.png!_ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5263) Optimize the cross compaction file selection and execution
[ https://issues.apache.org/jira/browse/IOTDB-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17650798#comment-17650798 ] Jinrui Zhang commented on IOTDB-5263: - We will do this optimization after bug fix in current implementation > Optimize the cross compaction file selection and execution > -- > > Key: IOTDB-5263 > URL: https://issues.apache.org/jira/browse/IOTDB-5263 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jinrui Zhang >Assignee: Jinrui Zhang >Priority: Major > Attachments: image-2022-12-21-18-06-40-580.png > > > In current implementation, when selecting the `overlapped` seq files for one > specific unseq file, one seq file will always be selected even though it > doesn't has overlap with the unseq file. See the sample below. > !image-2022-12-21-18-06-40-580.png|width=718,height=220! > That is, when selecting seq files for `3`, file-1 will be selected even > though there is no overlap between 1 and 3. It is because we need to find a > target file for 3 in current cross compaction implementation. Or there will > be overlapped seq files generated after cross compaction. > We need to do the optimization for it. > # Only select the seq files which has overlapped with target unseq file. > # change the implementation of the cross compaction to find the target seq > file and decrease unnecessary file write. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5263) Optimize the cross compaction file selection and execution
Jinrui Zhang created IOTDB-5263: --- Summary: Optimize the cross compaction file selection and execution Key: IOTDB-5263 URL: https://issues.apache.org/jira/browse/IOTDB-5263 Project: Apache IoTDB Issue Type: Improvement Reporter: Jinrui Zhang Assignee: Jinrui Zhang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5262) [load tsfile]The result of load tsfile is unexpected in both schema and data
changxue created IOTDB-5262:
---
Summary: [load tsfile]The result of load tsfile is unexpected in
both schema and data
Key: IOTDB-5262
URL: https://issues.apache.org/jira/browse/IOTDB-5262
Project: Apache IoTDB
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: changxue
Assignee: Haonan Hou
Attachments: load-tsfile_allnodes-log.tar.gz
[load tsfile]The result of load tsfile is unexpected in both schema and data
environment: 3C3D cluster rel/1.0
reproduction:
1. put the directories of data/datanode/data/sequence/root.test.g_0/ into the
same dirs (/data2/sequence/root.test.g_0) on 3 computers, which are 44,45 and
46.
dirs 1 is to 44
dirs 2 3 are to 45
dirs 4 5 are to 46
2. start the load-tsfile.sh the same time
/load-tsfile.sh -h iotdb-46 -u root -pw root -f /data2/sequence/root.test.g_0
--verify false --onSuccess none --sgLevel 2
actual result:
{code}
IoTDB> count devices root.**
+--+
|count(devices)|
+--+
|29|
+--+
Total line number = 1
It costs 0.024s
IoTDB> count timeseries root.**
+-+
|count(timeseries)|
+-+
|28001|
+-+
Total line number = 1
It costs 0.025s
IoTDB> select count(s_0) from root.test.g_0.d_0;
++
|count(root.test.g_0.d_0.s_0)|
++
| 1338671|
++
Total line number = 1
It costs 0.179s
{code}
expect:
{code}
IoTDB> count devices root.**
+--+
|count(devices)|
+--+
|100|
+--+
Total line number = 1
It costs 0.024s
IoTDB> count timeseries root.**
+-+
|count(timeseries)|
+-+
|10|
+-+
Total line number = 1
It costs 0.025s
IoTDB> select count(s_0) from root.test.g_0.d_0;
++
|count(root.test.g_0.d_0.s_0)|
++
| 2592000|
++
Total line number = 1
It costs 0.179s
{code}
问题：
为什么s_0的数据量比预期大很多？
和指定了 sgLevel 为 2 有关吗？
说明：
timeseries数目不对有可能是还有tsfile没有load，我kill了进程(46 node)。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5261) The number of clients in client pool of AsyncIoTConsensusServiceClient can not be changed through dn_max_connection_for_internal_service
Houliang Qi created IOTDB-5261: -- Summary: The number of clients in client pool of AsyncIoTConsensusServiceClient can not be changed through dn_max_connection_for_internal_service Key: IOTDB-5261 URL: https://issues.apache.org/jira/browse/IOTDB-5261 Project: Apache IoTDB Issue Type: Bug Components: Core/Server Affects Versions: 1.0.0 Reporter: Houliang Qi Attachments: image-2022-12-21-17-16-13-037.png The number of clients in the client pool of AsyncIoTConsensusServiceClient can not be changed through _dn_max_connection_for_internal_service_ we change the _dn_max_connection_for_internal_service_ to 500, however it not take effect. _!image-2022-12-21-17-16-13-037.png!_ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5260) Refactoring ClientManager API and Exception
Xinyu Tan created IOTDB-5260: Summary: Refactoring ClientManager API and Exception Key: IOTDB-5260 URL: https://issues.apache.org/jira/browse/IOTDB-5260 Project: Apache IoTDB Issue Type: Improvement Reporter: Xinyu Tan Assignee: Xinyu Tan * ClientManagerException is introduced to facilitate ClientManager users to distinguish borrowClient exception from other business exception. * remove purelyBorrowClient API to make ClientManager API clearer -- This message was sent by Atlassian Jira (v8.20.10#820010)
