[jira] [Created] (IOTDB-5867) Build device time index when validating tsfiles after compaction
周沛辰 created IOTDB-5867: -- Summary: Build device time index when validating tsfiles after compaction Key: IOTDB-5867 URL: https://issues.apache.org/jira/browse/IOTDB-5867 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 Fix For: 1.1.1 In order to strengthen the accuracy of compaction, all tsfiles will be checked for time range of devices to determine whether there is overlap. If it is fileTimeIndex, it needs to be upgraded to deviceTimeIndex. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5317) Migrating Region failed when the RegionGroup is disabled
[ https://issues.apache.org/jira/browse/IOTDB-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzao Dan reassigned IOTDB-5317: -- Assignee: Yongzao Dan (was: Xinyu Tan) > Migrating Region failed when the RegionGroup is disabled > > > Key: IOTDB-5317 > URL: https://issues.apache.org/jira/browse/IOTDB-5317 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster >Affects Versions: master branch >Reporter: 刘珍 >Assignee: Yongzao Dan >Priority: Blocker > Attachments: image-2022-12-29-17-16-00-294.png, > image-2022-12-29-17-17-26-853.png, log_confignode_all.log, mig.conf, > screenshot-1.png > > > m_1229_15c9231 > 3副本1C4D，迁移region的Id=1 ，这个region的信息是1个副本是Running，1个是Unkown，1个是ReadOnly，迁移失败。 > 1.迁移开始前，查看region信息 > !image-2022-12-29-17-16-00-294.png|width=998,height=433! > 2.迁移命令 > ./sbin/start-cli.sh -h 172.20.70.4 -e "migrate region 1 from 1 to 3" > 命令返回成功 > 3.实际迁移失败，ConfigNode日志报错 > !image-2022-12-29-17-17-26-853.png|width=1286,height=291! > 迁移失败后，再执行一遍命令，会报目标节点已存在迁移region，实际上，并不存在。 > !screenshot-1.png|width=1022,height=481! > 测试环境：私有云3期 > COMMON配置 > schema_replication_factor=3 > data_replication_factor=3 > DataNode配置 > MAX_HEAP_SIZE="20G" > MAX_DIRECT_MEMORY_SIZE="6G" > dn_max_connection_for_internal_service=300 > ConfigNode配置 > MAX_HEAP_SIZE="20G" > MAX_DIRECT_MEMORY_SIZE="6G" > 运行BM写入数据，全部写入成功。 > 再执行stop datanode和set system to readonly on local的操作。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4593) [Remove-DataNode] Removing nodes writes data
[
https://issues.apache.org/jira/browse/IOTDB-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzao Dan reassigned IOTDB-4593:
--
Assignee: Yongzao Dan (was: Xinyu Tan)
> [Remove-DataNode] Removing nodes writes data
>
>
> Key: IOTDB-4593
> URL: https://issues.apache.org/jira/browse/IOTDB-4593
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 1.1.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Yongzao Dan
>Priority: Blocker
> Attachments: image-2022-10-10-13-36-14-475.png,
> image-2023-03-08-11-29-52-352.png, image-2023-03-08-11-30-38-559.png,
> image-2023-03-08-11-30-51-220.png, image-2023-03-08-11-33-49-278.png,
> more_dev.conf, screenshot-1.png
>
>
> m_0930_2a30316
> 问题描述：
> 缩容datanode，{color:#DE350B}*节点置为Removing状态，但是在继续接受写入*{color}（benchmark运行1小时，执行缩容，*耗时3小时*，缩容完成）：
> 2022-10-08 13:23:54,686 [pool-20-IoTDB-DataNodeInternalRPC-Processor-148]
> INFO o.a.i.c.conf.CommonConfig:305 - *Set system mode from Running to
> Removing*.
> Removing状态后（create 207个tsfile），
> !image-2022-10-10-13-36-14-475.png!
> 测试环境
> 1. 192.168.10.71-76 6台物理机 48cpu 384GB
> 3C : 192.168.10.72 , 73,74
> 5D : 192.168.10.72 , 73,74 , 75 , 76
> benchmark：192.168.10.71
> ConfigNode配置参数
> MAX_HEAP_SIZE="8G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> connection_timeout_ms=12
> DataNode配置参数
> MAX_HEAP_SIZE="256G"
> MAX_DIRECT_MEMORY_SIZE="32G"
> connection_timeout_ms=12
> max_connection_for_internal_service=200
> max_waiting_time_when_insert_blocked=60
> query_timeout_threshold=3600
> 2. benchmark 配置文件见附件
> GROUP_NUMBER=10
> DEVICE_NUMBER=5
> SENSOR_NUMBER=600
> IS_OUT_OF_ORDER=false
> OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0
> CLIENT_NUMBER=100
> LOOP=100
> BATCH_SIZE_PER_WRITE=100
> 3. 运行1小时，ip72缩容
> liuzhen@fit-72:/data/mpp_test/m_0930_2a30316/datanode$ cat
> 1008_test_remove_1h.sh
> sleep 1h
> /data/mpp_test/m_0930_2a30316/datanode/sbin/start-cli.sh -h 192.168.10.72 -e
> "show cluster" > 1008_3c5d_bef_remove.out
> /data/mpp_test/m_0930_2a30316/datanode/sbin/start-cli.sh -h 192.168.10.72 -e
> "show regions" >> 1008_3c5d_bef_remove.out
> /data/mpp_test/m_0930_2a30316/datanode/sbin/remove-datanode.sh
> "192.168.10.72:6667" >> 1008_3c5d_1hour_remove_ip72.out
> 4. ip72 的日志见机器上的备份
> /data/mpp_test/m_0930_2a30316/datanode/logs_bm_1h_remove_ip72
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5866) Optimize the QueryTerminator usage when 1C1D situation
Gaofei Cao created IOTDB-5866: - Summary: Optimize the QueryTerminator usage when 1C1D situation Key: IOTDB-5866 URL: https://issues.apache.org/jira/browse/IOTDB-5866 Project: Apache IoTDB Issue Type: Improvement Reporter: Gaofei Cao Assignee: Gaofei Cao Attachments: image-2023-05-12-10-31-26-785.png, image-2023-05-12-10-38-48-532.png 1）In 1C1D situation, InternalDataServiceClientManager will also invoke RPC due to the usage of SimpleQueryTerminator. !image-2023-05-12-10-31-26-785.png|width=500,height=99! 2) In 1C1D query situation, syncTerminate() method takes 2.7% time, which can be optimized. !image-2023-05-12-10-38-48-532.png|width=610,height=345! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5865) Fix route balancing not trigger bug
Yongzao Dan created IOTDB-5865: -- Summary: Fix route balancing not trigger bug Key: IOTDB-5865 URL: https://issues.apache.org/jira/browse/IOTDB-5865 Project: Apache IoTDB Issue Type: Bug Reporter: Yongzao Dan Assignee: Yongzao Dan The route balancing will not be triggered if the leader election is faster than ConfigNode persist new created RegionGroup. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5849) [Atmos][WAL]Got a low performance after we changed default parameters for wal part
[ https://issues.apache.org/jira/browse/IOTDB-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721961#comment-17721961 ] Qingxin Feng commented on IOTDB-5849: - !image-2023-05-12-08-46-06-097.png! > [Atmos][WAL]Got a low performance after we changed default parameters for wal > part > -- > > Key: IOTDB-5849 > URL: https://issues.apache.org/jira/browse/IOTDB-5849 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Engine >Affects Versions: 1.1.1 >Reporter: Qingxin Feng >Assignee: Qingxin Feng >Priority: Major > Attachments: image-2023-05-08-14-14-41-839.png, > image-2023-05-08-14-18-54-663.png, image-2023-05-12-08-46-06-097.png > > > After we changed default parameters for wal part,we got a low performance > when we using wal mode as SYNC > !image-2023-05-08-14-14-41-839.png|width=644,height=278! > [https://github.com/apache/iotdb/commit/f77f3fd5726c044edb07f45017652db1ac73a76f] > > And this change is not friendly to some scenarios,like below: > !image-2023-05-08-14-18-54-663.png|width=548,height=186! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5818) [Atmos][Compaction]Cross_space compaction of Aligned timeseries is stucked
[ https://issues.apache.org/jira/browse/IOTDB-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721687#comment-17721687 ] 周沛辰 commented on IOTDB-5818: Cross space compaction of aligned timeseries is recovered after this pr merged in. !image-2023-05-11-16-42-10-694.png|width=1089,height=331! > [Atmos][Compaction]Cross_space compaction of Aligned timeseries is stucked > -- > > Key: IOTDB-5818 > URL: https://issues.apache.org/jira/browse/IOTDB-5818 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Compaction >Affects Versions: 1.1.0 >Reporter: Qingxin Feng >Assignee: 周沛辰 >Priority: Major > Labels: pull-request-available > Fix For: 1.1.1 > > Attachments: image-2023-04-25-15-01-43-840.png, > image-2023-05-09-15-40-32-907.png, image-2023-05-11-16-42-10-694.png > > > Cross_space compaction of Aligned timeseries is stucked since 0419 c97c0dc > was merged > !image-2023-04-25-15-01-43-840.png|width=646,height=214! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5859) Compaction error when using Version as first sort dimension
[ https://issues.apache.org/jira/browse/IOTDB-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721683#comment-17721683 ] 周沛辰 commented on IOTDB-5859: There are two main types of error: !image-2023-05-11-16-24-11-808.png|width=1128,height=559! !image-2023-05-11-16-24-42-712.png|width=1136,height=563! > Compaction error when using Version as first sort dimension > --- > > Key: IOTDB-5859 > URL: https://issues.apache.org/jira/browse/IOTDB-5859 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jinrui Zhang >Assignee: 周沛辰 >Priority: Major > Labels: pull-request-available > Fix For: 1.1.1 > > Attachments: image-2023-05-11-16-24-11-808.png, > image-2023-05-11-16-24-42-712.png > > > In current implementation of compaction, the default sort dimension is file > version when selecting compaction tasks. It will lead to compaction error in > tsfile load scenario. It is because the TsFile with higher version may not > has greater timestamp when the TsFile is loaded by tools. > > Solution: > change the sort dimension from file version to timestamp. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5864) Print failed to install snapshot warn log while restarting
[ https://issues.apache.org/jira/browse/IOTDB-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyu Tan reassigned IOTDB-5864: Assignee: Song Ziyang (was: Xinyu Tan) > Print failed to install snapshot warn log while restarting > -- > > Key: IOTDB-5864 > URL: https://issues.apache.org/jira/browse/IOTDB-5864 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Yuan Tian >Assignee: Song Ziyang >Priority: Major > Attachments: image-2023-05-11-15-28-57-190.png, > image-2023-05-11-15-34-54-296.png > > > The write throughput is also very low all the time after restarting. > !image-2023-05-11-15-28-57-190.png! > !image-2023-05-11-15-34-54-296.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5859) Compaction error when using Version as first sort dimension
[
https://issues.apache.org/jira/browse/IOTDB-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721679#comment-17721679
]
周沛辰 commented on IOTDB-5859:
*Description*
Starting from version 1.0, using the load function to import data and then
compact them will result in overlapped data in sequential files.
*Reason*
After version 1.0, using the load function may cause files with the same
timestamp to appear in the system, and the version corresponding to the
sequential file with the same timestamp is incremented, and the version of the
file under a time partition is unique. Compaction use {{version}} as first sort
dimension, resulting in an incorrect order.
*Solution*
Use {{timestamp}} as first sort dimension and {{version}} as second sort
dimension.
> Compaction error when using Version as first sort dimension
> ---
>
> Key: IOTDB-5859
> URL: https://issues.apache.org/jira/browse/IOTDB-5859
> Project: Apache IoTDB
> Issue Type: Improvement
>Reporter: Jinrui Zhang
>Assignee: 周沛辰
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.1
>
>
> In current implementation of compaction, the default sort dimension is file
> version when selecting compaction tasks. It will lead to compaction error in
> tsfile load scenario. It is because the TsFile with higher version may not
> has greater timestamp when the TsFile is loaded by tools.
>
> Solution:
> change the sort dimension from file version to timestamp.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5864) Print failed to install snapshot warn log while restarting
Yuan Tian created IOTDB-5864: Summary: Print failed to install snapshot warn log while restarting Key: IOTDB-5864 URL: https://issues.apache.org/jira/browse/IOTDB-5864 Project: Apache IoTDB Issue Type: Bug Reporter: Yuan Tian Assignee: Xinyu Tan Attachments: image-2023-05-11-15-28-57-190.png The write throughput is also very low all the time after restarting. !image-2023-05-11-15-28-57-190.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
