[ https://issues.apache.org/jira/browse/IOTDB-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
刘珍 reassigned IOTDB-4593: ------------------------- Component/s: mpp-cluster Sprint: 2023-1-ConfigNode, 2023-2-Catalyst (was: 2023-1-ConfigNode) Assignee: Xinyu Tan (was: Gaofei Cao) > [Remove-DataNode] Removing nodes writes data > -------------------------------------------- > > Key: IOTDB-4593 > URL: https://issues.apache.org/jira/browse/IOTDB-4593 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster > Affects Versions: 0.14.0-SNAPSHOT > Reporter: 刘珍 > Assignee: Xinyu Tan > Priority: Major > Attachments: image-2022-10-10-13-36-14-475.png, > image-2023-03-08-11-29-52-352.png, image-2023-03-08-11-30-38-559.png, > image-2023-03-08-11-30-51-220.png, more_dev.conf, screenshot-1.png > > > m_0930_2a30316 > 问题描述: > 缩容datanode,{color:#DE350B}*节点置为Removing状态,但是在继续接受写入*{color}(benchmark运行1小时,执行缩容,*耗时3小时*,缩容完成): > 2022-10-08 13:23:54,686 [pool-20-IoTDB-DataNodeInternalRPC-Processor-148] > INFO o.a.i.c.conf.CommonConfig:305 - *Set system mode from Running to > Removing*. > Removing状态后(create 207个tsfile), > !image-2022-10-10-13-36-14-475.png! > 测试环境 > 1. 192.168.10.71-76 6台物理机 48cpu 384GB > 3C : 192.168.10.72 , 73,74 > 5D : 192.168.10.72 , 73,74 , 75 , 76 > benchmark:192.168.10.71 > ConfigNode配置参数 > MAX_HEAP_SIZE="8G" > schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus > data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus > schema_replication_factor=3 > data_replication_factor=3 > connection_timeout_ms=120000 > DataNode配置参数 > MAX_HEAP_SIZE="256G" > MAX_DIRECT_MEMORY_SIZE="32G" > connection_timeout_ms=120000 > max_connection_for_internal_service=200 > max_waiting_time_when_insert_blocked=600000 > query_timeout_threshold=36000000 > 2. benchmark 配置文件见附件 > GROUP_NUMBER=10 > DEVICE_NUMBER=50000 > SENSOR_NUMBER=600 > IS_OUT_OF_ORDER=false > OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0 > CLIENT_NUMBER=100 > LOOP=100 > BATCH_SIZE_PER_WRITE=100 > 3. 运行1小时,ip72缩容 > liuzhen@fit-72:/data/mpp_test/m_0930_2a30316/datanode$ cat > 1008_test_remove_1h.sh > sleep 1h > /data/mpp_test/m_0930_2a30316/datanode/sbin/start-cli.sh -h 192.168.10.72 -e > "show cluster" > 1008_3c5d_bef_remove.out > /data/mpp_test/m_0930_2a30316/datanode/sbin/start-cli.sh -h 192.168.10.72 -e > "show regions" >> 1008_3c5d_bef_remove.out > /data/mpp_test/m_0930_2a30316/datanode/sbin/remove-datanode.sh > "192.168.10.72:6667" >> 1008_3c5d_1hour_remove_ip72.out > 4. ip72 的日志见机器上的备份 > /data/mpp_test/m_0930_2a30316/datanode/logs_bm_1h_remove_ip72 -- This message was sent by Atlassian Jira (v8.20.10#820010)