[ https://issues.apache.org/jira/browse/IOTDB-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17650101#comment-17650101 ]
周沛辰 commented on IOTDB-5165: ---------------------------- 这次合并报overlap的bug,是跨空间选文件的问题,当顺序文件的resource降级后,deviceEndTime就被放大了,是所有device的最大值,导致选文件的时候认为后续的顺序文件不再有overlap,就停止选择了。例如:顺序文件1 2,他们的d1.endTime分别是 100和200,乱序文件3的d1.endTime是150,按理应该选中 1 2 3,可是由于1文件被降级成FileTimeIndex,导致1的d1.endTime被扩大成200,在选择的时候因为150<200,就认为1后面的文件都没有重叠,就只选中1 3。总结:根本原因是选文件的时候没考虑到resource降级后获取到的deviceEndTime被放大了,导致后续文件选不上,所以合并后报overlap。 > [ compaction ] Failed to pass compaction validation > --------------------------------------------------- > > Key: IOTDB-5165 > URL: https://issues.apache.org/jira/browse/IOTDB-5165 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Compaction, mpp-cluster > Affects Versions: master branch, 1.0.0 > Reporter: 刘珍 > Assignee: 周沛辰 > Priority: Major > Labels: pull-request-available > Fix For: master branch, 1.0.1 > > Attachments: 1.conf, 10.conf, 2.conf, 3.conf, 4.conf, 5.conf, 6.conf, > 7.conf, 8.conf, 9.conf, run.sh, run_conf.sh > > > master 2022-12-09_a31441c > 合并失败,报错 > 2022-12-09 14:21:46,728 [pool-43-IoTDB-Compaction-8] ERROR > o.a.i.d.e.c.CompactionUtils:281 - root.test.g2_0 Device > root.test.g2_0.d_82215 {color:#DE350B}*is overlapped between file*{color} is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420064671-7-0-0.tsfile, > status: COMPACTING and file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420083777-8-0-0.tsfile, > status: COMPACTING, end time in file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420064671-7-0-0.tsfile, > status: COMPACTING is 1535568410000, start time in file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670420083777-8-0-0.tsfile, > status: COMPACTING is 1535558420000 > 2022-12-09 14:21:46,729 [pool-43-IoTDB-Compaction-8] ERROR > o.a.i.d.e.c.i.InnerSpaceCompactionTask:184 - {color:#DE350B}*Failed to pass > compaction validation*{color}, source files is: [file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444897033-4581-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444947849-4590-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444999954-4600-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445051784-4609-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445101595-4619-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445153290-4628-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445204996-4638-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445254210-4647-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445304094-4656-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445355765-4666-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445407476-4675-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445458633-4685-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445509050-4694-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445558911-4703-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445608483-4712-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445660518-4722-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445709301-4731-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445760226-4741-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445808744-4750-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445860031-4759-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445910381-4769-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670445959576-4778-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446011018-4787-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446062483-4797-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446113893-4806-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446165533-4816-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446216536-4825-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446267529-4835-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446318504-4844-0-0.tsfile, > status: COMPACTING, file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670446368398-4853-0-0.tsfile, > status: COMPACTING], target files is [file is > /data1/iotdb/m_1209_a31441c_fast1/./sbin/../data/datanode/data/sequence/root.test.g2_0/5/25/1670444897033-4581-1-0.tsfile, > status: CLOSED] > 2022-12-09 14:21:46,732 [pool-43-IoTDB-Compaction-8] ERROR > o.a.i.d.e.c.i.InnerSpaceCompactionTask:265 - root.test.g2_0-5 [Compaction] > Meet errors in inner space compaction. > java.lang.RuntimeException: Failed to pass compaction validation > at > org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask.doCompaction(InnerSpaceCompactionTask.java:188) > at > org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.start(AbstractCompactionTask.java:76) > at > org.apache.iotdb.db.engine.compaction.CompactionWorker.run(CompactionWorker.java:66) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > 测试流程 > 1. 私有云1期 172.16.2.3 8C32GB 1000GB HDD > 数据备份在 m_1207_a0b2c8c_fast1下的data > 如何生成这个数据备份: > IoTDB 配置文件 > ConfigNode配置 > MAX_HEAP_SIZE="2G" > DataNode配置 > MAX_HEAP_SIZE="20G" > MAX_DIRECT_MEMORY_SIZE="6G" > Common配置(因为1207的版本的bug,合并全部失败,生成的tsfile全部是0层文件,所以可以关合并,生成数据) > time_partition_interval=60480000000 > avg_series_point_number_threshold=10000 > 关合并 > config_node_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus > schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus > data_region_consensus_protocol_class=org.apache.iotdb.consensus.iot.IoTConsensus > 启动集群start-standalone.sh > BM在172.16.2.27 4C16GB /data/iotdb/benchmark_v1 > run.sh/run_conf.sh 放到/data/iotdb/benchmark_v1下 > 1.conf ~ 10.conf放到/data/iotdb/benchmark_v1/conf下, > 启动BM 运行run.sh , 约运行10小时 > 2. 把data文件夹copy到master 1209 a31441c对应的路径 > 配置参数同步骤1的集群,区别在于需要打开合并,显式设置 > cross_performer=fast > 启动1副本1C1D集群 > 查看合并日志 > 日志备份在 ip3的/data1/iotdb/m_1209_a31441c_fast1/logs -- This message was sent by Atlassian Jira (v8.20.10#820010)