[
https://issues.apache.org/jira/browse/IOTDB-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720490#comment-17720490
]
周沛辰 commented on IOTDB-5836:
----------------------------
报错的两条序列正好都是采用 FREQ 的编码方式，而采用其他编码方式的序列则不存在此问题（解码报 BufferUnderFlow 异常）。FREQ
编码方式是有损编码，当经过多次编解码的时候误差会增大，导致解码的位移量存在错误，因此，猜测是 FREQ
编码算法本身存在缺陷导致跨空间合并在对一层顺序文件进行解码时报错。
> [Compaction] Meet errors in cross space compaction.Caused by:
> java.nio.BufferUnderflowException: null
> -----------------------------------------------------------------------------------------------------
>
> Key: IOTDB-5836
> URL: https://issues.apache.org/jira/browse/IOTDB-5836
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Compaction, mpp-cluster
> Affects Versions: 1.1.0
> Reporter: 刘珍
> Assignee: 周沛辰
> Priority: Major
> Attachments: 0427_aligned_compress.conf,
> 0427_aligned_compress_unseq.conf, image-2023-05-08-20-17-31-168.png,
> image-2023-05-08-20-18-11-646.png, image-2023-05-08-20-19-19-673.png,
> image-2023-05-08-20-19-33-158.png, insert.sh
>
>
> 测试版本： iotdb master 0427_ed8d539
> 日志路径：192.168.130.2 16C32G /data2/i_m_0427_ed8d539/logs
> 问题描述：
> 2023-04-27 19:51:27,391 [pool-16-IoTDB-Compaction-1] ERROR
> o.a.i.d.e.c.e.t.CrossSpaceCompactionTask:251 - root.test.g_0-2 [Compaction]
> Meet errors in cross space compaction.
> java.lang.RuntimeException: java.nio.BufferUnderflowException
> at
> org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.perform(FastCompactionPerformer.java:152)
> at
> org.apache.iotdb.db.engine.compaction.execute.task.CrossSpaceCompactionTask.doCompaction(CrossSpaceCompactionTask.java:160)
> at
> org.apache.iotdb.db.engine.compaction.execute.task.AbstractCompactionTask.start(AbstractCompactionTask.java:80)
> at
> org.apache.iotdb.db.engine.compaction.schedule.CompactionWorker.run(CompactionWorker.java:65)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.BufferUnderflowException: null
> at java.nio.Buffer.nextGetIndex(Buffer.java:524)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:138)
> at org.apache.iotdb.tsfile.utils.BitReader.next(BitReader.java:67)
> at org.apache.iotdb.tsfile.utils.BitReader.next(BitReader.java:36)
> at
> org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.decodeIndex(FreqDecoder.java:129)
> at
> org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.loadBlock(FreqDecoder.java:86)
> at
> org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.readDouble(FreqDecoder.java:45)
> at
> org.apache.iotdb.tsfile.read.reader.page.ValuePageReader.writeColumnBuilderWithNextBatch(ValuePageReader.java:306)
> at
> org.apache.iotdb.tsfile.read.reader.page.AlignedPageReader.getAllSatisfiedData(AlignedPageReader.java:226)
> at
> org.apache.iotdb.tsfile.read.reader.chunk.AlignedChunkReader.readPageData(AlignedChunkReader.java:320)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.element.PageElement.deserializePage(PageElement.java:96)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.element.PointElement.<init>(PointElement.java:35)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.reader.PointPriorityReader.addNewPage(PointPriorityReader.java:175)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.checkAndCompactOverlapPage(SeriesCompactionExecutor.java:341)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactWithOverlapPages(SeriesCompactionExecutor.java:307)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactPages(SeriesCompactionExecutor.java:236)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactWithOverlapChunks(SeriesCompactionExecutor.java:167)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactChunks(SeriesCompactionExecutor.java:147)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.AlignedSeriesCompactionExecutor.compactFiles(AlignedSeriesCompactionExecutor.java:97)
> at
> org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.AlignedSeriesCompactionExecutor.execute(AlignedSeriesCompactionExecutor.java:84)
> at
> org.apache.iotdb.db.engine.compaction.execute.task.subtask.FastCompactionPerformerSubTask.call(FastCompactionPerformerSubTask.java:141)
> at
> org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.compactAlignedSeries(FastCompactionPerformer.java:199)
> at
> org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.perform(FastCompactionPerformer.java:133)
> ... 8 common frames omitted
> 测试流程
> 1. start-standalone.sh 启动1C1D
> confignode-env.sh
> MAX_HEAP_SIZE="2G"
> datanode-env.sh
> MAX_HEAP_SIZE="16G"
> MAX_DIRECT_MEMORY_SIZE="4G"
> iotdb-common.properties
> series_slot_num=1000
> default_storage_group_level=2
> 2.创建2个dev，12序列/dev，DOUBLE类型，指定不同的压缩方式
> 不同编码：
> dev定义如下：
> create database root.test.g_0;
> create ALIGNED timeseries root.test.g_0.d_0(s_0 DOUBLE compressor=SNAPPY,s_1
> DOUBLE compressor=LZ4,s_2 DOUBLE compressor=GZIP,s_3 DOUBLE
> compressor=ZSTD,s_4 DOUBLE compressor=LZMA2,s_5 DOUBLE
> compressor=UNCOMPRESSED,s_6 DOUBLE compressor=LZ4,s_7 DOUBLE
> compressor=GZIP,s_8 DOUBLE compressor=ZSTD,s_9 DOUBLE compressor=LZMA2,s_10
> DOUBLE compressor=SNAPPY,s_11 DOUBLE compressor=UNCOMPRESSED);
> create ALIGNED timeseries root.test.g_0.d_1(s_0 DOUBLE compressor=SNAPPY,s_1
> DOUBLE ENCODING=RLE compressor=LZ4,s_2 DOUBLE ENCODING=TS_2DIFF
> compressor=GZIP,s_3 DOUBLE ENCODING=GORILLA compressor=ZSTD,s_4 DOUBLE
> ENCODING=FREQ compressor=LZMA2,s_5 DOUBLE ENCODING=CHIMP
> compressor=UNCOMPRESSED,s_6 DOUBLE ENCODING=SPRINTZ compressor=LZ4,s_7 DOUBLE
> ENCODING=RLBE compressor=GZIP,s_8 DOUBLE compressor=ZSTD,s_9 DOUBLE
> compressor=LZMA2,s_10 DOUBLE compressor=SNAPPY,s_11 DOUBLE
> compressor=UNCOMPRESSED)
> 每个序列写入960个点。见insert.sh 先写d_0，再写d_1,需要编辑一下脚本中的dev名字。
> 3. 再创建998个dev，用bm写入数据
> 运行如下脚本，创建dev
> #!/bin/bash
> for i in {2..999}
> do
> ./sbin/start-cli.sh -e "create ALIGNED timeseries root.test.g_0.d_$i(s_0
> DOUBLE compressor=SNAPPY,\
> s_1 DOUBLE ENCODING=RLE compressor=LZ4,\
> s_2 DOUBLE ENCODING=TS_2DIFF compressor=GZIP,\
> s_3 DOUBLE ENCODING=GORILLA compressor=ZSTD,\
> s_4 DOUBLE ENCODING=FREQ compressor=LZMA2,\
> s_5 DOUBLE ENCODING=CHIMP compressor=UNCOMPRESSED,\
> s_6 DOUBLE ENCODING=SPRINTZ compressor=LZ4,\
> s_7 DOUBLE ENCODING=RLBE compressor=GZIP,\
> s_8 DOUBLE compressor=ZSTD,\
> s_9 DOUBLE compressor=LZMA2,\
> s_10 DOUBLE compressor=SNAPPY,\
> s_11 DOUBLE compressor=UNCOMPRESSED) "
> done
> 运行附件中的配置，写入数据：
> 先运行：0427_aligned_compress.conf
> 再运行：0427_aligned_compress_unseq.conf
> 大概2小时30分钟后，出现上述异常。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
