Hi, Good summary~
> Page header needs to maintain a Map<segmentStartTime, count> It's better to keep the structure of PageHeader the same as 0.10. You can store this information in the PageData. > Encoder, Decoder will take both a data column and a series. Try to provide interface with primitive data type. So a series should be put by encodeFloatPoint(long time, float value) Besides, there are some more details need to consider: - How to store the endpoint of each segment - In the SDT with timestamp encoding, how to store the timestamps? (Maybe just using TS2_DIFF is fine) This is not a small change to IoTDB... Thanks, -- Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 > -----原始邮件----- > 发件人: "Haimei Guo" <kel...@gmail.com> > 发送时间: 2020-09-29 15:19:49 (星期二) > 收件人: dev@iotdb.apache.org > 抄送: > 主题: Re: Support SDT compression > > Hi, > > Following is a summary of SDT's encoding and decoding implementation in > IoTDB. > > - > > SDT is mainly to calculate the up and down slopes of the data to the > segment starting point. If it is within the compression deviation CD range, > discard the data. If it exceeds the CD, the original data is stored > - > > In IoTDB, the SDT can act as a new Encoding method. It works inside each > Page (PageWriter and PageReader). > - > > Will support with and without timestamp encoding. > - > > For without timestamps encoding, we will record the count of data points > in each segment in the page header. Page header needs to maintain a > Map<segmentStartTime, count> > > Encoder will be changed to encode(long time, long value) > Data buffer will be stored in each Encoder > Decoder will be changed to getTime(), getXXValue() > Encoder, Decoder will take both a data column and a series. > > > If you have any question or comment, you are more than welcome to reply! > > > Thank you, > > Haimei > > > On Mon, Sep 28, 2020 at 1:20 PM Jialin Qiao <qj...@mails.tsinghua.edu.cn> > wrote: > > > Hi Haimei, > > > > Good work! This doc is comprehensive :) > > > > As for the implementation in IoTDB, here are some points: > > > > (1) First, SDT could act as a new Encoding method in IoTDB. It works > > inside each Page (PageWriter and PageReader). > > (2) The interface of Encoder could be changed to encode(long time, XX > > value). The interface of Decoder could be change to getTime(), > > getXXValue(). Which is, the encoder and decoder is not only responsible for > > one data column but a series. This involves some reconstruction of the > > Encoder and Decoder, the data buffers should be stored inside each encoder. > > (3) For the SDT without timestamps, we need to record the count of each > > segment. > > (4) We could offer two encodings, SDT with timestamps and SDT without > > timestamps. > > > > Thanks, > > -- > > Jialin Qiao > > School of Software, Tsinghua University > > > > 乔嘉林 > > 清华大学 软件学院 > > > > > -----原始邮件----- > > > 发件人: "runhus...@foxmail.com" <runhus...@foxmail.com> > > > 发送时间: 2020-09-25 11:56:16 (星期五) > > > 收件人: dev <dev@iotdb.apache.org> > > > 抄送: > > > 主题: Re: Support SDT compression > > > > > > Great work！ > > > > > > > > > > > > Thanks. > > > > > > Chao Wang > > > BONC Ltd > > > > > > > > > From: Eileen Guo > > > Date: 2020-09-25 11:47 > > > To: dev > > > Subject: Support SDT compression > > > Hi all, > > > > > > I've completed a design draft for supporting swinging door compression. > > > > > > Jira: jira SDT link > > > < > > https://issues.apache.org/jira/browse/IOTDB-890?filter=-4&jql=assignee%20in%20(haimeiguo)%20order%20by%20created%20DESC > > > > > > design doc: SDT design doc link > > > < > > https://docs.google.com/document/d/1VeTwVsm4CkQSVR65bWw9pKg6gRdiDYUz0lBBhWXHl5A/edit?usp=sharing > > > > > > > > > The doc explains SDT algorithm, compression and decompression process, > > > performance tests and SDT + IoTDB implementation and usage. > > > > > > There is still some question about where to use this algorithm. If you > > have > > > any idea, welcome to comment. > > > > > > Thank you! > > > Haimei Guo > >
