Thanks! I think it cold be a very cool index. Due to linearity it would be pretty easy to translate regular queries to queries on the sdt signal which had way lower data points.
Or in some sense this is a visualisation optimized version of the time series. Really cool feature! Julian Holen Sie sich Outlook für Android<https://aka.ms/ghei36> ________________________________ From: Jialin Qiao <[email protected]> Sent: Saturday, October 3, 2020 5:04:20 AM To: [email protected] <[email protected]> Subject: Re: Support SDT compression Hi, Yes, it's lossy. Users need to config the tolerant error. Thanks, -- Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 > -----原始邮件----- > 发件人: "Julian Feinauer" <[email protected]> > 发送时间: 2020-10-03 02:09:44 (星期六) > 收件人: "[email protected]" <[email protected]> > 抄送: > 主题: Re: Support SDT compression > > Hi, > > I read the document (which is excellent, good work!) and it sounds very > interesting but as far as I understand the algorithm its lossy. > Is this true? > Or do I miss something? > > Thanks! > Julian > > Am 29.09.20, 12:04 schrieb "Jialin Qiao" <[email protected]>: > > Hi, > > Good summary~ > > > Page header needs to maintain a Map<segmentStartTime, count> > > It's better to keep the structure of PageHeader the same as 0.10. > You can store this information in the PageData. > > > Encoder, Decoder will take both a data column and a series. > > Try to provide interface with primitive data type. So a series should be > put by encodeFloatPoint(long time, float value) > > Besides, there are some more details need to consider: > > - How to store the endpoint of each segment > - In the SDT with timestamp encoding, how to store the timestamps? (Maybe > just using TS2_DIFF is fine) > > This is not a small change to IoTDB... > > Thanks, > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > -----原始邮件----- > > 发件人: "Haimei Guo" <[email protected]> > > 发送时间: 2020-09-29 15:19:49 (星期二) > > 收件人: [email protected] > > 抄送: > > 主题: Re: Support SDT compression > > > > Hi, > > > > Following is a summary of SDT's encoding and decoding implementation in > > IoTDB. > > > > - > > > > SDT is mainly to calculate the up and down slopes of the data to the > > segment starting point. If it is within the compression deviation CD > range, > > discard the data. If it exceeds the CD, the original data is stored > > - > > > > In IoTDB, the SDT can act as a new Encoding method. It works inside > each > > Page (PageWriter and PageReader). > > - > > > > Will support with and without timestamp encoding. > > - > > > > For without timestamps encoding, we will record the count of data > points > > in each segment in the page header. Page header needs to maintain a > > Map<segmentStartTime, count> > > > > Encoder will be changed to encode(long time, long value) > > Data buffer will be stored in each Encoder > > Decoder will be changed to getTime(), getXXValue() > > Encoder, Decoder will take both a data column and a series. > > > > > > If you have any question or comment, you are more than welcome to reply! > > > > > > Thank you, > > > > Haimei > > > > > > On Mon, Sep 28, 2020 at 1:20 PM Jialin Qiao > <[email protected]> > > wrote: > > > > > Hi Haimei, > > > > > > Good work! This doc is comprehensive :) > > > > > > As for the implementation in IoTDB, here are some points: > > > > > > (1) First, SDT could act as a new Encoding method in IoTDB. It works > > > inside each Page (PageWriter and PageReader). > > > (2) The interface of Encoder could be changed to encode(long time, XX > > > value). The interface of Decoder could be change to getTime(), > > > getXXValue(). Which is, the encoder and decoder is not only > responsible for > > > one data column but a series. This involves some reconstruction of the > > > Encoder and Decoder, the data buffers should be stored inside each > encoder. > > > (3) For the SDT without timestamps, we need to record the count of > each > > > segment. > > > (4) We could offer two encodings, SDT with timestamps and SDT without > > > timestamps. > > > > > > Thanks, > > > -- > > > Jialin Qiao > > > School of Software, Tsinghua University > > > > > > 乔嘉林 > > > 清华大学 软件学院 > > > > > > > -----原始邮件----- > > > > 发件人: "[email protected]" <[email protected]> > > > > 发送时间: 2020-09-25 11:56:16 (星期五) > > > > 收件人: dev <[email protected]> > > > > 抄送: > > > > 主题: Re: Support SDT compression > > > > > > > > Great work！ > > > > > > > > > > > > > > > > Thanks. > > > > > > > > Chao Wang > > > > BONC Ltd > > > > > > > > > > > > From: Eileen Guo > > > > Date: 2020-09-25 11:47 > > > > To: dev > > > > Subject: Support SDT compression > > > > Hi all, > > > > > > > > I've completed a design draft for supporting swinging door > compression. > > > > > > > > Jira: jira SDT link > > > > < > > > > https://issues.apache.org/jira/browse/IOTDB-890?filter=-4&jql=assignee%20in%20(haimeiguo)%20order%20by%20created%20DESC > > > > > > > > design doc: SDT design doc link > > > > < > > > > https://docs.google.com/document/d/1VeTwVsm4CkQSVR65bWw9pKg6gRdiDYUz0lBBhWXHl5A/edit?usp=sharing > > > > > > > > > > > > The doc explains SDT algorithm, compression and decompression > process, > > > > performance tests and SDT + IoTDB implementation and usage. > > > > > > > > There is still some question about where to use this algorithm. If > you > > > have > > > > any idea, welcome to comment. > > > > > > > > Thank you! > > > > Haimei Guo > > > >
