Supporting Similarity Search in IoTDB: Scenarios, SQL and Result Format
Hi all: I hope to let IoTDB support similar research by adding the indexing mechanism. Similarity search is one of the most important directions in the field of time series, and some users have shown their requirements for similarity search. Different from the existing query conditions in IoTDB, similarity search takes a sequence as input, and aim to find a list of similar sequences from the database efficiently. The similarity between the two sequences depends on their Euclidean distance or other distance functions. The similarity index techniques are widely used for speeding up the similarity search. I demonstrate two industrial scenarios as well as the SQL and the query result format. The detailed SQL and result format can be found in [1]. Case 1: Erythromycin Fermentation The pharmaceutical factory has many fermentors, each of which produces erythromycin batch by batch. For each batch, the factory will monitor some measurements in the fermentation process, like glucose feeding rate (Glu), carbon dioxide exit rate (CER), and pH value. Researchers have found that the glucose feeding rate (Glu) is critical to the final erythromycin output. Therefore, analysts want to build a similarity index (e.g. RTree+PAA [2]) for Glu sequences on all fermenters and all batches. After building the index, the analyst inputs a Glu sequence, finds batches with similar Glu curves, and makes further analysis. Case 2: Extreme Operating Gust A manufacturer has many wind turbines, each of which continuously monitors the state of the turbine itself and the surrounding environment, such as wind speed, wind direction, generator power, etc. The analyst hopes to build an index (e.g. ELB index [3]) for the speed series of a certain turbine. After building the index, the analyst inputs an EOG pattern (Extreme Operating Gust), finds all results in speed series for further process control, fault diagnosis and predictive maintenance. Any suggestions are welcomed. If it makes sense, we could add SQL formats to the IoTDB as the first step of the IoTDB index mechanism. [1] https://cwiki.apache.org/confluence/display/IOTDB/Supporting+Similarity+Search%3A+Scenarios%2C+SQL+and+Result+Format [2] R-trees: a dynamic index structure for spatial searching [3] Matching Consecutive Subpatterns over Streaming Time Series Best, 康荣 Kang Rong Ph.D. Student School of Software, Tsinghua University, Beijing, China (+86) 188-1030-5401 kang...@mails.tsinghua.edu.cn
Re: Embdedded IoTDB Grafana Endpoint
Hi, I dislike the idea of having spring in the server as well. +1, could we combine the small handler with our RestAPI? Thanks, -- Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 -原始邮件- 发件人: "Julian Feinauer" 发送时间: 2020-11-06 22:38:50 (星期五) 收件人: "dev@iotdb.apache.org" 抄送: 主题: Re: Embdedded IoTDB Grafana Endpoint Hey, yes, to do it clean we could do it directly without Spring. Then ist just a very small handler. Similar to what we already use optionally in the prometheus module. I dislike the idea of having spring in the server as well. Julian Am 06.11.20, 12:03 schrieb "Xiangdong Huang" : Hi, what do you think of an option to run the SimpleJson Grafana Connector directly in IoTDBs Server Context (could be disbaled by default). it is ok. But I think we need to prune some dependencies of Spring... It it too large now.. (I remember it is ~ 20MB to add such a module) Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Julian Feinauer 于2020年11月6日周五 下午6:13写道： Hi folks, as we just discussed the prometheus integration, what do you think of an option to run the SimpleJson Grafana Connector directly in IoTDBs Server Context (could be disbaled by default). This would make it way easier for users to just „look“ aat their IoTDB Data. WDYT? Julian
IoTDB Travis CI is back to normal
Hi, recently our Travis CI is not stable because there are some tests failure because of "ports not closed" or "mergeTasks not finished".. After some fixes [1], [2] and [3], now I believe the Travis CI is back to normal, WHICH MEANS if you see CI failure on your PR, please double-check why... For existing PRs, either re-run the failed CI or merge with the master to get a stable CI result. [1] https://github.com/apache/iotdb/pull/1976 [2] https://github.com/apache/iotdb/pull/1967 [3] https://github.com/apache/iotdb/pull/1978 Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Re: Embdedded IoTDB Grafana Endpoint
Hey, yes, to do it clean we could do it directly without Spring. Then ist just a very small handler. Similar to what we already use optionally in the prometheus module. I dislike the idea of having spring in the server as well. Julian Am 06.11.20, 12:03 schrieb "Xiangdong Huang" : Hi, > what do you think of an option to run the SimpleJson Grafana Connector directly in IoTDBs Server Context (could be disbaled by default). it is ok. But I think we need to prune some dependencies of Spring... It it too large now.. (I remember it is ~ 20MB to add such a module) Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Julian Feinauer 于2020年11月6日周五 下午6:13写道： > Hi folks, > > as we just discussed the prometheus integration, what do you think of an > option to run the SimpleJson Grafana Connector directly in IoTDBs Server > Context (could be disbaled by default). > This would make it way easier for users to just „look“ aat their IoTDB > Data. > > WDYT? > > Julian >
Re: Iotdb integrated Prometheus
Nice! I want this feature for about one year!! Will review the design document tonight. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Houliang Qi 于2020年11月6日周五 下午5:51写道： > Nice work! > This will enrich the ecology of IoTDB. > > > Thanks, > --- > Houliang Qi > > > On 11/6/2020 17:45，王艳虹<1101967...@qq.com> wrote： > Hi,all > > > The following is the design of iotdb integrated Prometheus. > > https://cwiki.apache.org/confluence/display/IOTDB/Iotdb+integrated+Prometheus > > > As mentioned in > https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-519?filter=allopenissues, > IoTDB is a highly efficient time series database.Prometheus is a monitoring > and alerting toolkit, which supports collecting data from other systems, > servers, and IoT devices, saving data into a DB, visualizing data and > provides some query APIs. > Prometheus allows users to use their database rather than just Prometheus > DB for storing time series databases.So we should integrate Prometheus, so > that we can provide the function of collecting data. > > > If you have any comments，looking forward to your reply. > > > Thanks, > Yanhong Wang
Re: Embdedded IoTDB Grafana Endpoint
Hi, > what do you think of an option to run the SimpleJson Grafana Connector directly in IoTDBs Server Context (could be disbaled by default). it is ok. But I think we need to prune some dependencies of Spring... It it too large now.. (I remember it is ~ 20MB to add such a module) Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Julian Feinauer 于2020年11月6日周五 下午6:13写道： > Hi folks, > > as we just discussed the prometheus integration, what do you think of an > option to run the SimpleJson Grafana Connector directly in IoTDBs Server > Context (could be disbaled by default). > This would make it way easier for users to just „look“ aat their IoTDB > Data. > > WDYT? > > Julian >
Embdedded IoTDB Grafana Endpoint
Hi folks, as we just discussed the prometheus integration, what do you think of an option to run the SimpleJson Grafana Connector directly in IoTDBs Server Context (could be disbaled by default). This would make it way easier for users to just „look“ aat their IoTDB Data. WDYT? Julian
Re:Iotdb integrated Prometheus
Nice work! This will enrich the ecology of IoTDB. Thanks, --- Houliang Qi On 11/6/2020 17:45<1101967...@qq.com> wrote?? Hi,all The following is the design of iotdb integrated Prometheus. https://cwiki.apache.org/confluence/display/IOTDB/Iotdb+integrated+Prometheus As mentioned inhttps://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-519?filter=allopenissues, IoTDB is a highly efficient time series database.Prometheus is a monitoring and alerting toolkit, which supports collecting data from other systems, servers, and IoT devices, saving data into a DB, visualizing data and provides some query APIs. Prometheus allows users to use their database rather than just Prometheus DB for storing time series databases.So we should integrate Prometheus, so that we can provide the function of collecting data. If you have any comments??looking forward to your reply. Thanks, Yanhong Wang
SDT Encoding and Query Design
Hi, The following is a more detailed design[1] of SDT encoding and query in IoTDB. The encoding and query methods will consult OSISoft Pi's design. The document first explains Pi's approach to encoding and query, then shows the modification of PageWriter, PageReader, SDTEncoder, SDTDecoder. Please let me know if you have any questions or recommendations. Thank you for your time and consideration! [1] https://cwiki.apache.org/confluence/display/IOTDB/SDT+encoding Best, Haimei
Iotdb integrated Prometheus
Hi,all The following is the design of iotdb integrated Prometheus. https://cwiki.apache.org/confluence/display/IOTDB/Iotdb+integrated+Prometheus As mentioned inhttps://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-519?filter=allopenissues, IoTDB is a highly efficient time series database.Prometheus is a monitoring and alerting toolkit, which supports collecting data from other systems, servers, and IoT devices, saving data into a DB, visualizing data and provides some query APIs. Prometheus allows users to use their database rather than just Prometheus DB for storing time series databases.So we should integrate Prometheus, so that we can provide the function of collecting data. If you have any comments??looking forward to your reply. Thanks, Yanhong Wang
