Hi Xiangdong, I come from the StreamPipes community and I also think it's a great idea to work together. We have already integrated IoTDB as a data sink in StreamPipes. Users can model their analytics pipelines and write the results directly into IoTDB. The next step would be to also get data from IoTDB, then we can read data, do stream calculations and write the results back.
We have already integrated several adapters for databases (e.g. InfluxDB, mysql, ...) [1]. I would like to add such an adapter for IoTDB. Usually we distinguish between dataset and datastream adapters. With a data set adapter we poll the data once from the database and stream it through the pipeline. With data stream adapters we start the adapter and continuously read and process events. I think data set integration should be straightforward, but what about the data stream adapter? Is it possible to subscribe to changes in IoTDB? In [2] it is described that a user must add a trigger. Does this trigger have to be set in the database or can a client subscribe to changes? Philipp [1] https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/streampipes-connect-adapter/src/main/java/org/apache/streampipes/connect/adapters <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/streampipes-connect-adapter/src/main/java/org/apache/streampipes/connect/adapters> [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues <https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues> > On 20. Mar 2020, at 10:36, Julian Feinauer <[email protected]> > wrote: > > Hi Xiangdong, > > very nice to share it here! > Looking forward to Prometheus and Streampipes! > Andi f I can support with DBCP (or commons pool2 or whatever, I'm happy to > help!). > > Julian > > Am 20.03.20, 02:55 schrieb "Xiangdong Huang" <[email protected]>: > > Hi all, > > I come from IoTDB community. > Following Christofer's suggestion, I'd like to share the process of the > integration work between IoTDB and some other Apache projects to let users > manage time-series data easier. > > 1. PLC4X (done) > Under the help of Julian and other guys, PLC4X has provided an example to > write data directly to IoTDB [1]. > > 2. Prometheus (just beginning) > Prometheus is a popular data collection and event alert system for many > applications (but maybe not so popular for IoT). > We'd like to integrate Prometheus with IoTDB, by replacing Prometheus's > data store with IoTDB [2]. > > 3. Flink and RocketMQ (in-progress) > Both Flink and rocketMQ are message queues and in many applications, they > are the entrances of data and IoTDB is behind them [3] [4] [5]. > > 4. MiniFi (just beginning) > MiniFi is a dataflow management system. We'd like to integrate IoTDB with > it to allow: write data from a processor to IoTDB, and consume data from > IoTDB to other processors [6]. > > 5. Streampipes (just beginning) > We'd like to add trigger function in IoTDB to allow publish alerts, or do > some stream calculation. A current possible solution is integrating with > StreamPipes [8]. > > IoTDB also begin to integrate with some other projects (which are irrelated > to IoT) but make it friendly to use IoTDB. like: > > 6. Calcite (almost done) > Calcite provides a Standard SQL language to IoTDB, which make it easy to > use. A pr is open and ask for code review [7]. > > 7. Zeppelin (in-progress) > Zeppelin gives us a web-based GUI to let users operate IoTDB interactively > [9]. > > 8. DataBase Connection Pool (just beginning) > with a DBCP, developers do not need to write too many codes in their > business logic codes. > > Those are what we can consider. > We also would like to hear more ideas to make it easier to use IoTDB in IoT > applications. > Welcome to join us if you are interested in some integration ideas. > > [1] > > https://github.com/apache/plc4x/tree/develop/plc4j/examples/hello-integration-iotdb > [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-519 > [3] https://issues.apache.org/jira/browse/COMDEV-350 > [4] https://issues.apache.org/jira/browse/IOTDB-560 > [5] https://github.com/apache/incubator-iotdb/pull/817 > [6] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-518 > [7] https://github.com/apache/incubator-iotdb/pull/902 > [8] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516 > [9] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-515 > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
