Hi Philipp, Thanks for your recognition of our design.
To use the feature, adding a specific StreamPipes sink within the IoTDB code base is indeed a way. We will also consider adding support for the APIs in the client. We still need some time to develop the feature, so now we can’t see the code for the specific sink running. Once we have completed the development, I will inform here. Please look forward to the new feature! Steve Philipp Zehnder <[email protected]> 于2020年4月2日周四 下午2:54写道： > > Hi Steven, > > I think the design looks very good and I am looking forward to use the > feature. > > I have one more question. Where is the code for the specific sink running? > Do we have to add a specific StreamPipes sink within the IoTDB code base or > can this API be used in the client? > > > Philipp > > On 2. Apr 2020, at 04:00, Steve Yurong Su <[email protected]> wrote: > > Hi Philipp, > > I am currently working on [IOTDB-516]. > > The data sink module in my proposal is actually designed for forwarding > events. > When users want to write a program to push data to somewhere after the > trigger processing is done, > the data sink module could help. By implementing the interfaces > provided by the data sink module, > triggers could push the data to brokers, or directly push the data to > a specific stream processing engine (e.g., StreamPipes). > We would implement interfaces for some data sinks, and then users only > need to override data-related methods > to implement data forwarding. > > In addition, the triggers would provide Java hooks which allow users > to get the data changes and > to register a callback to process the changes. > > Therefore, the adapter could be built like this: > 1. subscribe to data changes by registering a trigger in IoTDB > 2. forward data by implementing the data sink APIs > > I don't know if I've answered your question. What do you think of the design? > :) > > Steve > > Philipp Zehnder <[email protected]> 于2020年4月1日周三 上午5:02写道： > > > Hi Xiangdong, > > yes I think this feature is exactly what we need. > > I also had a look at the proposal in the comments and it looks very > interesting. > > I have one question regarding the sinks. Do you also plan to integrate a > programming API (e.g. in Java) or do you plan to rely on broker technologies? > For us it would be good to have a Java API, which registers the query and a > callback receiving the events. > Alternatively, we could use a message broker to forward the events into the > StreamPipes adapter, but then we would depend on another service. > What do you think about that? > > Philipp > > > On 23. Mar 2020, at 15:17, Xiangdong Huang <[email protected]> wrote: > > Hi Philipp, > > That is really really big good news! > > I think data set integration should be straightforward, but what about > > the data stream adapter? Is it possible to subscribe to changes in IoTDB? > > Yes the data set adapter is straightforward. > > As for the data stream adapter.... > > Does this trigger have to be set in the database or can a client > > subscribe to changes? > > We are doing that! Have a look about issue [1]. It is called trigger in > IoTDB. > In our design, if a user registers a trigger for a time series and claims > to send new data points to somewhere (e.g., Streampipes), > then, you can get the changes of data on that timeseries in stream way. > It is just a design and we have not implement it now. > How do you think about this? > > [1] > https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Philipp Zehnder <[email protected]> 于2020年3月23日周一 下午8:17写道： > > Hi Xiangdong, > > I come from the StreamPipes community and I also think it's a great idea > to work together. > We have already integrated IoTDB as a data sink in StreamPipes. Users can > model their analytics pipelines and write the results directly into IoTDB. > The next step would be to also get data from IoTDB, then we can read data, > do stream calculations and write the results back. > > We have already integrated several adapters for databases (e.g. InfluxDB, > mysql, ...) [1]. I would like to add such an adapter for IoTDB. > Usually we distinguish between dataset and datastream adapters. With a > data set adapter we poll the data once from the database and stream it > through the pipeline. > With data stream adapters we start the adapter and continuously read and > process events. > I think data set integration should be straightforward, but what about the > data stream adapter? Is it possible to subscribe to changes in IoTDB? > > In [2] it is described that a user must add a trigger. Does this trigger > have to be set in the database or can a client subscribe to changes? > > Philipp > > [1] > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/streampipes-connect-adapter/src/main/java/org/apache/streampipes/connect/adapters > < > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters/streampipes-connect-adapter/src/main/java/org/apache/streampipes/connect/adapters > > > [2] > https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues > < > https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516?filter=allopenissues > > > > On 20. Mar 2020, at 10:36, Julian Feinauer <[email protected]> > > wrote: > > > Hi Xiangdong, > > very nice to share it here! > Looking forward to Prometheus and Streampipes! > Andi f I can support with DBCP (or commons pool2 or whatever, I'm happy > > to help!). > > > Julian > > Am 20.03.20, 02:55 schrieb "Xiangdong Huang" <[email protected]>: > > Hi all, > > I come from IoTDB community. > Following Christofer's suggestion, I'd like to share the process of > > the > > integration work between IoTDB and some other Apache projects to let > > users > > manage time-series data easier. > > 1. PLC4X (done) > Under the help of Julian and other guys, PLC4X has provided an > > example to > > write data directly to IoTDB [1]. > > 2. Prometheus (just beginning) > Prometheus is a popular data collection and event alert system for > > many > > applications (but maybe not so popular for IoT). > We'd like to integrate Prometheus with IoTDB, by replacing > > Prometheus's > > data store with IoTDB [2]. > > 3. Flink and RocketMQ (in-progress) > Both Flink and rocketMQ are message queues and in many applications, > > they > > are the entrances of data and IoTDB is behind them [3] [4] [5]. > > 4. MiniFi (just beginning) > MiniFi is a dataflow management system. We'd like to integrate IoTDB > > with > > it to allow: write data from a processor to IoTDB, and consume data > > from > > IoTDB to other processors [6]. > > 5. Streampipes (just beginning) > We'd like to add trigger function in IoTDB to allow publish alerts, > > or do > > some stream calculation. A current possible solution is integrating > > with > > StreamPipes [8]. > > IoTDB also begin to integrate with some other projects (which are > > irrelated > > to IoT) but make it friendly to use IoTDB. like: > > 6. Calcite (almost done) > Calcite provides a Standard SQL language to IoTDB, which make it easy > > to > > use. A pr is open and ask for code review [7]. > > 7. Zeppelin (in-progress) > Zeppelin gives us a web-based GUI to let users operate IoTDB > > interactively > > [9]. > > 8. DataBase Connection Pool (just beginning) > with a DBCP, developers do not need to write too many codes in their > business logic codes. > > Those are what we can consider. > We also would like to hear more ideas to make it easier to use IoTDB > > in IoT > > applications. > Welcome to join us if you are interested in some integration ideas. > > [1] > > https://github.com/apache/plc4x/tree/develop/plc4j/examples/hello-integration-iotdb > > [2] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-519 > [3] https://issues.apache.org/jira/browse/COMDEV-350 > [4] https://issues.apache.org/jira/browse/IOTDB-560 > [5] https://github.com/apache/incubator-iotdb/pull/817 > [6] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-518 > [7] https://github.com/apache/incubator-iotdb/pull/902 > [8] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-516 > [9] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-515 > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > ......................................................... > M. Sc. Philipp Zehnder > Wissenschaftlicher Mitarbeiter | Research Scientist > Information Process Engineering (IPE) > > FZI Forschungszentrum Informatik > Haid-und-Neu-Str. 10–14 > 76131 Karlsruhe, Germany > Tel.: +49 721 9654-805 > Fax: +49 721 9654-806 > > [email protected] > https://www.fzi.de/mitarbeiter/philipp-zehnder > > ......................................................... > FZI Forschungszentrum Informatik > Stiftung des bürgerlichen Rechts > Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe > Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. > Marius Zöllner > Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus > ......................................................... > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
