Re: Historical data from Ditto
Thanks for the very detailed analysis Xiangdong! Indeed a full and proper integration may be hard, but one could start with smaller steps and e.g. leave out the access management yet. Ditto even supports Access Management on each single point. I also wanted to share the github issue with you that the ditto community is using to discuss that point: https://github.com/eclipse/ditto/issues/545 Julian Am 11.01.20, 17:03 schrieb "Xiangdong Huang" : Hi, I have quickly read the document of Ditto. In my understanding, Ditto allow consume data from kafka or MQTT server to get the latest status of a Things (or, a device), and provides API to let users know the latest value of a device's feature (measurement concept in IoTDB). I think it is easy to persist the data into IoTDB meanwhile. But if we want to provide high write performance (e.g., more than 10 million data points per second), we may need to use batch_write api, which is a little complex. Another thing that need to be discussed is that as ditto does not have api for query historical data. we have to design and add one. The third is, it sounds that Ditto does not require K8s and Keycloak Authentication. I quickly searched Keycloak and find it is a solution for Identity and Access Management. Hmm... IoTDB has a user/role management module, but does not support Keycloak... I think currently the community lacks committers to develop this feature. But we can create an issue about that, and maybe some familiar guys can do it. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Xiangdong Huang 于2020年1月11日周六 下午10:05写道： > Hi, > > Thanks Lukas. I also need to take some time to familiarize with Ditto. > :-) > > > One important question for me would be the possibility to deploy Apache > IoTDB on Kubernetes > > Now we can deploy IoTDB as a docker container. > So I think it is ok to deploy it on K8S. But as it is a database, > you may have to use a network filesystem or run IoTDB on a certain node by > force. > > As for the Keycloak Authentication, I also need time to familiarize with > it.. > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Lukas Ott 于2020年1月11日周六 下午8:46写道： > >> Hi Julian, >> >> Great to hear from you. Just for information I was active on >> https://gitter.im/eclipse/ditto and raised same issues on stackoverflow >> (ended up in a mini PR for Ditto concerning Kubernetes API break) >> >> So lets see how we can bring these two systems closer together. :) >> >> One important question for me would be the possibility to deploy Apache >> IoTDB on Kubernetes (including Keycloak Authentication -> That is already >> included in Eclipse Ditto) >> >> Lukas >> >> Am Sa., 11. Jan. 2020 um 13:41 Uhr schrieb Julian Feinauer < >> j.feina...@pragmaticminds.de>: >> >> > Hi Lukas, >> > >> > What a coincidence! >> > Yesterday I met some of the ditto devs from Bosch and am currently >> > thinking about exactly that! >> > >> > So let me familiarize a bit more with ditto then everything its >> possible! >> > >> > Julian >> > >> > Von meinem Mobiltelefon gesendet >> > >> > >> > Ursprüngliche Nachricht >> > Von: Lukas Ott >> > Datum: Sa., 11. Jan. 2020, 13:36 >> > An: dev@iotdb.apache.org >> > Betreff: Historical data from Ditto >> > Hi Iotdb developers, >> > >> > First short introduction: I met Julian Feinhauer and saw the talk about >> > Apache IoTDB and then the following question came up: How can Apache >> IoTDB >> > support the Histical data from Eclipse Ditto? >> > >> > This issue has been raised on Eclipse Ditto code repository: >> > https://github.com/eclipse/ditto/issues/545 and i was wondering if >> there >> > are plans to support this. As one of the developers already mentioned >> IoTDB >> > as potential solution it would be awesome if IoTDB would provide a way >> to >> > safe Historical Data of Ditto. >> > >> > Additionally >> > >> > >> https://stackoverflow.com/questions/56993884/how-to-get-historical-sensor-data-of-a-registered-hono-device >> > Bosch IoT Suite seems to have an own way to safe historical data. >> > >> > Thanks in advance for your answers! >> > >> > Kind regards >> > Lukas >> > >> >
[jira] [Closed] (IOTDB-420) Avoid flush encoding task dying silently.
[ https://issues.apache.org/jira/browse/IOTDB-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jialin Qiao closed IOTDB-420. - Resolution: Fixed > Avoid flush encoding task dying silently. > - > > Key: IOTDB-420 > URL: https://issues.apache.org/jira/browse/IOTDB-420 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.10.0-SNAPSHOT >Reporter: Tian Jiang >Priority: Minor > Labels: Flush, pull-request-available > Fix For: 0.10.0-SNAPSHOT > > Time Spent: 20m > Remaining Estimate: 0h > > If a runtime exception is thrown in an encoding sub-task, it will die > silently and prevents the io task from ending. > To avoid this, the future of the encoding task should be got before that of > the io task is got. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-420) Avoid flush encoding task dying silently.
Tian Jiang created IOTDB-420: Summary: Avoid flush encoding task dying silently. Key: IOTDB-420 URL: https://issues.apache.org/jira/browse/IOTDB-420 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.10.0-SNAPSHOT Reporter: Tian Jiang Fix For: 0.10.0-SNAPSHOT If a runtime exception is thrown in an encoding sub-task, it will die silently and prevents the io task from ending. To avoid this, the future of the encoding task should be got before that of the io task is got. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-418) Refactor query process in IoTDB
[ https://issues.apache.org/jira/browse/IOTDB-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014809#comment-17014809 ] Jialin Qiao commented on IOTDB-418: --- [https://github.com/apache/incubator-iotdb/pull/713] > Refactor query process in IoTDB > --- > > Key: IOTDB-418 > URL: https://issues.apache.org/jira/browse/IOTDB-418 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently, readers in IoTDB are classified into sequence reader && unsequence > reader && series reader. > It's better to combine them together to get benefit from managing all > resources in one reader. > Therefore, raw data queries, aggregations, and group by query need to be > improved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (IOTDB-333) Storage group needs to divide by time range
[ https://issues.apache.org/jira/browse/IOTDB-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jialin Qiao closed IOTDB-333. - Fix Version/s: 0.10.0-SNAPSHOT Resolution: Fixed > Storage group needs to divide by time range > --- > > Key: IOTDB-333 > URL: https://issues.apache.org/jira/browse/IOTDB-333 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Kaifeng Xue >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0-SNAPSHOT > > Time Spent: 20m > Remaining Estimate: 0h > > Our storage group needs to divide by time range, which can support > distributed model better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[DISCUSS] When Release 0.10.0
Hi, It has been almost two months since the release tag of `release/0.9.0` was created on Nov 23, 2019. 1. New changes that have taken place: Refer to the temporary release note for `0.10.0-SNAPSHOT`, auto-generated by JIRA [1]. (about 16 bug fixes, 4 new features, 13 improvements) 2. Upcoming new changes: As far as I know, the develop team has been working towards two directions recently: 1. the distributed version of IoTDB 2. the refactor of both physical design and query processing Shall we withhold the release of 0.10.0 until the two upcoming changes are finished? What do you think? Regards, Lei Rui [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12322720=12346616
[jira] [Created] (IOTDB-419) Rename first/last aggregate query to first_value/last_value
Lei Rui created IOTDB-419: - Summary: Rename first/last aggregate query to first_value/last_value Key: IOTDB-419 URL: https://issues.apache.org/jira/browse/IOTDB-419 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-418) Refactor query process in IoTDB
[ https://issues.apache.org/jira/browse/IOTDB-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014782#comment-17014782 ] Jialin Qiao commented on IOTDB-418: --- [https://github.com/apache/incubator-iotdb/pull/692] > Refactor query process in IoTDB > --- > > Key: IOTDB-418 > URL: https://issues.apache.org/jira/browse/IOTDB-418 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Priority: Major > > Currently, readers in IoTDB are classified into sequence reader && unsequence > reader && series reader. > It's better to combine them together to get benefit from managing all > resources in one reader. > Therefore, raw data queries, aggregations, and group by query need to be > improved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-418) Refactor query process in IoTDB
Jialin Qiao created IOTDB-418: - Summary: Refactor query process in IoTDB Key: IOTDB-418 URL: https://issues.apache.org/jira/browse/IOTDB-418 Project: Apache IoTDB Issue Type: Improvement Reporter: Jialin Qiao Currently, readers in IoTDB are classified into sequence reader && unsequence reader && series reader. It's better to combine them together to get benefit from managing all resources in one reader. Therefore, raw data queries, aggregations, and group by query need to be improved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-417) [0.9.0] With German Locale it does NOT work as one test fails due to decimal comma vs decimal point
Lei Rui created IOTDB-417: - Summary: [0.9.0] With German Locale it does NOT work as one test fails due to decimal comma vs decimal point Key: IOTDB-417 URL: https://issues.apache.org/jira/browse/IOTDB-417 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.9.1 Reporter: Lei Rui Fix For: 0.10.0-SNAPSHOT Credit to Julian Feinauer. See [https://lists.apache.org/thread.html/rfef7f0035f28c3ee81ce32135e46e53774ce19b4af0bc06db426d4e3%40%3Cdev.iotdb.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-416) [0.9.1] ASF header format issues
Lei Rui created IOTDB-416:
-
Summary: [0.9.1] ASF header format issues
Key: IOTDB-416
URL: https://issues.apache.org/jira/browse/IOTDB-416
Project: Apache IoTDB
Issue Type: Improvement
Reporter: Lei Rui
see
[https://lists.apache.org/thread.html/209371ab5158fc61f2342115c1eef199f78555059dfb98bfdad70d72%40%3Cdev.iotdb.apache.org%3E]
Hi,
using `find **/src -type f -exec grep -i "copyright" {} \; -print | sort
-u`, many apache header format issues can be found..
e.g.,
1. showTTLPlan.java
2. all files for QueryMetricUI function...
3. All scala files are using /** so the blank space is two chars while it
is one char in all java files.
4. there are two blank spaces between "#" and content in all Python files
while it is one blank space in all sh files.
I do not vote -1 but at least we need to fix these issues in next release.
Best,
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] Upcoming Workshop "Industrial IoT wth Open Source" in Germany
Hey, nice, and thanks fort he offer, that’s of course Welcome : ) Julian Am 13.01.20, 05:10 schrieb "Xiangdong Huang" : Hi, I am one of the guys who are just reading the mailing list.. It is wonderful especially there is a tutorial about PLC4X+IoTDB. Let me know any help if you needed, slides, example projects, etc.. Julian, I will send you updated slides latter, which may include more content about "how to use IoTDB". Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Otto Fowler 于2020年1月13日周一 上午2:18写道： > Maybe you guys should start recording this things that are in Germany so > that people not in your immediate area can see them. > > This includes meetups etc. > > > > > On January 9, 2020 at 07:47:11, Julian Feinauer ( > j.feina...@pragmaticminds.de) wrote: > > Hi folks, > > I know that there are some people on these lists that are „just reading“ > and I know that some people just recently started to play around with PLC4X > (and I just recently got asked for our IoTDB slide deck from the > ApacheCON). > So I thought its probably interesting for some of you, that Chris and I > will host a Workshop at the Conference Building IoT 2020 ( > https://www.buildingiot.de/) in Germany for “Industrial IoT with Open > Source”: https://www.buildingiot.de/lecture.php?id=10856=0. > > Although we both try to answer as many questions as we can on the lists, I > guess this is a good opportunity to play around together with some time. > I think its even reasonable to hack around a bit on some (for us unknown) > controllers and give a bit of details about mspec, if interesting. > > Julian > > PS.: We don’t get money for promoting this, its really just an offer as > it’s a nice opportunity for us to spend time on talking / teaching about > the projects we love : ) >
