Hi Jialin, thanks for that info … so it seems as if IoTDB would have everything I’m looking for … but not just now. That’s ok, as I currently wouldn’t have the time to get working on this …. I’m just brainstorming for possibilities to solve problems I saw.
Thanks again and I’ll keep an eye out for hearing that these features are available. Chris From: Jialin Qiao <[email protected]> Date: Thursday, 20. October 2022 at 18:21 To: [email protected] <[email protected]> Subject: Re: Is there any form of hierarchical storage and processing possible with the new cluster mode? Hi Chris, Nice to hear from you! The architecture of the new Cluster version is ready in Chinese, we plan to do the translation and release the English version next month. The new cluster supports scalability in different layers: (1) ConfigNode or DataNode process (2) storage or schema engine (3) data or schema partition. So, it's supported now that we add a Cluster DataNode without data migration (just like you store data on leaf node, and add a computing node in the cloud). Besides, for the scenario you raised, we have two functions may help: (1) Data synchronization framework, which supports TsFile-sync or Operation-Sync from one IoTDB(single node or Cluster) to another. Currently, we are working on the TsFile-sync mode. But the Operation-Sync is the next step, and the operation sync we support Users to use a "sql" to specify sync which data. (2) Trigger: You could register some triggers, each trigger could collect some timeseries that could be matched by a pattern(e.g., root.sg.*.s1), and do the logic that user defined. The logic could be sending data to an upper IoTDB. These two functions are all under developing, we plan to merge the code into master and add UserGuide this month. Thanks, ――――――――――――――――― Jialin Qiao Apache IoTDB PMC Christofer Dutz <[email protected]> 于2022年10月20日周四 20:27写道： > > Hi, > > Als one of your previous Mentors I’m still closely following the project. > Still looking for ways to use it in my day job. > > Now I am thinking about, how to process manufacturing data. > > The problem is: In manufacturing a LOT of Data is produced. However, I would > say the further away from the Source the data is, the less important most of > it becomes. > > So, I am thinking of a tree-like structure where Most data is stored close to > the leaves of the tree (The Manufacturing machinery), because this is where I > would like to run predictive maintenance and optimization jobs and these > usually require a lot of data in a fine granularity. Most of this data is > however irrelevant for the higher levels on not needed in this granularity. > > But again, some data is important on higher levels (Like detecting patterns > in production, problem scenarios etc.) this can probably go up until some > Factory dashboard for management with just some KPIs and perhaps even > company-wide over multiple Factories. > > Here I’m thinking that an IoTDB node or even a leaf-node-cluster could store > all the data produced by leaf-nodes and only data that is requested from an > upstream instance is replicated there (Or even just virtually replicated, so > queries would be forwarded to the leaf-nodes). > > So, my question is: Does IoTDB support this sort of hierarchies? If not, is > something like this planned? > I do imagine, that doing this manually would be possible, where every branch > of the tree is a separate IoTDB node or cluster and then custom written code > replicates data based on queries. However I do imagine this should be a > pretty universal concept and therefore everyone implementing that manually > would be a waste of resources. > > If something like this was possible, I know someone who might be really > interested in this. > > Chris >
