Re: The structure of distribution
Hi, If we use one script, what should we naming it, start-datanode.sh or others? Just feel "start-datanode.sh -r" look strange, user may also need to read the UserGuide to find the removing function. If we put remove-datanode.sh in sbin, users could find it directly under sbin. Thanks, — Jialin Qiao Apache IoTDB PMC Chao Wang 于2022年10月22日周六 10:56写道： > > And， why do we need remove-datanode.sh and remove-confignode.sh scripts? I > think those two scripts need to be merged into start-datanode.sh and > start-confignode.sh, after all, they only have one parameter diffierence. > > > > > Thanks! > > > Chao Wang > BONC ltd > ccgow...@163.com > On 10/21/2022 17:52，Chao Wang wrote： > +1 > > > > > Thanks! > > > Chao Wang > BONC ltd > ccgow...@163.com > On 10/21/2022 09:15，Houliang Qi wrote： > +1 > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | Jialin Qiao | > | Date | 10/20/2022 23:56 | > | To | | > | Subject | Re: The structure of distribution | > +1 > — > Jialin Qiao > Apache IoTDB PMC > > Haonan Hou 于2022年10月20日周四 18:32写道： > > Hi all, > > We will release v1.0 in the nearly future, it's time to decide the final > version > of distribution structure. After investigate other projects, like Hadoop and > Spark. I think it will be better that remove the confignode and datanode > folder, > and combine their script and configuration files into the conf and sbin under > IOTDB_HOME. The design document is in [1]. > > I also submitted a PR[2] to make such changes, in where you can find the > screenshot of new structure. > > [1] https://apache-iotdb.feishu.cn/docx/doxcnFpCzjZadvMPQHUROW4mrTe > [2] https://github.com/apache/iotdb/pull/7672 > > Best, > — > Haonan Hou > >
Re: The structure of distribution
And， why do we need remove-datanode.sh and remove-confignode.sh scripts? I think those two scripts need to be merged into start-datanode.sh and start-confignode.sh, after all, they only have one parameter diffierence. Thanks! Chao Wang BONC ltd ccgow...@163.com On 10/21/2022 17:52，Chao Wang wrote： +1 Thanks! Chao Wang BONC ltd ccgow...@163.com On 10/21/2022 09:15，Houliang Qi wrote： +1 Thanks, --- Houliang Qi BONC, Ltd Replied Message | From | Jialin Qiao | | Date | 10/20/2022 23:56 | | To | | | Subject | Re: The structure of distribution | +1 — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2022年10月20日周四 18:32写道： Hi all, We will release v1.0 in the nearly future, it's time to decide the final version of distribution structure. After investigate other projects, like Hadoop and Spark. I think it will be better that remove the confignode and datanode folder, and combine their script and configuration files into the conf and sbin under IOTDB_HOME. The design document is in [1]. I also submitted a PR[2] to make such changes, in where you can find the screenshot of new structure. [1] https://apache-iotdb.feishu.cn/docx/doxcnFpCzjZadvMPQHUROW4mrTe [2] https://github.com/apache/iotdb/pull/7672 Best, — Haonan Hou
[BUILD-STABLE]: Job 'IoTDB/IoTDB-Pipe/master [master] [743]'
BUILD-STABLE: Job 'IoTDB/IoTDB-Pipe/master [master] [743]': Is back to normal.
Re: Is there any form of hierarchical storage and processing possible with the new cluster mode?
Hi Jialin, thanks for that info … so it seems as if IoTDB would have everything I’m looking for … but not just now. That’s ok, as I currently wouldn’t have the time to get working on this …. I’m just brainstorming for possibilities to solve problems I saw. Thanks again and I’ll keep an eye out for hearing that these features are available. Chris From: Jialin Qiao Date: Thursday, 20. October 2022 at 18:21 To: dev@iotdb.apache.org Subject: Re: Is there any form of hierarchical storage and processing possible with the new cluster mode? Hi Chris, Nice to hear from you! The architecture of the new Cluster version is ready in Chinese, we plan to do the translation and release the English version next month. The new cluster supports scalability in different layers: (1) ConfigNode or DataNode process (2) storage or schema engine (3) data or schema partition. So, it's supported now that we add a Cluster DataNode without data migration (just like you store data on leaf node, and add a computing node in the cloud). Besides, for the scenario you raised, we have two functions may help: (1) Data synchronization framework, which supports TsFile-sync or Operation-Sync from one IoTDB(single node or Cluster) to another. Currently, we are working on the TsFile-sync mode. But the Operation-Sync is the next step, and the operation sync we support Users to use a "sql" to specify sync which data. (2) Trigger: You could register some triggers, each trigger could collect some timeseries that could be matched by a pattern(e.g., root.sg.*.s1), and do the logic that user defined. The logic could be sending data to an upper IoTDB. These two functions are all under developing, we plan to merge the code into master and add UserGuide this month. Thanks, ― Jialin Qiao Apache IoTDB PMC Christofer Dutz 于2022年10月20日周四 20:27写道： > > Hi, > > Als one of your previous Mentors I’m still closely following the project. > Still looking for ways to use it in my day job. > > Now I am thinking about, how to process manufacturing data. > > The problem is: In manufacturing a LOT of Data is produced. However, I would > say the further away from the Source the data is, the less important most of > it becomes. > > So, I am thinking of a tree-like structure where Most data is stored close to > the leaves of the tree (The Manufacturing machinery), because this is where I > would like to run predictive maintenance and optimization jobs and these > usually require a lot of data in a fine granularity. Most of this data is > however irrelevant for the higher levels on not needed in this granularity. > > But again, some data is important on higher levels (Like detecting patterns > in production, problem scenarios etc.) this can probably go up until some > Factory dashboard for management with just some KPIs and perhaps even > company-wide over multiple Factories. > > Here I’m thinking that an IoTDB node or even a leaf-node-cluster could store > all the data produced by leaf-nodes and only data that is requested from an > upstream instance is replicated there (Or even just virtually replicated, so > queries would be forwarded to the leaf-nodes). > > So, my question is: Does IoTDB support this sort of hierarchies? If not, is > something like this planned? > I do imagine, that doing this manually would be possible, where every branch > of the tree is a separate IoTDB node or cluster and then custom written code > replicates data based on queries. However I do imagine this should be a > pretty universal concept and therefore everyone implementing that manually > would be a waste of resources. > > If something like this was possible, I know someone who might be really > interested in this. > > Chris >
Re: The structure of distribution
+1 Thanks! Chao Wang BONC ltd ccgow...@163.com On 10/21/2022 09:15，Houliang Qi wrote： +1 Thanks, --- Houliang Qi BONC, Ltd Replied Message | From | Jialin Qiao | | Date | 10/20/2022 23:56 | | To | | | Subject | Re: The structure of distribution | +1 — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2022年10月20日周四 18:32写道： Hi all, We will release v1.0 in the nearly future, it's time to decide the final version of distribution structure. After investigate other projects, like Hadoop and Spark. I think it will be better that remove the confignode and datanode folder, and combine their script and configuration files into the conf and sbin under IOTDB_HOME. The design document is in [1]. I also submitted a PR[2] to make such changes, in where you can find the screenshot of new structure. [1] https://apache-iotdb.feishu.cn/docx/doxcnFpCzjZadvMPQHUROW4mrTe [2] https://github.com/apache/iotdb/pull/7672 Best, — Haonan Hou
