Hi, Thanks jincheng and jialin.
Sorry for absence of the discussion because of busy work these days.. Some comments: > Core > Core/TsFile > Core/Merge > Core/Storage > Core/WAL Not so clearly.. maybe TsFile, WAL, Storage Engine, Query Engine? (now query engine is in the storage engine..) Merge is a submodule of storage engine and query engine. If we pick out the merge module, we also need to pick out the FLUSH, MODIFICATION, etc.. I think. Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao <[email protected]> 于2020年1月16日周四 下午11:21写道： > Hi Jincheng, > > Thank you! Looks good :) > > jincheng sun <[email protected]> 于2020年1月16日周四 下午4:01写道： > > > Thanks for your explanation! @Jialin > > > > I thought about the composition of our components from the external > system, > > the user level, to the IoTDB bottom-level implementation to the auxiliary > > tools, and I adjusted a little based on yours. as follows: > > > > 1. --Connectors------------- > > Connectors components include all data integration with external systems, > > The components are listed as follows: > > > > Connectors > > Connectors/Grafana > > Connectors/Spark-TsFile > > Connectors/Spark-IoTDB > > Connectors/Hive-tsfile > > Connectors/Hadoop-tsfile > > > > 2. --Client------------- > > Client components involves RPC between client and server, a command-line > > interface（multi-languages).The components are listed as follows: > > > > Client > > Client/Java > > Client/Python > > > > 3. --Planner------------- > > Planner components involves the query process in IoTDB, including > multiple > > types of queries, such as raw data query, aggregations, group by time, > > group by device, and includes many optimized rules.The components are > > listed as follows: > > > > Planner > > Planner/sql parsar > > Planner/sql optimizer > > > > 4. --Core------------- > > Core components is the storage engine of IoTDB. The components are listed > > as follows: > > > > Core > > Core/TsFile > > Core/Merge > > Core/Storage > > Core/WAL > > > > 5. --Tools------------- > > Tools components involves many utils of IoTDB, Such as data Sync. The > > components are listed as follows: > > Tools > > Tools/Sync > > Tools/Other > > > > 6. --Authority--------- > > Authority > > > > 7. --Distribution------ > > Distribution > > > > 8. --Example----------- > > Example > > > > 9. --WebSite----------- > > WebSite > > > > 10. --Doc--------------- > > Doc > > > > 11. --Other------------ > > Other （I'm not sure if we really need the Other component, but there is > no > > harm in adding one） > > > > This does not change much from your solution. The core is to add a layer > of > > abstraction. > > > > Does this make sense to you? Anyway we can continue to improve if needed! > > > > Best, > > Jincheng > > > > Jialin Qiao <[email protected]> 于2020年1月15日周三 下午4:28写道： > > > > > Hi, > > > > > > WAL has an independent physical layout that has nothing to do with > > TsFile. > > > It is used in the writing process and recovering in IoTDB. > > > It could be merged into Storage components. > > > > > > I'm not sure whether I leave out some components, so I add an Other and > > > therefore can not give an example... > > > > > > Thanks, > > > > > > jincheng sun <[email protected]> 于2020年1月15日周三 下午4:01写道： > > > > > > > Sorry for the late reply, I just came back from team outing. > > > > > > > > Thanks for adding more detail information. I have two questions then: > > > > > > > > - Is WAL a write strategy for TsFlie, can WAL be incorporated into > > other > > > > components and what is the reason for being a separate component? > > > > - Can you give an example of what changes belongs to the Other > > component? > > > > > > > > Best, > > > > Jincheng > > > > > > > > > > > > Jialin Qiao <[email protected]> 于2020年1月9日周四 下午6:39写道： > > > > > > > > > Hi, > > > > > > > > > > Sure, maybe a bit more... > > > > > > > > > > - TsFile : A columnar file format that has flexible schema and > data > > > > > management for time series. > > > > > - Storage : The storage engine of IoTDB, including writing to > > > memtable > > > > > (buffer), flushing, system recovery, and memtable management. > > > > > - Query : The query process in IoTDB, including multiple types of > > > > > queries, such as raw data query, aggregations, group by time, group > > by > > > > > device. > > > > > - Merge : Compaction of data files, including multiple compaction > > > > > strategies, such as (1) merging unsequence TsFile with sequence > > TsFile, > > > > (2) > > > > > merging small sequence TsFiles into a large TsFile. > > > > > - Sync: Synchronize data files from one IoTDB instance to > another. > > > > > - Authority: Administration of IoTDB, including users, roles and > > > > > privileges. > > > > > - WAL : Write-ahead-log. > > > > > - Client: Rpc between client and server, a command-line interface > > and > > > > > client of different languages. > > > > > - Distribution : The shared-nothing distribution of IoTDB, > > including > > > > raft > > > > > protocol, data partition, and system information management. > > > > > - Grafana : The connector for using IoTDB as a data source in > > > Grafana. > > > > > - Spark-TsFile : A connector that allows users read/write TsFile > > > > through > > > > > SparkSQL (treat TsFile as an external data source for SparkSQL). > > > > > - Spark-IoTDB : A connector that allows users read/write IoTDB > > > through > > > > > SparkSQL (treat IoTDB as an external data source for SparkSQL). > > > > > - Hive-tsfile : A connector that maps TsFile as a table in Hive > for > > > the > > > > > query. > > > > > - Hadoop-tsfile : A connector that allows users read/write TsFile > > > > through > > > > > MapReduce. > > > > > - Tools : Other tools for IoTDB, such as watermark, memory > > > estimation, > > > > > JMX, query history visualization. > > > > > - Example : Examples of different modules, such as JDBC, Session, > > > > TsFile. > > > > > - WebSite : IoTDB official website. > > > > > - Doc: documents of IoTDB, docs folder under root directory. > > > > > - Other: Those do not belong to the above components. > > > > > > > > > > > > > > > jincheng sun <[email protected]> 于2020年1月9日周四 下午5:28写道： > > > > > > > > > > > Hi, > > > > > > > > > > > > @Xiangdong thank you for helping to clarify my original > intention. > > > you > > > > > are > > > > > > right, I really want to list some components to let users choose > > > which > > > > > > component the given issue belongs to. > > > > > > > > > > > > @Jialin I think approach 4 would be more meaningful for IoTDB. > > Could > > > > you > > > > > > please add the detailed description for each component, such as, > > > > describe > > > > > > the relationship between `tsfile` and > > > > > > (`tsfile/hive-tsfile/spark-tsfile/Hadoop-tsfile`), and which > > changes > > > > > belong > > > > > > to `Other` component, etc., which can help us better understand > the > > > > > reasons > > > > > > for such classification. On the other hand, we will further > discuss > > > > > whether > > > > > > there is optimization space for this component classification. > > > > > > > > > > > > What do you think? > > > > > > > > > > > > Best, > > > > > > Jincheng > > > > > > > > > > > > Xiangdong Huang <[email protected]> 于2020年1月9日周四 下午4:46写道： > > > > > > > > > > > > > > No that is not the Apache Way. No one should be in charge of > > an > > > > area > > > > > > > > > > > > > > I think Jincheng's mean is, we can list some components to let > > > users > > > > > > > choose which component the given issue belongs to. > > > > > > > > > > > > > > As for which components should be listed, we need to have a > > > > discussion. > > > > > > > > > > > > > > > No one should be in charge of an area > > > > > > > > > > > > > > Yes everyone can assign an issue to himself/herself. > > > > > > > > > > > > > > Best, > > > > > > > ---------------------------------- > > > > > > > Xiangdong Huang > > > > > > > School of Software, Tsinghua University > > > > > > > > > > > > > > 黄向东 > > > > > > > 清华大学 软件学院 > > > > > > > > > > > > > > > > > > > > > jincheng sun <[email protected]> 于2020年1月9日周四 下午4:38写道： > > > > > > > > > > > > > > > > No that is not the Apache Way. No one should be in charge > of > > an > > > > > area. > > > > > > > > > > > > > > > > Oh, great thanks for correcting me Justin, maybe it's my > > language > > > > > > > > expression problems. I mean some contributors and committers > > will > > > > be > > > > > > very > > > > > > > > interested in some components, then spend more time, and be > > more > > > > > > familiar > > > > > > > > with the code. > > > > > > > > > > > > > > > > Best, > > > > > > > > Jincheng > > > > > > > > > > > > > > > > > > > > > > > > Justin Mclean <[email protected]> 于2020年1月9日周四 > > 下午4:27写道： > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > When our community grows stronger, each\ module may have > > some > > > > > > > specific > > > > > > > > > person in charge. > > > > > > > > > > > > > > > > > > No that is not the Apache Way. No one should be in charge > of > > an > > > > > area. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Justin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > ————————————————— > > > > > Jialin Qiao > > > > > School of Software, Tsinghua University > > > > > > > > > > 乔嘉林 > > > > > 清华大学 软件学院 > > > > > > > > > > > > > > > > > > -- > > > ————————————————— > > > Jialin Qiao > > > School of Software, Tsinghua University > > > > > > 乔嘉林 > > > 清华大学 软件学院 > > > > > > > > -- > ————————————————— > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 >
