Re: Welcome to join working on "cluster-" branch together
Hi, I just finished some works in hand and can find some time back to the development. Are there some updates for the branch? I do not see any PR for this branch. If there is, please let me know and submit PRs. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Xiangdong Huang 于2021年8月31日周二 上午9:12写道： > > Hi, > > I'd like to open an online discussion at 10:30 AM (Beijing Time) for > this branch development. > Welcome to join if you are interested. > The meeting address and meeting minutes can be found here [1] > > [1] > https://cwiki.apache.org/confluence/display/IOTDB/Cluster+branch+discussion > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > Xiangdong Huang 于2021年8月27日周五 下午2:32写道： > > > > > Hi, > > > > I have worked on "cluster-" branch for several days, but still not finished. > > As I have no much time to finish it, I'd like to ask cooperators > > working on it together. > > Please let me know if you want to join. > > > > "cluster-" branch is not for fixing any bug in the cluster module, > > but just for refactoring the codes to make the structure similar to > > the server module. > > > > Some ideas in the branch: > > > > 1. split Thrift RPC service and RPC implementation. > > In the server module, we have extracted a class called ThriftService, > > which is responsible for starting a RPC service. We just need to > > ingest a RPC implementation to it. > > However, in the cluster module, some RPC classes mixed them, which > > causes code duplication, code inconsistencies, and hard to understand. > > > > 2. weaken the role of MetaGroupMember. metaGroupMember is just an > > engine for serving meta raft group, which should not control the whole > > server too deep. So, many fields (like coordinator, etc.) are > > extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB > > is responsible for creating them. MetaGroupMember only can modify > > these fields, rather than creating them. > > > > That is, the lifecycle of these fields belongs to ClusterIoTDB, rather > > than MetaGroupMember. > > > > BTW, MetaGroupMember will be renamed to MetaGroupEngine. > > > > 3. Similar with the relationship between StorageEngine and > > StorageProcessor in the server module, DataGroupMember can be > > considered as StorageProcessor, and we need a DataGroupEngine to > > control them. > > > > 4. I am considering how to refactor the ClientPool. there are too many > > in the codes. But I have no clear idea now. > > > > The 2nd and 3rd refactors will lead to many unexpected issues as I am > > not also very familiar with the codes. So, I have to check all UT and > > ITs one by one. > > > > I think after the refactoring, developers who know the server module > > will be easy to understand the codes in the cluster module. > > > > And, once the ClusterIoTDB dominates the module, it will be clear to > > modify the startup process like using ID to replace Node. > > > > Best, > > --- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > 黄向东 > > 清华大学 软件学院
Re: Welcome to join working on "cluster-" branch together
Hi, I'd like to open an online discussion at 10:30 AM (Beijing Time) for this branch development. Welcome to join if you are interested. The meeting address and meeting minutes can be found here [1] [1] https://cwiki.apache.org/confluence/display/IOTDB/Cluster+branch+discussion Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Xiangdong Huang 于2021年8月27日周五 下午2:32写道： > > Hi, > > I have worked on "cluster-" branch for several days, but still not finished. > As I have no much time to finish it, I'd like to ask cooperators > working on it together. > Please let me know if you want to join. > > "cluster-" branch is not for fixing any bug in the cluster module, > but just for refactoring the codes to make the structure similar to > the server module. > > Some ideas in the branch: > > 1. split Thrift RPC service and RPC implementation. > In the server module, we have extracted a class called ThriftService, > which is responsible for starting a RPC service. We just need to > ingest a RPC implementation to it. > However, in the cluster module, some RPC classes mixed them, which > causes code duplication, code inconsistencies, and hard to understand. > > 2. weaken the role of MetaGroupMember. metaGroupMember is just an > engine for serving meta raft group, which should not control the whole > server too deep. So, many fields (like coordinator, etc.) are > extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB > is responsible for creating them. MetaGroupMember only can modify > these fields, rather than creating them. > > That is, the lifecycle of these fields belongs to ClusterIoTDB, rather > than MetaGroupMember. > > BTW, MetaGroupMember will be renamed to MetaGroupEngine. > > 3. Similar with the relationship between StorageEngine and > StorageProcessor in the server module, DataGroupMember can be > considered as StorageProcessor, and we need a DataGroupEngine to > control them. > > 4. I am considering how to refactor the ClientPool. there are too many > in the codes. But I have no clear idea now. > > The 2nd and 3rd refactors will lead to many unexpected issues as I am > not also very familiar with the codes. So, I have to check all UT and > ITs one by one. > > I think after the refactoring, developers who know the server module > will be easy to understand the codes in the cluster module. > > And, once the ClusterIoTDB dominates the module, it will be clear to > modify the startup process like using ID to replace Node. > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院
答复: Re: Welcome to join working on "cluster-" branch together
Sorry for the inconvenience for the email title change. Later I'll use my personal email account(chengjian...@outlook.com) to avoid the issue. Actually, I don't where I could start from. When I checked out the `cluster-` branch, I saw so many changes... Could we book a time to talk about the form of cooperation and how community member get start? Anyone in the community can join and I can take the meeting notes and publish to community after the meeting. Thanks! Jianyun Cheng 发件人: Xiangdong Huang 发送时间: 2021年8月28日 10:49:04 收件人: dev 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Re: Welcome to join working on "cluster-" branch together Hi, It is ok. online meeting is fine, but we need to leave a meeting records to the community. Before that, I noticed that the email is marked as "【可能伪造邮件 POSSIBLY SPOOFED 】". Please change some settings of your mailbox... All these will be recorded on the Internet and it is not good for searching content... (BTW, if you want to reply on this thread, please modify the subject manually) Best --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 程建云 于2021年8月27日周五 下午6:37写道： > > Hi, Xiangdong > > > We are glad to join in to work together to help refactor the cluster module! > > > Do you have any idea about the form of cooperation? It's better to have some > regular online meeting or something like to help improve the productivity. > > > > Thanks! > > Jianyun Cheng > > > 发件人: 李思佳 > 发送时间: 2021年8月27日 18:25 > 收件人: dev@iotdb.apache.org > 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join > working on "cluster-" branch together > > Hi, > > This is Sijia Li, I am a big data engineer of Qihoo 360. > > I'd like to participate in this work with my colleague @Jianyun Cheng. > > I hope that we can have a pleasant cooperation. > > Thanks > > -邮件原件----- > 发件人: Xiangdong Huang > 发送时间: 2021年8月27日 14:33 > 收件人: dev > 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch > together > > Hi, > > I have worked on "cluster-" branch for several days, but still not finished. > As I have no much time to finish it, I'd like to ask cooperators working on > it together. > Please let me know if you want to join. > > "cluster-" branch is not for fixing any bug in the cluster module, but just > for refactoring the codes to make the structure similar to the server module. > > Some ideas in the branch: > > 1. split Thrift RPC service and RPC implementation. > In the server module, we have extracted a class called ThriftService, which > is responsible for starting a RPC service. We just need to ingest a RPC > implementation to it. > However, in the cluster module, some RPC classes mixed them, which causes > code duplication, code inconsistencies, and hard to understand. > > 2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for > serving meta raft group, which should not control the whole server too deep. > So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB > (renamed from ClusterMain), and ClusterIoTDB is responsible for creating > them. MetaGroupMember only can modify these fields, rather than creating them. > > That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than > MetaGroupMember. > > BTW, MetaGroupMember will be renamed to MetaGroupEngine. > > 3. Similar with the relationship between StorageEngine and StorageProcessor > in the server module, DataGroupMember can be considered as StorageProcessor, > and we need a DataGroupEngine to control them. > > 4. I am considering how to refactor the ClientPool. there are too many in the > codes. But I have no clear idea now. > > The 2nd and 3rd refactors will lead to many unexpected issues as I am not > also very familiar with the codes. So, I have to check all UT and ITs one by > one. > > I think after the refactoring, developers who know the server module will be > easy to understand the codes in the cluster module. > > And, once the ClusterIoTDB dominates the module, it will be clear to modify > the startup process like using ID to replace Node. > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院
Re: Welcome to join working on "cluster-" branch together
Hi, It is ok. online meeting is fine, but we need to leave a meeting records to the community. Before that, I noticed that the email is marked as "【可能伪造邮件 POSSIBLY SPOOFED 】". Please change some settings of your mailbox... All these will be recorded on the Internet and it is not good for searching content... (BTW, if you want to reply on this thread, please modify the subject manually) Best --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 程建云 于2021年8月27日周五 下午6:37写道： > > Hi, Xiangdong > > > We are glad to join in to work together to help refactor the cluster module! > > > Do you have any idea about the form of cooperation? It's better to have some > regular online meeting or something like to help improve the productivity. > > > > Thanks! > > Jianyun Cheng > > > 发件人: 李思佳 > 发送时间: 2021年8月27日 18:25 > 收件人: dev@iotdb.apache.org > 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join > working on "cluster-" branch together > > Hi, > > This is Sijia Li, I am a big data engineer of Qihoo 360. > > I'd like to participate in this work with my colleague @Jianyun Cheng. > > I hope that we can have a pleasant cooperation. > > Thanks > > -----邮件原件- > 发件人: Xiangdong Huang > 发送时间: 2021年8月27日 14:33 > 收件人: dev > 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch > together > > Hi, > > I have worked on "cluster-" branch for several days, but still not finished. > As I have no much time to finish it, I'd like to ask cooperators working on > it together. > Please let me know if you want to join. > > "cluster-" branch is not for fixing any bug in the cluster module, but just > for refactoring the codes to make the structure similar to the server module. > > Some ideas in the branch: > > 1. split Thrift RPC service and RPC implementation. > In the server module, we have extracted a class called ThriftService, which > is responsible for starting a RPC service. We just need to ingest a RPC > implementation to it. > However, in the cluster module, some RPC classes mixed them, which causes > code duplication, code inconsistencies, and hard to understand. > > 2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for > serving meta raft group, which should not control the whole server too deep. > So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB > (renamed from ClusterMain), and ClusterIoTDB is responsible for creating > them. MetaGroupMember only can modify these fields, rather than creating them. > > That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than > MetaGroupMember. > > BTW, MetaGroupMember will be renamed to MetaGroupEngine. > > 3. Similar with the relationship between StorageEngine and StorageProcessor > in the server module, DataGroupMember can be considered as StorageProcessor, > and we need a DataGroupEngine to control them. > > 4. I am considering how to refactor the ClientPool. there are too many in the > codes. But I have no clear idea now. > > The 2nd and 3rd refactors will lead to many unexpected issues as I am not > also very familiar with the codes. So, I have to check all UT and ITs one by > one. > > I think after the refactoring, developers who know the server module will be > easy to understand the codes in the cluster module. > > And, once the ClusterIoTDB dominates the module, it will be clear to modify > the startup process like using ID to replace Node. > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院
答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together
Hi, Xiangdong We are glad to join in to work together to help refactor the cluster module! Do you have any idea about the form of cooperation? It's better to have some regular online meeting or something like to help improve the productivity. Thanks! Jianyun Cheng 发件人: 李思佳 发送时间: 2021年8月27日 18:25 收件人: dev@iotdb.apache.org 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together Hi, This is Sijia Li, I am a big data engineer of Qihoo 360. I'd like to participate in this work with my colleague @Jianyun Cheng. I hope that we can have a pleasant cooperation. Thanks -邮件原件- 发件人: Xiangdong Huang 发送时间: 2021年8月27日 14:33 收件人: dev 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together Hi, I have worked on "cluster-" branch for several days, but still not finished. As I have no much time to finish it, I'd like to ask cooperators working on it together. Please let me know if you want to join. "cluster-" branch is not for fixing any bug in the cluster module, but just for refactoring the codes to make the structure similar to the server module. Some ideas in the branch: 1. split Thrift RPC service and RPC implementation. In the server module, we have extracted a class called ThriftService, which is responsible for starting a RPC service. We just need to ingest a RPC implementation to it. However, in the cluster module, some RPC classes mixed them, which causes code duplication, code inconsistencies, and hard to understand. 2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for serving meta raft group, which should not control the whole server too deep. So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB is responsible for creating them. MetaGroupMember only can modify these fields, rather than creating them. That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than MetaGroupMember. BTW, MetaGroupMember will be renamed to MetaGroupEngine. 3. Similar with the relationship between StorageEngine and StorageProcessor in the server module, DataGroupMember can be considered as StorageProcessor, and we need a DataGroupEngine to control them. 4. I am considering how to refactor the ClientPool. there are too many in the codes. But I have no clear idea now. The 2nd and 3rd refactors will lead to many unexpected issues as I am not also very familiar with the codes. So, I have to check all UT and ITs one by one. I think after the refactoring, developers who know the server module will be easy to understand the codes in the cluster module. And, once the ClusterIoTDB dominates the module, it will be clear to modify the startup process like using ID to replace Node. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
答复: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together
Hi, This is Sijia Li, I am a big data engineer of Qihoo 360. I'd like to participate in this work with my colleague @Jianyun Cheng. I hope that we can have a pleasant cooperation. Thanks -邮件原件- 发件人: Xiangdong Huang 发送时间: 2021年8月27日 14:33 收件人: dev 主题: 【可能伪造邮件 POSSIBLY SPOOFED 】Welcome to join working on "cluster-" branch together Hi, I have worked on "cluster-" branch for several days, but still not finished. As I have no much time to finish it, I'd like to ask cooperators working on it together. Please let me know if you want to join. "cluster-" branch is not for fixing any bug in the cluster module, but just for refactoring the codes to make the structure similar to the server module. Some ideas in the branch: 1. split Thrift RPC service and RPC implementation. In the server module, we have extracted a class called ThriftService, which is responsible for starting a RPC service. We just need to ingest a RPC implementation to it. However, in the cluster module, some RPC classes mixed them, which causes code duplication, code inconsistencies, and hard to understand. 2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for serving meta raft group, which should not control the whole server too deep. So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB is responsible for creating them. MetaGroupMember only can modify these fields, rather than creating them. That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than MetaGroupMember. BTW, MetaGroupMember will be renamed to MetaGroupEngine. 3. Similar with the relationship between StorageEngine and StorageProcessor in the server module, DataGroupMember can be considered as StorageProcessor, and we need a DataGroupEngine to control them. 4. I am considering how to refactor the ClientPool. there are too many in the codes. But I have no clear idea now. The 2nd and 3rd refactors will lead to many unexpected issues as I am not also very familiar with the codes. So, I have to check all UT and ITs one by one. I think after the refactoring, developers who know the server module will be easy to understand the codes in the cluster module. And, once the ClusterIoTDB dominates the module, it will be clear to modify the startup process like using ID to replace Node. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Welcome to join working on "cluster-" branch together
Hi, I have worked on "cluster-" branch for several days, but still not finished. As I have no much time to finish it, I'd like to ask cooperators working on it together. Please let me know if you want to join. "cluster-" branch is not for fixing any bug in the cluster module, but just for refactoring the codes to make the structure similar to the server module. Some ideas in the branch: 1. split Thrift RPC service and RPC implementation. In the server module, we have extracted a class called ThriftService, which is responsible for starting a RPC service. We just need to ingest a RPC implementation to it. However, in the cluster module, some RPC classes mixed them, which causes code duplication, code inconsistencies, and hard to understand. 2. weaken the role of MetaGroupMember. metaGroupMember is just an engine for serving meta raft group, which should not control the whole server too deep. So, many fields (like coordinator, etc.) are extracted to ClusterIoTDB (renamed from ClusterMain), and ClusterIoTDB is responsible for creating them. MetaGroupMember only can modify these fields, rather than creating them. That is, the lifecycle of these fields belongs to ClusterIoTDB, rather than MetaGroupMember. BTW, MetaGroupMember will be renamed to MetaGroupEngine. 3. Similar with the relationship between StorageEngine and StorageProcessor in the server module, DataGroupMember can be considered as StorageProcessor, and we need a DataGroupEngine to control them. 4. I am considering how to refactor the ClientPool. there are too many in the codes. But I have no clear idea now. The 2nd and 3rd refactors will lead to many unexpected issues as I am not also very familiar with the codes. So, I have to check all UT and ITs one by one. I think after the refactoring, developers who know the server module will be easy to understand the codes in the cluster module. And, once the ClusterIoTDB dominates the module, it will be clear to modify the startup process like using ID to replace Node. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
