good idea! Need to clarify how to test new cluster architecture
gaojun2048 <[email protected]> 于2024年4月10日周三 10:24写道： > At present, the Zeta engine does not distinguish between master nodes and > worker nodes. All nodes can serve as master and worker nodes, and only one > node in the entire cluster is marked as a master node at the same time. The > master is responsible for receiving tasks submitted by clients and > scheduling tasks, while also taking on the task running work. > > When there are many tasks running in the cluster, the load on the master > node can be very high, which may cause the master node to hang up and > perform fault tolerance on the master node. Master node fault tolerance > will further increase the responsibility of the cluster, especially newly > selected master nodes have a greater chance of failure due to excessive > load. > > The goal of this proposal is to separate the roles of the master and worker > nodes. The master node is only responsible for task scheduling, IMAP data > storage, and access; Worker nodes are only responsible for the operation of > tasks, and do not participate in the storage and access of IMAP data, nor > do they participate in parameter elections. > > > Detailed design document link： > > https://cwiki.apache.org/confluence/display/SEATUNNEL/STIP7-The+Zeta+Master+is+separated+from+the+Worker > > -- > > Best Regards > > ------------ > > Apache ID: gaojun2048 > > Github ID: EricJoy2048 > > Mail: [email protected] >
