At present, the Zeta engine does not distinguish between master nodes and worker nodes. All nodes can serve as master and worker nodes, and only one node in the entire cluster is marked as a master node at the same time. The master is responsible for receiving tasks submitted by clients and scheduling tasks, while also taking on the task running work.
When there are many tasks running in the cluster, the load on the master node can be very high, which may cause the master node to hang up and perform fault tolerance on the master node. Master node fault tolerance will further increase the responsibility of the cluster, especially newly selected master nodes have a greater chance of failure due to excessive load. The goal of this proposal is to separate the roles of the master and worker nodes. The master node is only responsible for task scheduling, IMAP data storage, and access; Worker nodes are only responsible for the operation of tasks, and do not participate in the storage and access of IMAP data, nor do they participate in parameter elections. Detailed design document linkļ¼ https://cwiki.apache.org/confluence/display/SEATUNNEL/STIP7-The+Zeta+Master+is+separated+from+the+Worker -- Best Regards ------------ Apache ID: gaojun2048 Github ID: EricJoy2048 Mail: [email protected]
