Title: Message Title
There's 1 new edit on this page
[WIP] System Architecture
Dominic Kim edited this page
Here's what changed:
... I introduced a few new components. 1) Scheduler Scheduler The scheduler is a new components component it include includes many critical features. It is in charge of two major features, 1. queueing and routing activations 2. decide whether to add more containers based on the loads. It has a sub-component `Queue`. The role of the queue is similar to the Kafka topic. A dedicated queue is created for each action. Each queue will receive activation messages from the Kafka for a given action and send them to the ContainerProxy in respond response to requests from it. 2) ... ETCD etcd ETCD is a distributed reliable key-value store. etcd is mostly used for transactional support and information sharing among components. ... In this version, I could not fully rule out it due to heavy dependencies. My final objective is to exclude it at least from the critical path. It is also being used to send queue creation request to the scheduler. 4) Akka- ... remote Akka-cluster is used for schedulers to communicate with each other. Akka-grpc is required to define a grpc message, there are some advantages in akka-cluster when it is being used for simple inter-cluster communication. It is also being used to send queue creation request to the scheduler. Entire The entire system is comprised of three flows, 1. queue creation flow, 2. container creation flow, 3. activation flow. ... ① Once a new request comes, there is no queue for the action, a PoolBalancer starts queue creation flow by randomly choosing a scheduler and calling QueueCreationAPI(Akka-grpc)send a CreateQueue request via akka-remote. ② If a scheduler receives a QueueCreation request, it checks whether there is already queues for the action. If there are already three queues(1 leader + 2 followers), it responds with Success. If there are lesser than 3 queues, it first tries ETCD transaction, and if succeeded, write the information in ETCD to describe queue creation is in progress. ③ During this step, In the meantime, if other schedulers can also receives the receive queue creation requests then the transaction will be failed and they will respond with Success because queue creation is already in progress. ④ A scheduler who achieves the transaction, select the three schedulers with the least number of queues and send queue creation request to QueueManagers in each scheduler via Akka-cluster(remoting). This implies all schedulers should be clustered using Akka-cluster. ⑤ Once a QueueManager receives the request, it will create a queue and send a Start message to it and wait for the Ack. ⑥ Each created queue tries to become a leader and only one of them becomes a leader. The leader writes the endpoint of the scheduler in ETCD for activation clients(ContainerProxy), and responds with Ack[Leader|Follower] to QueueManager. All followers watch the leader key and wait for the leadership change to become a leader in case of the leader failure. ⑦ Once a QueueManager receives Ack from the leader queue, it creates a MessageConsumer with the ActorRef of the queue then the MessageConsumer fetches activation messages from Kafka and send them to the leader queue. In the future, PoolBalancer would send activation requests to the scheduler directly③ The QueueManager creates a queue. ④ Once a queue is created, it tries to store its endpoint to ETCD so that containers can connect to the queue. It sends queue endpoint data to DataManagementService. ⑤ A DataManagementService receives the data it stores it in ETCD. Since PoolBalancers are watching ETCD events for queue endpoints. So once a queue is created, it does not try to create a queue for the given action anymore and send activation messages to the scheduler which has the queue.2. Container Creation Flow Image Modified ① When a new queue is created, it immediately sends a container creation request to the ContainerManager because there is no container yet. ② ContainerManager ContainerManager looks for available invokers. The word "available" means healthy invokers with enough resources. ③ ContainerManager selects one of healthy the invokers and sends a ContainerCreation request via invokerN `invokerN` Kafka topic. ③ Invoker While scheduling, it takes Blackbox and resource tags into account. Invokers can have heterogeneous resources and tags describe such resources. ③ At the same time, ContainerManager will register a job for the container creation request to CreationJobManager. ④ CreationJobManager is in charge of managing container creation, it stores "in-progress" data for container creation in ETCD. In case container creation takes too long, a timeout
