wardlican opened a new issue, #3670: URL: https://github.com/apache/amoro/issues/3670
### Description Currently, AMS uses a master-slave mode. If a large number of Iceberg tables are managed, the master-slave mode will have performance bottlenecks, and a solution that can support horizontal performance expansion is needed. ### Use case/motivation Multiple catalogs need to be registered on amoro. Under each catalog, there are many Iceberg tables that need to be managed automatically. If only a single-node AMS is working, the single node will consume a lot of resources. It is necessary to support a horizontally scalable performance expansion solution. By expanding the capacity of AMS nodes, each node can be responsible for processing a part of the Iceberg table. In this way, the resources of each node can be controlled and horizontal performance expansion can be achieved. ### Describe the solution Change the master-slave mode to the master-slave mode. Each AMS node is only responsible for processing a part of the Iceberg table. It supports dynamic expansion and contraction of the number of AMS nodes and automatic migration of the Iceberg table. The optimizer will pull tasks from each AMS node for processing. The AMS master node is responsible for evenly distributing the Iceberg table to all AMS nodes in the cluster and managing the life cycle of the AMS nodes in the cluster. ### Subtasks -[]#1 Change tableService to SPI mode to support expansion of other implementation methods. -[]#2 Implement the master-slave mode in the extended tableService. The master node is responsible for synchronizing table information from the catalog. -[]#3 To extend the implementation of ZK, in addition to being responsible for the election of the master, it is also necessary to implement functions such as the registration of ordinary AMS nodes -[]#4 Modify the optimizer to support obtaining tasks from each AMS node for processing. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
