mbrodmann opened a new pull request #84: URL: https://github.com/apache/incubator-crail/pull/84
This PR is the first of three planned PRs for implementing resource elasticity within Crail. More specifically, the goal is to allow to automatically and dynamically adapt the number of running datanodes based on different possible parameters (e.g. resource consumption, available storage capacities, ...). This requires support for starting and terminating running datanodes automatically. This functionality will be implemented mostly within the Crail namenode. Therefore, the Crail namenode will not only maintain information on the current state of the datanodes and the deployment but will also start or terminate datanodes as required. For deciding how to adjust the number of datanodes based on the current state observed in the Crail namenode so-called _policies_ are used. The following changes will be included within the seperate PRs: 1) This first PR implements a new RPC (for both darpc and narpc) that allows to specify a datanode (using IP-address and port number) that should be shutdown. Additionally, a `status` field is added to `DataNodeStatistics`. This will be used in the `getDataNode` RPC to allow the namenode to inform a datanode that it can safely shutdown. 2) The seconds PR will implement the actual mechanism for shutting down a datanode. 3) The third PR implements a simple policy for implementing elasticity based on the system's storage consumption. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
