Hello Dev, I have worked on finding solutions to implement a cloud agnostic Continuous Integration/Continuous Deployment Pipeline for Airavata.
For the Continuous Integration part, we have considered Travis CI and Jenkins. They both come with their own pros and cons. But we found that the fine-grained control that can be achieved using plugins to create complex pipelines makes Jenkins better suited for Airavata's use case. For continuous delivery, we have considered Spinnaker and Jenkins in Master/Slave configuration. Spinnaker operates on the principle of immutable infrastructure i.e, every instance in the cluster is pre-baked with the applications that it is supposed to run and is not modified in any way after it is deployed. Whenever a new change is committed, a new immutable cluster is created and replaces the existing cluster. Though Spinnaker seems to be a full-fledged solution, it lacks proper support and its installation process is difficult. Its concept of constantly creating new instances and destroying old instances may not be suitable for Airavata as we may need to reuse existing reserved instances.Using Jenkins for Continuous Delivery using a master/slave configuration seems to be the better option. We have considered three deployment strategies for rolling updates. - In-place deployment - Red/Black or Blue/Green deployment - Canary Deployment In-place deployment is the basic approach where we update each instance sequentially. The potential drawbacks are the loss of availability of the service during the upgrade cycle. In Red/Black strategy, red is the color denoting the current production cluster. Whenever a new update is committed, a new cluster which is denoted by the black color is created. The black cluster will be running the updated version of the application. The red cluster is slowly scaled down to about 50% of its capacity while the black is scaled up to 50% of the total capacity. Then the router switches all requests to the black cluster. The red cluster is then kept alive for about 2 hours to make a switch back in case of failures or bugs. In Canary deployments, a very small cluster with the updated application is created and 2% or so of the requests are routed to this cluster. All the requests sent to the canary cluster are monitored to check if there are any failures. If successful, the update is deployed using Red/Black Strategy. Thanks, Vidya Sagar -- Vidya Sagar Kalvakunta | Graduate MS CS Student | IU School of Informatics and Computing | Indiana University Bloomington | (812) 691-5002 | vkalv...@iu.edu