On 12/6/15, 9:47 PM, "Jay Pipes" <jaypi...@gmail.com> wrote:
>On 12/07/2015 04:28 AM, Michał Jastrzębski wrote: >> On 6 December 2015 at 09:33, Jay Pipes <jaypi...@gmail.com> wrote: >>> >>> On 12/04/2015 11:50 PM, Michał Jastrzębski wrote: >>>> >>>> Hey guys, >>>> >>>> Orchiestrated upgrades is one of our highest priorities for M in >>>> kolla, so following up after discussion on summit I'd like to >>>> suggest an approach: >>>> >>>> Instead of creating playbook called "upgrade my openstack" we >>>> will create "upgrade my nova" instead and approach to each >>>> service case by case (since all of our running services are in >>>> dockers, this is possible). We will also make use of image tags >>>> as version carriers, so ansible will deploy new container only if >>>> version tag differs from what we ask it to deploy. This will help >>>> with idempotency of upgrade process. >>>> >>>> So let's start with nova. Upgrade-my-nova playbook will do >>>> something like this: >>>> >>>> 0. We create snapshot of our mariadb-data container. This will >>>> affect every service, but it's always good to have backup and >>>> rollback of db will be manual action >>>> >>>> 1. Nova bootstrap will be called and it will perform >>>> db-migration. Since current approach to nova code is add-only we >>>> shouldn't need to stop services and old services should keep >>>> working on newer database. Also for minor version upgrades there >>>> will be no action here unless there is migration. 2. We upgrade >>>> all conductor at the same time. This should take mere seconds >>>> since we'll have prebuilt containers 3. We will upgrade rest of >>>> controller services with using "series: 1" in ansible to ensure >>>> rolling upgrades. 4. We will upgrade all of nova-compute services >>>> on it's own pace. >>>> >>>> This workflow should be pretty robust (I think it is) and it >>>> should also provide idempotency. >>> >>> >>> From Nova's perspective, most of the above looks OK to me, however, >>> I am in more in favor of an upgrade strategy that builds a "new" >>> set of conductor or controller service containers all at once and >>> then flips the VIP or managed IP from the current controller pod >>> containers to the newly-updated controller pod containers. >> >> Well, we can't really do that because that would affect all other >> services on controller host, and we want to minimize impact on rest >> of cluster during upgrade. Containers gives us option to upgrade >> "just nova", or even "just nova conductors" without affecting >> anything else. On the bright side, redeploying pre-built container is >> a matter of seconds (or even less). Whole process from turning off >> last conductor to spawn first new conductor should take less than >> 10s, and that's only downtime we should get there. > >Sorry, I guess I wasn't clear. I was not proposing to put all controller >services in a single container. I was proposing simply to build the >various container sets (nova-conductor, nova-api, etc) with the updated >code for that service and then "flip the target IP or port" for that >particular service from the existing container set's VIP to the >new/updated container set's VIP. > >In Kubernetes-speak, you would create a new Pod with containers having >the updated Nova conductor code, and would set the new Pod's selector to >something different than the existing Pod's selector -- using the Git >SHA1 hash for the selector would be perfect... You would then update the >Service object's selector to target the new Pod instead of the old one. > >Or, alternately, you could use Kubernetes' support for rolling updates >[1] of a service using a ReplicationController that essentially does the >Pod and Service orchestration for you. > >Hope that is a little more clear what I was referring to... > >Best, >-jay Jay, Forgive me if this is a little incoherent - I have a root canal scheduled for tomorrow with the fun-filled day that comes comes with :( We don't use docker-proxy in kolla. Instead we use net=host mode. I think what you propose would require an extra global VIP for upgrades (or using docker-proxy, which we don't want to do). I am not really hot on this approach for that reason. Regards -stee > >[1] >https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/repli >cation-controller.md#rolling-updates > >>> This seems to me to be the most efficient (in terms of minimal >>> overall downtime) and most robust (because a "rollback" is simply >>> flipping the VIP back to the original controller/conductor service >>> pod containers) solution for upgrading Nova. >> >> >>> >>> Dan Smith, can you comment on this since you are the God of >>> Upgrades? >>> >>> Best, -jay >>> >>> >>> >>>________________________________________________________________________ >>>__ >>> >>> >OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >>_________________________________________________________________________ >>_ >> >> >OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >__________________________________________________________________________ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
