Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le mardi 16 février 2016 à 17:39 -0800, Amye Scavarda a écrit : > On Tue, Feb 16, 2016 at 12:20 PM, Michael Scherer wrote: > > Le mardi 16 février 2016 à 23:05 +0530, Kaushal M a écrit : > >> On Feb 12, 2016 10:50 PM, "Kaushal M" wrote: > >> > > >> > On Fri, Feb 12, 2016 at 10:47 PM, Michael Scherer > >> wrote: > >> > > Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit : > >> > >> Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : > >> > >> > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer > >> > >> > > >> wrote: > >> > >> > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : > >> > >> > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer < > >> msche...@redhat.com> wrote: > >> > >> > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > >> > >> > >> >> I think starting around 0900 UTC on Friday of next week (12th > >> Feb) > >> > >> > >> >> should be possible. We should be done with 3.7.8 before that > >> and can > >> > >> > >> >> afford a bit of downtime then. In case any assistance is > >> needed post the > >> > >> > >> >> migration, we can have folks around the clock to help. > >> > >> > >> >> > >> > >> > >> >> If migration fails for an unforeseen reason, would we be able > >> to > >> > >> > >> >> rollback and maintain status quo? > >> > >> > >> > > >> > >> > >> > Yes. Worst case, I think people would just redo a few reviews > >> or push > >> > >> > >> > again patch. > >> > >> > >> > > >> > >> > >> > Also, since gerrit is critical, I wonder what is the support of > >> gerrit > >> > >> > >> > for slaves and replication. > >> > >> > >> > >> > >> > >> Michael are you okay with the time? If you are, I think we should > >> > >> > >> announce the migration. > >> > >> > > > >> > >> > > I am ok. > >> > >> > > >> > >> > So as discussed in the community meeting, we will be announcing the > >> > >> > migration and downtime. Vijay, you said you required the exact > >> > >> > schedule? This is what I expect > >> > >> > > >> > >> > Friday 12th Feb 2016 > >> > >> > > >> > >> > 0900 UTC : build.gluster.org and review.gluster.org are taken down > >> and > >> > >> > migration begins. > >> > >> > <1-2 hrs?>: Michael copies over data onto the RH community infra and > >> > >> > sets up the VMs. > >> > >> > : The DNS records are updated, and some time is for it to > >> propogate. > >> > >> > <3hr?>: Verify everything is working well (I can help with this). > >> We'd > >> > >> > possibly need to run a regression job, so this will take longest I > >> > >> > think. > >> > >> > > >> > >> > 1700UTC : We announce the finish of the migration and > >> open > >> > >> > the services back up. If migration failed, we bring the existing > >> > >> > servers back on, and continue on. > >> > >> > >> > >> So, for people wanting to know, the migration has started. > >> > >> > >> > >> 1) gerrit > >> > >> -- > >> > >> gerrit seems to be ok, it has a new IP 66.187.224.201 > >> > >> > >> > >> people who used to have access there need to contact me so I can > >> > >> explain/create required account on the new virt host. > >> > >> > >> > >> the old VM is shutdown for now. I will make a backup copy of the disk. > >> > >> > >> > >> 2) jenkins > >> > >> -- > >> > >> > >> > >> so the preparation of the migration didn't worked as well, so we have > >> to > >> > >> copy the VM when offline, and then prepare it later ( ie, import in > >> > >> libvirt, adjust network, etc). So jenkins master is offline (sorry > >> > >> manu > >> > >> for your tcpdump monitoring), we are making a copy of the disk that is > >> > >> gonna take between 3 and 14h (most likely < 4h, but rsync estimation > >> > >> is > >> > >> moving a lot), and then restart like before. > >> > > > >> > > So turn out I was wrong, and the copy is taking a much longer time. > >> > > Despites me trying to investigate why, it didn't copied everything. > >> > > > >> > > So the new plan is to stop the VM, make a local copy of the disk, start > >> > > the VM, and then copy that one over the new server, prepare for the new > >> > > host (firewall, public ip, etc), and sync from the old VM to the new > >> > > host. > >> > > > >> > > All is ok with this plan ? > >> > > >> > Sounds good to me. No need to wait for the copy to happen and the > >> > downtime should be minimal like with gerrit. > >> > > >> > >> Can we schedule this minimal downtime sometime tomorrow? Or if possible > >> tonight itself. Since we don't require a long downtime we can do this > >> anytime we like. > > > > No. I am at ansiblefest/contributor meeting tomorrow and the day after, > > and then I took a PTO on friday, until end of February. > > So it sounds like we're going to have to wait until March for this, if > I'm reading this correctly. > Accurate? Yep. Cf my calendar for the date I am out -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part __
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
On Tue, Feb 16, 2016 at 12:20 PM, Michael Scherer wrote: > Le mardi 16 février 2016 à 23:05 +0530, Kaushal M a écrit : >> On Feb 12, 2016 10:50 PM, "Kaushal M" wrote: >> > >> > On Fri, Feb 12, 2016 at 10:47 PM, Michael Scherer >> wrote: >> > > Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit : >> > >> Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : >> > >> > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer >> wrote: >> > >> > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : >> > >> > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer < >> msche...@redhat.com> wrote: >> > >> > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : >> > >> > >> >> I think starting around 0900 UTC on Friday of next week (12th >> Feb) >> > >> > >> >> should be possible. We should be done with 3.7.8 before that >> and can >> > >> > >> >> afford a bit of downtime then. In case any assistance is >> needed post the >> > >> > >> >> migration, we can have folks around the clock to help. >> > >> > >> >> >> > >> > >> >> If migration fails for an unforeseen reason, would we be able >> to >> > >> > >> >> rollback and maintain status quo? >> > >> > >> > >> > >> > >> > Yes. Worst case, I think people would just redo a few reviews >> or push >> > >> > >> > again patch. >> > >> > >> > >> > >> > >> > Also, since gerrit is critical, I wonder what is the support of >> gerrit >> > >> > >> > for slaves and replication. >> > >> > >> >> > >> > >> Michael are you okay with the time? If you are, I think we should >> > >> > >> announce the migration. >> > >> > > >> > >> > > I am ok. >> > >> > >> > >> > So as discussed in the community meeting, we will be announcing the >> > >> > migration and downtime. Vijay, you said you required the exact >> > >> > schedule? This is what I expect >> > >> > >> > >> > Friday 12th Feb 2016 >> > >> > >> > >> > 0900 UTC : build.gluster.org and review.gluster.org are taken down >> and >> > >> > migration begins. >> > >> > <1-2 hrs?>: Michael copies over data onto the RH community infra and >> > >> > sets up the VMs. >> > >> > : The DNS records are updated, and some time is for it to >> propogate. >> > >> > <3hr?>: Verify everything is working well (I can help with this). >> We'd >> > >> > possibly need to run a regression job, so this will take longest I >> > >> > think. >> > >> > >> > >> > 1700UTC : We announce the finish of the migration and >> open >> > >> > the services back up. If migration failed, we bring the existing >> > >> > servers back on, and continue on. >> > >> >> > >> So, for people wanting to know, the migration has started. >> > >> >> > >> 1) gerrit >> > >> -- >> > >> gerrit seems to be ok, it has a new IP 66.187.224.201 >> > >> >> > >> people who used to have access there need to contact me so I can >> > >> explain/create required account on the new virt host. >> > >> >> > >> the old VM is shutdown for now. I will make a backup copy of the disk. >> > >> >> > >> 2) jenkins >> > >> -- >> > >> >> > >> so the preparation of the migration didn't worked as well, so we have >> to >> > >> copy the VM when offline, and then prepare it later ( ie, import in >> > >> libvirt, adjust network, etc). So jenkins master is offline (sorry manu >> > >> for your tcpdump monitoring), we are making a copy of the disk that is >> > >> gonna take between 3 and 14h (most likely < 4h, but rsync estimation is >> > >> moving a lot), and then restart like before. >> > > >> > > So turn out I was wrong, and the copy is taking a much longer time. >> > > Despites me trying to investigate why, it didn't copied everything. >> > > >> > > So the new plan is to stop the VM, make a local copy of the disk, start >> > > the VM, and then copy that one over the new server, prepare for the new >> > > host (firewall, public ip, etc), and sync from the old VM to the new >> > > host. >> > > >> > > All is ok with this plan ? >> > >> > Sounds good to me. No need to wait for the copy to happen and the >> > downtime should be minimal like with gerrit. >> > >> >> Can we schedule this minimal downtime sometime tomorrow? Or if possible >> tonight itself. Since we don't require a long downtime we can do this >> anytime we like. > > No. I am at ansiblefest/contributor meeting tomorrow and the day after, > and then I took a PTO on friday, until end of February. So it sounds like we're going to have to wait until March for this, if I'm reading this correctly. Accurate? - amye > >> I'd like to get the migration done by the end of this week (if possible) >> before Michael goes on vacation. > > > -- > Michael Scherer > Sysadmin, Community Infrastructure and Platform, OSAS > > -- Amye Scavarda | a...@redhat.com | Gluster Community Lead ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le mardi 16 février 2016 à 23:05 +0530, Kaushal M a écrit : > On Feb 12, 2016 10:50 PM, "Kaushal M" wrote: > > > > On Fri, Feb 12, 2016 at 10:47 PM, Michael Scherer > wrote: > > > Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit : > > >> Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : > > >> > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer > wrote: > > >> > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : > > >> > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer < > msche...@redhat.com> wrote: > > >> > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > > >> > >> >> I think starting around 0900 UTC on Friday of next week (12th > Feb) > > >> > >> >> should be possible. We should be done with 3.7.8 before that > and can > > >> > >> >> afford a bit of downtime then. In case any assistance is > needed post the > > >> > >> >> migration, we can have folks around the clock to help. > > >> > >> >> > > >> > >> >> If migration fails for an unforeseen reason, would we be able > to > > >> > >> >> rollback and maintain status quo? > > >> > >> > > > >> > >> > Yes. Worst case, I think people would just redo a few reviews > or push > > >> > >> > again patch. > > >> > >> > > > >> > >> > Also, since gerrit is critical, I wonder what is the support of > gerrit > > >> > >> > for slaves and replication. > > >> > >> > > >> > >> Michael are you okay with the time? If you are, I think we should > > >> > >> announce the migration. > > >> > > > > >> > > I am ok. > > >> > > > >> > So as discussed in the community meeting, we will be announcing the > > >> > migration and downtime. Vijay, you said you required the exact > > >> > schedule? This is what I expect > > >> > > > >> > Friday 12th Feb 2016 > > >> > > > >> > 0900 UTC : build.gluster.org and review.gluster.org are taken down > and > > >> > migration begins. > > >> > <1-2 hrs?>: Michael copies over data onto the RH community infra and > > >> > sets up the VMs. > > >> > : The DNS records are updated, and some time is for it to > propogate. > > >> > <3hr?>: Verify everything is working well (I can help with this). > We'd > > >> > possibly need to run a regression job, so this will take longest I > > >> > think. > > >> > > > >> > 1700UTC : We announce the finish of the migration and > open > > >> > the services back up. If migration failed, we bring the existing > > >> > servers back on, and continue on. > > >> > > >> So, for people wanting to know, the migration has started. > > >> > > >> 1) gerrit > > >> -- > > >> gerrit seems to be ok, it has a new IP 66.187.224.201 > > >> > > >> people who used to have access there need to contact me so I can > > >> explain/create required account on the new virt host. > > >> > > >> the old VM is shutdown for now. I will make a backup copy of the disk. > > >> > > >> 2) jenkins > > >> -- > > >> > > >> so the preparation of the migration didn't worked as well, so we have > to > > >> copy the VM when offline, and then prepare it later ( ie, import in > > >> libvirt, adjust network, etc). So jenkins master is offline (sorry manu > > >> for your tcpdump monitoring), we are making a copy of the disk that is > > >> gonna take between 3 and 14h (most likely < 4h, but rsync estimation is > > >> moving a lot), and then restart like before. > > > > > > So turn out I was wrong, and the copy is taking a much longer time. > > > Despites me trying to investigate why, it didn't copied everything. > > > > > > So the new plan is to stop the VM, make a local copy of the disk, start > > > the VM, and then copy that one over the new server, prepare for the new > > > host (firewall, public ip, etc), and sync from the old VM to the new > > > host. > > > > > > All is ok with this plan ? > > > > Sounds good to me. No need to wait for the copy to happen and the > > downtime should be minimal like with gerrit. > > > > Can we schedule this minimal downtime sometime tomorrow? Or if possible > tonight itself. Since we don't require a long downtime we can do this > anytime we like. No. I am at ansiblefest/contributor meeting tomorrow and the day after, and then I took a PTO on friday, until end of February. > I'd like to get the migration done by the end of this week (if possible) > before Michael goes on vacation. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
On Feb 12, 2016 10:50 PM, "Kaushal M" wrote: > > On Fri, Feb 12, 2016 at 10:47 PM, Michael Scherer wrote: > > Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit : > >> Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : > >> > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer wrote: > >> > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : > >> > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer < msche...@redhat.com> wrote: > >> > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > >> > >> >> I think starting around 0900 UTC on Friday of next week (12th Feb) > >> > >> >> should be possible. We should be done with 3.7.8 before that and can > >> > >> >> afford a bit of downtime then. In case any assistance is needed post the > >> > >> >> migration, we can have folks around the clock to help. > >> > >> >> > >> > >> >> If migration fails for an unforeseen reason, would we be able to > >> > >> >> rollback and maintain status quo? > >> > >> > > >> > >> > Yes. Worst case, I think people would just redo a few reviews or push > >> > >> > again patch. > >> > >> > > >> > >> > Also, since gerrit is critical, I wonder what is the support of gerrit > >> > >> > for slaves and replication. > >> > >> > >> > >> Michael are you okay with the time? If you are, I think we should > >> > >> announce the migration. > >> > > > >> > > I am ok. > >> > > >> > So as discussed in the community meeting, we will be announcing the > >> > migration and downtime. Vijay, you said you required the exact > >> > schedule? This is what I expect > >> > > >> > Friday 12th Feb 2016 > >> > > >> > 0900 UTC : build.gluster.org and review.gluster.org are taken down and > >> > migration begins. > >> > <1-2 hrs?>: Michael copies over data onto the RH community infra and > >> > sets up the VMs. > >> > : The DNS records are updated, and some time is for it to propogate. > >> > <3hr?>: Verify everything is working well (I can help with this). We'd > >> > possibly need to run a regression job, so this will take longest I > >> > think. > >> > > >> > 1700UTC : We announce the finish of the migration and open > >> > the services back up. If migration failed, we bring the existing > >> > servers back on, and continue on. > >> > >> So, for people wanting to know, the migration has started. > >> > >> 1) gerrit > >> -- > >> gerrit seems to be ok, it has a new IP 66.187.224.201 > >> > >> people who used to have access there need to contact me so I can > >> explain/create required account on the new virt host. > >> > >> the old VM is shutdown for now. I will make a backup copy of the disk. > >> > >> 2) jenkins > >> -- > >> > >> so the preparation of the migration didn't worked as well, so we have to > >> copy the VM when offline, and then prepare it later ( ie, import in > >> libvirt, adjust network, etc). So jenkins master is offline (sorry manu > >> for your tcpdump monitoring), we are making a copy of the disk that is > >> gonna take between 3 and 14h (most likely < 4h, but rsync estimation is > >> moving a lot), and then restart like before. > > > > So turn out I was wrong, and the copy is taking a much longer time. > > Despites me trying to investigate why, it didn't copied everything. > > > > So the new plan is to stop the VM, make a local copy of the disk, start > > the VM, and then copy that one over the new server, prepare for the new > > host (firewall, public ip, etc), and sync from the old VM to the new > > host. > > > > All is ok with this plan ? > > Sounds good to me. No need to wait for the copy to happen and the > downtime should be minimal like with gerrit. > Can we schedule this minimal downtime sometime tomorrow? Or if possible tonight itself. Since we don't require a long downtime we can do this anytime we like. I'd like to get the migration done by the end of this week (if possible) before Michael goes on vacation. > > -- > > Michael Scherer > > Sysadmin, Community Infrastructure and Platform, OSAS > > > > ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
On Fri, Feb 12, 2016 at 10:47 PM, Michael Scherer wrote: > Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit : >> Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : >> > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer >> > wrote: >> > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : >> > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer >> > >> wrote: >> > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : >> > >> >> I think starting around 0900 UTC on Friday of next week (12th Feb) >> > >> >> should be possible. We should be done with 3.7.8 before that and can >> > >> >> afford a bit of downtime then. In case any assistance is needed post >> > >> >> the >> > >> >> migration, we can have folks around the clock to help. >> > >> >> >> > >> >> If migration fails for an unforeseen reason, would we be able to >> > >> >> rollback and maintain status quo? >> > >> > >> > >> > Yes. Worst case, I think people would just redo a few reviews or push >> > >> > again patch. >> > >> > >> > >> > Also, since gerrit is critical, I wonder what is the support of gerrit >> > >> > for slaves and replication. >> > >> >> > >> Michael are you okay with the time? If you are, I think we should >> > >> announce the migration. >> > > >> > > I am ok. >> > >> > So as discussed in the community meeting, we will be announcing the >> > migration and downtime. Vijay, you said you required the exact >> > schedule? This is what I expect >> > >> > Friday 12th Feb 2016 >> > >> > 0900 UTC : build.gluster.org and review.gluster.org are taken down and >> > migration begins. >> > <1-2 hrs?>: Michael copies over data onto the RH community infra and >> > sets up the VMs. >> > : The DNS records are updated, and some time is for it to propogate. >> > <3hr?>: Verify everything is working well (I can help with this). We'd >> > possibly need to run a regression job, so this will take longest I >> > think. >> > >> > 1700UTC : We announce the finish of the migration and open >> > the services back up. If migration failed, we bring the existing >> > servers back on, and continue on. >> >> So, for people wanting to know, the migration has started. >> >> 1) gerrit >> -- >> gerrit seems to be ok, it has a new IP 66.187.224.201 >> >> people who used to have access there need to contact me so I can >> explain/create required account on the new virt host. >> >> the old VM is shutdown for now. I will make a backup copy of the disk. >> >> 2) jenkins >> -- >> >> so the preparation of the migration didn't worked as well, so we have to >> copy the VM when offline, and then prepare it later ( ie, import in >> libvirt, adjust network, etc). So jenkins master is offline (sorry manu >> for your tcpdump monitoring), we are making a copy of the disk that is >> gonna take between 3 and 14h (most likely < 4h, but rsync estimation is >> moving a lot), and then restart like before. > > So turn out I was wrong, and the copy is taking a much longer time. > Despites me trying to investigate why, it didn't copied everything. > > So the new plan is to stop the VM, make a local copy of the disk, start > the VM, and then copy that one over the new server, prepare for the new > host (firewall, public ip, etc), and sync from the old VM to the new > host. > > All is ok with this plan ? Sounds good to me. No need to wait for the copy to happen and the downtime should be minimal like with gerrit. > -- > Michael Scherer > Sysadmin, Community Infrastructure and Platform, OSAS > > ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit : > Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : > > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer > > wrote: > > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : > > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer > > >> wrote: > > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > > >> >> I think starting around 0900 UTC on Friday of next week (12th Feb) > > >> >> should be possible. We should be done with 3.7.8 before that and can > > >> >> afford a bit of downtime then. In case any assistance is needed post > > >> >> the > > >> >> migration, we can have folks around the clock to help. > > >> >> > > >> >> If migration fails for an unforeseen reason, would we be able to > > >> >> rollback and maintain status quo? > > >> > > > >> > Yes. Worst case, I think people would just redo a few reviews or push > > >> > again patch. > > >> > > > >> > Also, since gerrit is critical, I wonder what is the support of gerrit > > >> > for slaves and replication. > > >> > > >> Michael are you okay with the time? If you are, I think we should > > >> announce the migration. > > > > > > I am ok. > > > > So as discussed in the community meeting, we will be announcing the > > migration and downtime. Vijay, you said you required the exact > > schedule? This is what I expect > > > > Friday 12th Feb 2016 > > > > 0900 UTC : build.gluster.org and review.gluster.org are taken down and > > migration begins. > > <1-2 hrs?>: Michael copies over data onto the RH community infra and > > sets up the VMs. > > : The DNS records are updated, and some time is for it to propogate. > > <3hr?>: Verify everything is working well (I can help with this). We'd > > possibly need to run a regression job, so this will take longest I > > think. > > > > 1700UTC : We announce the finish of the migration and open > > the services back up. If migration failed, we bring the existing > > servers back on, and continue on. > > So, for people wanting to know, the migration has started. > > 1) gerrit > -- > gerrit seems to be ok, it has a new IP 66.187.224.201 > > people who used to have access there need to contact me so I can > explain/create required account on the new virt host. > > the old VM is shutdown for now. I will make a backup copy of the disk. > > 2) jenkins > -- > > so the preparation of the migration didn't worked as well, so we have to > copy the VM when offline, and then prepare it later ( ie, import in > libvirt, adjust network, etc). So jenkins master is offline (sorry manu > for your tcpdump monitoring), we are making a copy of the disk that is > gonna take between 3 and 14h (most likely < 4h, but rsync estimation is > moving a lot), and then restart like before. So turn out I was wrong, and the copy is taking a much longer time. Despites me trying to investigate why, it didn't copied everything. So the new plan is to stop the VM, make a local copy of the disk, start the VM, and then copy that one over the new server, prepare for the new host (firewall, public ip, etc), and sync from the old VM to the new host. All is ok with this plan ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer wrote: > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer > >> wrote: > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > >> >> I think starting around 0900 UTC on Friday of next week (12th Feb) > >> >> should be possible. We should be done with 3.7.8 before that and can > >> >> afford a bit of downtime then. In case any assistance is needed post the > >> >> migration, we can have folks around the clock to help. > >> >> > >> >> If migration fails for an unforeseen reason, would we be able to > >> >> rollback and maintain status quo? > >> > > >> > Yes. Worst case, I think people would just redo a few reviews or push > >> > again patch. > >> > > >> > Also, since gerrit is critical, I wonder what is the support of gerrit > >> > for slaves and replication. > >> > >> Michael are you okay with the time? If you are, I think we should > >> announce the migration. > > > > I am ok. > > So as discussed in the community meeting, we will be announcing the > migration and downtime. Vijay, you said you required the exact > schedule? This is what I expect > > Friday 12th Feb 2016 > > 0900 UTC : build.gluster.org and review.gluster.org are taken down and > migration begins. > <1-2 hrs?>: Michael copies over data onto the RH community infra and > sets up the VMs. > : The DNS records are updated, and some time is for it to propogate. > <3hr?>: Verify everything is working well (I can help with this). We'd > possibly need to run a regression job, so this will take longest I > think. > > 1700UTC : We announce the finish of the migration and open > the services back up. If migration failed, we bring the existing > servers back on, and continue on. So, for people wanting to know, the migration has started. 1) gerrit -- gerrit seems to be ok, it has a new IP 66.187.224.201 people who used to have access there need to contact me so I can explain/create required account on the new virt host. the old VM is shutdown for now. I will make a backup copy of the disk. 2) jenkins -- so the preparation of the migration didn't worked as well, so we have to copy the VM when offline, and then prepare it later ( ie, import in libvirt, adjust network, etc). So jenkins master is offline (sorry manu for your tcpdump monitoring), we are making a copy of the disk that is gonna take between 3 and 14h (most likely < 4h, but rsync estimation is moving a lot), and then restart like before. If anything is broken, please do not hesitate to tell us. I will send a email to tell when people can start again to submit patch and stuff :) -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit : > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer wrote: > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer > >> wrote: > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > >> >> I think starting around 0900 UTC on Friday of next week (12th Feb) > >> >> should be possible. We should be done with 3.7.8 before that and can > >> >> afford a bit of downtime then. In case any assistance is needed post the > >> >> migration, we can have folks around the clock to help. > >> >> > >> >> If migration fails for an unforeseen reason, would we be able to > >> >> rollback and maintain status quo? > >> > > >> > Yes. Worst case, I think people would just redo a few reviews or push > >> > again patch. > >> > > >> > Also, since gerrit is critical, I wonder what is the support of gerrit > >> > for slaves and replication. > >> > >> Michael are you okay with the time? If you are, I think we should > >> announce the migration. > > > > I am ok. > > So as discussed in the community meeting, we will be announcing the > migration and downtime. Vijay, you said you required the exact > schedule? This is what I expect > > Friday 12th Feb 2016 > > 0900 UTC : build.gluster.org and review.gluster.org are taken down and > migration begins. > <1-2 hrs?>: Michael copies over data onto the RH community infra and > sets up the VMs. > : The DNS records are updated, and some time is for it to propogate. > <3hr?>: Verify everything is working well (I can help with this). We'd > possibly need to run a regression job, so this will take longest I > think. > > 1700UTC : We announce the finish of the migration and open > the services back up. If migration failed, we bring the existing > servers back on, and continue on. > > > Also, Michael are you happy migrating just one server or both? ideally, I would prefer both, but as i had trouble to move the jenkins VM while it was running, I couldn't prepare the network setup. I would need to open a few ports, etc, so that requires tickets to IT. > I case > you can only do one server we'd like Jenkins to migrated out. The problem is that this requires tome network changes, unlike the gerrit one, where the work was already done. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer wrote: > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer wrote: >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : >> >> I think starting around 0900 UTC on Friday of next week (12th Feb) >> >> should be possible. We should be done with 3.7.8 before that and can >> >> afford a bit of downtime then. In case any assistance is needed post the >> >> migration, we can have folks around the clock to help. >> >> >> >> If migration fails for an unforeseen reason, would we be able to >> >> rollback and maintain status quo? >> > >> > Yes. Worst case, I think people would just redo a few reviews or push >> > again patch. >> > >> > Also, since gerrit is critical, I wonder what is the support of gerrit >> > for slaves and replication. >> >> Michael are you okay with the time? If you are, I think we should >> announce the migration. > > I am ok. So as discussed in the community meeting, we will be announcing the migration and downtime. Vijay, you said you required the exact schedule? This is what I expect Friday 12th Feb 2016 0900 UTC : build.gluster.org and review.gluster.org are taken down and migration begins. <1-2 hrs?>: Michael copies over data onto the RH community infra and sets up the VMs. : The DNS records are updated, and some time is for it to propogate. <3hr?>: Verify everything is working well (I can help with this). We'd possibly need to run a regression job, so this will take longest I think. 1700UTC : We announce the finish of the migration and open the services back up. If migration failed, we bring the existing servers back on, and continue on. Also, Michael are you happy migrating just one server or both? I case you can only do one server we'd like Jenkins to migrated out. > -- > Michael Scherer > Sysadmin, Community Infrastructure and Platform, OSAS > > ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit : > On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer wrote: > > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > >> I think starting around 0900 UTC on Friday of next week (12th Feb) > >> should be possible. We should be done with 3.7.8 before that and can > >> afford a bit of downtime then. In case any assistance is needed post the > >> migration, we can have folks around the clock to help. > >> > >> If migration fails for an unforeseen reason, would we be able to > >> rollback and maintain status quo? > > > > Yes. Worst case, I think people would just redo a few reviews or push > > again patch. > > > > Also, since gerrit is critical, I wonder what is the support of gerrit > > for slaves and replication. > > Michael are you okay with the time? If you are, I think we should > announce the migration. I am ok. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer wrote: > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : >> I think starting around 0900 UTC on Friday of next week (12th Feb) >> should be possible. We should be done with 3.7.8 before that and can >> afford a bit of downtime then. In case any assistance is needed post the >> migration, we can have folks around the clock to help. >> >> If migration fails for an unforeseen reason, would we be able to >> rollback and maintain status quo? > > Yes. Worst case, I think people would just redo a few reviews or push > again patch. > > Also, since gerrit is critical, I wonder what is the support of gerrit > for slaves and replication. Michael are you okay with the time? If you are, I think we should announce the migration. > -- > Michael Scherer > Sysadmin, Community Infrastructure and Platform, OSAS > > ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit : > On 02/06/2016 07:37 AM, Michael Scherer wrote: > > Le samedi 06 février 2016 à 05:41 -0500, Kaleb Keithley a écrit : > >> > >> - Original Message - > >>> From: "Kaushal M" > >>> > >>> Hi Vijay. > >>> > >>> I spoke with Michael about migrating off of iWeb. > >> > >> Another option we discussed yesterday (sorry, should have thought to have > >> you join us) with Karanbir, in the context of using more of CentOS CI, was > >> moving gerrit to gerrithub. > >> > >> (Keeping our own jenkins master on Rackspace, but using slaves in CentOS > >> CI.) > >> > >> The advantage is that gerrithub is then responsible for keeping gerrit up > >> to date (and up). > >> > >> KB and Amye will be sending email about this soon. Watch this space. > > > > So as I suspect that gerrithub is gonna take some time before we do > > evaluate everything, I will proceed with the migration of gerrit to RH > > DC for now. We would be likely in a better shape for a migration > > anywhere from here. > > > > And when I say "8h", that's a worst case. I suspect it would be better, > > but I propose to schedule a documentation day or something where we can > > put developpers to good use when gerrit is down and copied. > > > > > I think starting around 0900 UTC on Friday of next week (12th Feb) > should be possible. We should be done with 3.7.8 before that and can > afford a bit of downtime then. In case any assistance is needed post the > migration, we can have folks around the clock to help. > > If migration fails for an unforeseen reason, would we be able to > rollback and maintain status quo? Yes. Worst case, I think people would just redo a few reviews or push again patch. Also, since gerrit is critical, I wonder what is the support of gerrit for slaves and replication. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
On 02/06/2016 07:37 AM, Michael Scherer wrote: Le samedi 06 février 2016 à 05:41 -0500, Kaleb Keithley a écrit : - Original Message - From: "Kaushal M" Hi Vijay. I spoke with Michael about migrating off of iWeb. Another option we discussed yesterday (sorry, should have thought to have you join us) with Karanbir, in the context of using more of CentOS CI, was moving gerrit to gerrithub. (Keeping our own jenkins master on Rackspace, but using slaves in CentOS CI.) The advantage is that gerrithub is then responsible for keeping gerrit up to date (and up). KB and Amye will be sending email about this soon. Watch this space. So as I suspect that gerrithub is gonna take some time before we do evaluate everything, I will proceed with the migration of gerrit to RH DC for now. We would be likely in a better shape for a migration anywhere from here. And when I say "8h", that's a worst case. I suspect it would be better, but I propose to schedule a documentation day or something where we can put developpers to good use when gerrit is down and copied. I think starting around 0900 UTC on Friday of next week (12th Feb) should be possible. We should be done with 3.7.8 before that and can afford a bit of downtime then. In case any assistance is needed post the migration, we can have folks around the clock to help. If migration fails for an unforeseen reason, would we be able to rollback and maintain status quo? Thanks, Vijay ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Le samedi 06 février 2016 à 05:41 -0500, Kaleb Keithley a écrit : > > - Original Message - > > From: "Kaushal M" > > > > Hi Vijay. > > > > I spoke with Michael about migrating off of iWeb. > > Another option we discussed yesterday (sorry, should have thought to have you > join us) with Karanbir, in the context of using more of CentOS CI, was moving > gerrit to gerrithub. > > (Keeping our own jenkins master on Rackspace, but using slaves in CentOS CI.) > > The advantage is that gerrithub is then responsible for keeping gerrit up to > date (and up). > > KB and Amye will be sending email about this soon. Watch this space. So as I suspect that gerrithub is gonna take some time before we do evaluate everything, I will proceed with the migration of gerrit to RH DC for now. We would be likely in a better shape for a migration anywhere from here. And when I say "8h", that's a worst case. I suspect it would be better, but I propose to schedule a documentation day or something where we can put developpers to good use when gerrit is down and copied. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
- Original Message - > From: "Kaushal M" > > Hi Vijay. > > I spoke with Michael about migrating off of iWeb. Another option we discussed yesterday (sorry, should have thought to have you join us) with Karanbir, in the context of using more of CentOS CI, was moving gerrit to gerrithub. (Keeping our own jenkins master on Rackspace, but using slaves in CentOS CI.) The advantage is that gerrithub is then responsible for keeping gerrit up to date (and up). KB and Amye will be sending email about this soon. Watch this space. > > He's already done trial runs of migrating the b.g.o and r.g.o VMs into > the community hardware we have lying around. It's easier to migrate > these VMs into the community cage, than into rackspace. > > We know we can do this migration. But this migration would require a > downtime of about 8 hours for both Gerrit and Jenkins. If we can come > up with a time for this downtime, we can get the migration done by the > end of the upcoming week. One way we could do this (credit to misc) > would be to spend a day as a documentation day, when all developers > instead of developing work on getting documentation fixed up. > > We also need to update Gerrit, but Michael suggested doing it after > the migration. That would possibly also require some downtime later. > > In any case, what we want now is a time set for doing the migration. > Can you help us set this time? > > Thanks, > Kaushal > ___ > Gluster-infra mailing list > Gluster-infra@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-infra > ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
[Gluster-infra] Migrating Gerrit and Jenkins out of iWeb
Hi Vijay. I spoke with Michael about migrating off of iWeb. He's already done trial runs of migrating the b.g.o and r.g.o VMs into the community hardware we have lying around. It's easier to migrate these VMs into the community cage, than into rackspace. We know we can do this migration. But this migration would require a downtime of about 8 hours for both Gerrit and Jenkins. If we can come up with a time for this downtime, we can get the migration done by the end of the upcoming week. One way we could do this (credit to misc) would be to spend a day as a documentation day, when all developers instead of developing work on getting documentation fixed up. We also need to update Gerrit, but Michael suggested doing it after the migration. That would possibly also require some downtime later. In any case, what we want now is a time set for doing the migration. Can you help us set this time? Thanks, Kaushal ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
