[OUTAGE] ovirt.org wiki outage report [07/12/14]
fyi, ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. REASON: === after logging into the openshift gear, seems like there was 99.9% space taken. RESOLUTION: === deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql using 'ctl_app restart'. space is still not great - at 84%, so we'll need to think soon how to resolve this. wiki is up now and seems to working well. infra - we should look into migrating to a new gear maybe? or extend again the existing one. Eyal. ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > fyi, > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. > > REASON: > === > after logging into the openshift gear, seems like there was 99.9% space taken. > > > RESOLUTION: > === > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql using > 'ctl_app restart'. > space is still not great - at 84%, so we'll need to think soon how to resolve > this. > > > > wiki is up now and seems to working well. > infra - we should look into migrating to a new gear maybe? or extend again > the existing one. Happened again today. I still maintain that we should move to a dedicated hosting on the phx2 cluster :) ( or request quota extensions ). -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
- Original Message - > From: "Michael Scherer" > To: infra@ovirt.org > Sent: Monday, December 8, 2014 1:03:01 PM > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > fyi, > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. > > > > REASON: > > === > > after logging into the openshift gear, seems like there was 99.9% space > > taken. > > > > > > RESOLUTION: > > === > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql > > using 'ctl_app restart'. > > space is still not great - at 84%, so we'll need to think soon how to > > resolve this. > > > > > > > > wiki is up now and seems to working well. > > infra - we should look into migrating to a new gear maybe? or extend again > > the existing one. > > Happened again today. > > I still maintain that we should move to a dedicated hosting on the phx2 > cluster :) > ( or request quota extensions ). whether if we'll migrate or not, that will take time and won't happen until the ILO issue will be resolved. for now we need to extend our volume there ASAP, cause it will happen again soon, not too many log files are there to delete/gzip. e. > > -- > Michael Scherer > Open Source and Standards, Sysadmin > > ___ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : > > - Original Message - > > From: "Michael Scherer" > > To: infra@ovirt.org > > Sent: Monday, December 8, 2014 1:03:01 PM > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > > fyi, > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. > > > > > > REASON: > > > === > > > after logging into the openshift gear, seems like there was 99.9% space > > > taken. > > > > > > > > > RESOLUTION: > > > === > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql > > > using 'ctl_app restart'. > > > space is still not great - at 84%, so we'll need to think soon how to > > > resolve this. > > > > > > > > > > > > wiki is up now and seems to working well. > > > infra - we should look into migrating to a new gear maybe? or extend again > > > the existing one. > > > > Happened again today. > > > > I still maintain that we should move to a dedicated hosting on the phx2 > > cluster :) > > ( or request quota extensions ). > > whether if we'll migrate or not, that will take time and won't happen until > the ILO issue will be resolved. > for now we need to extend our volume there ASAP, cause it will happen again > soon, not too many log files > are there to delete/gzip. Agreed. We did request already a quota extension, IIRC, so I guess that was not sufficient ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
- Original Message - > From: "Michael Scherer" > To: "Eyal Edri" > Cc: infra@ovirt.org > Sent: Monday, December 8, 2014 2:28:46 PM > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : > > > > - Original Message - > > > From: "Michael Scherer" > > > To: infra@ovirt.org > > > Sent: Monday, December 8, 2014 1:03:01 PM > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > > > fyi, > > > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. > > > > > > > > REASON: > > > > === > > > > after logging into the openshift gear, seems like there was 99.9% space > > > > taken. > > > > > > > > > > > > RESOLUTION: > > > > === > > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql > > > > using 'ctl_app restart'. > > > > space is still not great - at 84%, so we'll need to think soon how to > > > > resolve this. > > > > > > > > > > > > > > > > wiki is up now and seems to working well. > > > > infra - we should look into migrating to a new gear maybe? or extend > > > > again > > > > the existing one. > > > > > > Happened again today. > > > > > > I still maintain that we should move to a dedicated hosting on the phx2 > > > cluster :) > > > ( or request quota extensions ). > > > > whether if we'll migrate or not, that will take time and won't happen until > > the ILO issue will be resolved. > > for now we need to extend our volume there ASAP, cause it will happen again > > soon, not too many log files > > are there to delete/gzip. > > Agreed. We did request already a quota extension, IIRC, so I guess that > was not sufficient ? > looking at the mysql db, it seems to be taking most of the space (9.8GB data file). so i guess no choice but to extend it more. > -- > Michael Scherer > Open Source and Standards, Sysadmin > > ___ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : > Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : > > > > - Original Message - > > > From: "Michael Scherer" > > > To: infra@ovirt.org > > > Sent: Monday, December 8, 2014 1:03:01 PM > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > > > fyi, > > > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. > > > > > > > > REASON: > > > > === > > > > after logging into the openshift gear, seems like there was 99.9% space > > > > taken. > > > > > > > > > > > > RESOLUTION: > > > > === > > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql > > > > using 'ctl_app restart'. > > > > space is still not great - at 84%, so we'll need to think soon how to > > > > resolve this. > > > > > > > > > > > > > > > > wiki is up now and seems to working well. > > > > infra - we should look into migrating to a new gear maybe? or extend > > > > again > > > > the existing one. > > > > > > Happened again today. > > > > > > I still maintain that we should move to a dedicated hosting on the phx2 > > > cluster :) > > > ( or request quota extensions ). > > > > whether if we'll migrate or not, that will take time and won't happen until > > the ILO issue will be resolved. > > for now we need to extend our volume there ASAP, cause it will happen again > > soon, not too many log files > > are there to delete/gzip. > > Agreed. We did request already a quota extension, IIRC, so I guess that > was not sufficient ? Filled the form to get more, will tell you when I have a response. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
- Original Message - > From: "Michael Scherer" > To: infra@ovirt.org > Sent: Monday, December 8, 2014 8:07:33 AM > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : > > Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : > > > > > > - Original Message - > > > > From: "Michael Scherer" > > > > To: infra@ovirt.org > > > > Sent: Monday, December 8, 2014 1:03:01 PM > > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > > > > fyi, > > > > > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. > > > > > > > > > > REASON: > > > > > === > > > > > after logging into the openshift gear, seems like there was 99.9% > > > > > space > > > > > taken. > > > > > > > > > > > > > > > RESOLUTION: > > > > > === > > > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted > > > > > mysql > > > > > using 'ctl_app restart'. > > > > > space is still not great - at 84%, so we'll need to think soon how to > > > > > resolve this. > > > > > > > > > > > > > > > > > > > > wiki is up now and seems to working well. > > > > > infra - we should look into migrating to a new gear maybe? or extend > > > > > again > > > > > the existing one. > > > > > > > > Happened again today. > > > > > > > > I still maintain that we should move to a dedicated hosting on the phx2 > > > > cluster :) > > > > ( or request quota extensions ). > > > > > > whether if we'll migrate or not, that will take time and won't happen > > > until the ILO issue will be resolved. > > > for now we need to extend our volume there ASAP, cause it will happen > > > again soon, not too many log files > > > are there to delete/gzip. > > > > Agreed. We did request already a quota extension, IIRC, so I guess that > > was not sufficient ? > > Filled the form to get more, will tell you when I have a response. > -- > Michael Scherer > Open Source and Standards, Sysadmin > > ___ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > Went out again this afternoon (12/9), but managed to squeeze out a little more disk space running rhc app tidy. (Thanks to Jason and Justin for the assist!) If there is any way I can help expedite more disk quota, please let me know! BKP -- Brian Proffitt Community Liaison oVirt Open Source and Standards, Red Hat - http://community.redhat.com Phone: +1 574 383 9BKP IRC: bkp @ OFTC ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
- Original Message - > From: "Brian Proffitt" > To: infra@ovirt.org > Sent: Wednesday, December 10, 2014 1:00:48 AM > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > - Original Message - > > From: "Michael Scherer" > > To: infra@ovirt.org > > Sent: Monday, December 8, 2014 8:07:33 AM > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : > > > Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : > > > > > > > > - Original Message - > > > > > From: "Michael Scherer" > > > > > To: infra@ovirt.org > > > > > Sent: Monday, December 8, 2014 1:03:01 PM > > > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > > > > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > > > > > fyi, > > > > > > > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 > > > > > > IST. > > > > > > > > > > > > REASON: > > > > > > === > > > > > > after logging into the openshift gear, seems like there was 99.9% > > > > > > space > > > > > > taken. > > > > > > > > > > > > > > > > > > RESOLUTION: > > > > > > === > > > > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted > > > > > > mysql > > > > > > using 'ctl_app restart'. > > > > > > space is still not great - at 84%, so we'll need to think soon how > > > > > > to > > > > > > resolve this. > > > > > > > > > > > > > > > > > > > > > > > > wiki is up now and seems to working well. > > > > > > infra - we should look into migrating to a new gear maybe? or > > > > > > extend > > > > > > again > > > > > > the existing one. > > > > > > > > > > Happened again today. > > > > > > > > > > I still maintain that we should move to a dedicated hosting on the > > > > > phx2 > > > > > cluster :) > > > > > ( or request quota extensions ). > > > > > > > > whether if we'll migrate or not, that will take time and won't happen > > > > until the ILO issue will be resolved. > > > > for now we need to extend our volume there ASAP, cause it will happen > > > > again soon, not too many log files > > > > are there to delete/gzip. > > > > > > Agreed. We did request already a quota extension, IIRC, so I guess that > > > was not sufficient ? > > > > Filled the form to get more, will tell you when I have a response. > > -- > > Michael Scherer > > Open Source and Standards, Sysadmin > > > > ___ > > Infra mailing list > > Infra@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > Went out again this afternoon (12/9), but managed to squeeze out a little > more disk space running rhc app tidy. (Thanks to Jason and Justin for the > assist!) > > If there is any way I can help expedite more disk quota, please let me know! i think the best short term solution will be to get more space from openshift, right now i think we stand at less than 10G, so doubling it will give us some air to think how we best want to progress with the wiki. > > BKP > > > -- > Brian Proffitt > > Community Liaison > oVirt > Open Source and Standards, Red Hat - http://community.redhat.com > Phone: +1 574 383 9BKP > IRC: bkp @ OFTC > ___ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > > ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le mardi 09 décembre 2014 à 18:00 -0500, Brian Proffitt a écrit : > > - Original Message - > > From: "Michael Scherer" > > To: infra@ovirt.org > > Sent: Monday, December 8, 2014 8:07:33 AM > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : > > > Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : > > > > > > > > - Original Message - > > > > > From: "Michael Scherer" > > > > > To: infra@ovirt.org > > > > > Sent: Monday, December 8, 2014 1:03:01 PM > > > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > > > > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > > > > > fyi, > > > > > > > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. > > > > > > > > > > > > REASON: > > > > > > === > > > > > > after logging into the openshift gear, seems like there was 99.9% > > > > > > space > > > > > > taken. > > > > > > > > > > > > > > > > > > RESOLUTION: > > > > > > === > > > > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted > > > > > > mysql > > > > > > using 'ctl_app restart'. > > > > > > space is still not great - at 84%, so we'll need to think soon how > > > > > > to > > > > > > resolve this. > > > > > > > > > > > > > > > > > > > > > > > > wiki is up now and seems to working well. > > > > > > infra - we should look into migrating to a new gear maybe? or extend > > > > > > again > > > > > > the existing one. > > > > > > > > > > Happened again today. > > > > > > > > > > I still maintain that we should move to a dedicated hosting on the > > > > > phx2 > > > > > cluster :) > > > > > ( or request quota extensions ). > > > > > > > > whether if we'll migrate or not, that will take time and won't happen > > > > until the ILO issue will be resolved. > > > > for now we need to extend our volume there ASAP, cause it will happen > > > > again soon, not too many log files > > > > are there to delete/gzip. > > > > > > Agreed. We did request already a quota extension, IIRC, so I guess that > > > was not sufficient ? > > > > Filled the form to get more, will tell you when I have a response. > > -- > > Michael Scherer > > Open Source and Standards, Sysadmin > > > > ___ > > Infra mailing list > > Infra@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > Went out again this afternoon (12/9), but managed to squeeze out a little > more disk space running rhc app tidy. (Thanks to Jason and Justin for the > assist!) > > If there is any way I can help expedite more disk quota, please let me know! It take a few days. Last time we did ask, it was request => go to marketing for approval => go to the sysadmins team for action. As long as this in the marketing side, I have no visibility ( but can ask ). On the openshift sysadmin side, I know where to ping :) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le mercredi 10 décembre 2014 à 10:41 +0100, Michael Scherer a écrit : > Le mardi 09 décembre 2014 à 18:00 -0500, Brian Proffitt a écrit : > > > > - Original Message - > > > From: "Michael Scherer" > > > To: infra@ovirt.org > > > Sent: Monday, December 8, 2014 8:07:33 AM > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > > > Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : > > > > Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : > > > > > > > > > > - Original Message - > > > > > > From: "Michael Scherer" > > > > > > To: infra@ovirt.org > > > > > > Sent: Monday, December 8, 2014 1:03:01 PM > > > > > > Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] > > > > > > > > > > > > Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : > > > > > > > fyi, > > > > > > > > > > > > > > ovirt.org wiki page was down today from 01:33:07 IST till 10:22 > > > > > > > IST. > > > > > > > > > > > > > > REASON: > > > > > > > === > > > > > > > after logging into the openshift gear, seems like there was 99.9% > > > > > > > space > > > > > > > taken. > > > > > > > > > > > > > > > > > > > > > RESOLUTION: > > > > > > > === > > > > > > > deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted > > > > > > > mysql > > > > > > > using 'ctl_app restart'. > > > > > > > space is still not great - at 84%, so we'll need to think soon > > > > > > > how to > > > > > > > resolve this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > wiki is up now and seems to working well. > > > > > > > infra - we should look into migrating to a new gear maybe? or > > > > > > > extend > > > > > > > again > > > > > > > the existing one. > > > > > > > > > > > > Happened again today. > > > > > > > > > > > > I still maintain that we should move to a dedicated hosting on the > > > > > > phx2 > > > > > > cluster :) > > > > > > ( or request quota extensions ). > > > > > > > > > > whether if we'll migrate or not, that will take time and won't happen > > > > > until the ILO issue will be resolved. > > > > > for now we need to extend our volume there ASAP, cause it will happen > > > > > again soon, not too many log files > > > > > are there to delete/gzip. > > > > > > > > Agreed. We did request already a quota extension, IIRC, so I guess that > > > > was not sufficient ? > > > > > > Filled the form to get more, will tell you when I have a response. > > > -- > > > Michael Scherer > > > Open Source and Standards, Sysadmin > > > > > > ___ > > > Infra mailing list > > > Infra@ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > > > Went out again this afternoon (12/9), but managed to squeeze out a little > > more disk space running rhc app tidy. (Thanks to Jason and Justin for the > > assist!) > > > > If there is any way I can help expedite more disk quota, please let me know! > > It take a few days. Last time we did ask, it was request => go to > marketing for approval => go to the sysadmins team for action. > > As long as this in the marketing side, I have no visibility ( but can > ask ). On the openshift sysadmin side, I know where to ping :) So, today it was full again, and openshift people did contact me. We increased the quota, and while on it, I did the dump/restore needed to clean the ever increasing ibdata1 file ( http://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/ ). It took a few hours to process, and since things would not be fun without it, my internet connexion started to behave badly. But now, we are down to 6G out of 10G, so wait and see. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
