Re: Major dns issues
On Mon, Dec 14, 2009 at 10:27 PM, Mike McGrath mmcgr...@redhat.com wrote: So I woke up today and we're still having dns issues on at least one of my hosts. Could everyone that has access please do a dig fedoraproject.org on all their hosts and tell me if any of them cannot resolve? -Mike working from Bangladesh. cheers // Imtiaz Rahi Ogden Nash http://www.brainyquote.com/quotes/authors/o/ogden_nash.html - The trouble with a kitten is that when it grows up, it's always a cat. ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: [FOSDEM'10] Community infrastructure talk
Dea Thanks a lot On Mon, Dec 14, 2009 at 10:52 PM, Jeroen van Meeuwen kana...@kanarip.comwrote: On 12/14/2009 10:46 PM, Jeroen van Meeuwen wrote: On 12/14/2009 09:36 PM, Frederic Hornain wrote: Dear *, Would someone be interested to make a talk on Fedora Community infrastructure at FOSDEM'10 - Belgium -? * FedoraCommunity, opensourced LaunchPad, Bugzilla extensions, etc... BTW, as the rule have changed at FOSDEM, you will do your talk with other distributions on the same subject. Thanks for your time and your help. Sure, here's a candidate! I already asked permission from our fearless leader; [22:51:30] kanarip mmcgrath, that is, if you'll let me go tete-a-tete with other distributions on the topic of infrastructure ;-))) -- Jeroen ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list -- - Fedora-ambassadors-list mailing list fedora-ambassadors-l...@redhat.com Olpc mailing list olpc-o...@laptop.org ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
SOP creation made easier
Since we were directly inspired to create the Marketing SOPs by seeing how much good it's done Infrastructure to have them, I thought I'd share some back - we made a template and a SOP for making SOPs to make the SOP-generation process itself easier on folks who might not want to spend a lot of time on wiki editing. https://fedoraproject.org/wiki/Creating_a_Marketing_SOP https://fedoraproject.org/wiki/Template:Marketing_SOP They're both marketing-specific at the moment, but could be cloned for Infrastructure in a few minutes. Another way to make it even easier to make SOPs would be to use http://www.mediawiki.org/wiki/Extension:InputBox, which would give you a text field with a button next to it; if you type a title into the text field and hit the button, it throws you straight into the Edit page for a new page with that title, with a template you prespecify already in there. If someone wants to implement that extension, I'd be happy to do the wiki work needed to document/use it for various types of pages-that-should-be-easy-to-create once the extension is installed. --Mel Original Message Subject: Marketing SOPs Date: Tue, 15 Dec 2009 13:24:30 -0500 From: Mel Chua m...@redhat.com Reply-To: For discussions about marketing and expanding the Fedora user base fedora-marketing-l...@redhat.com To: For discussions about marketing and expanding the Fedora user base fedora-marketing-l...@redhat.com One of the projects we have between now and Alpha is to document how to create each of our primary release deliverables in the form of a SOP[0]. You can see our current SOPs here: https://fedoraproject.org/wiki/Category:Marketing_SOPs#How_to_contribute. Some are more polished than others. Of course, I had to start by making a SOP on how to make SOPs. :) https://fedoraproject.org/wiki/Creating_a_Marketing_SOP It makes use of the (also new) Marketing SOP template, https://fedoraproject.org/wiki/Template:Marketing_SOP. That template is meant to be a suggestion/aide to make life easier, so don't feel compelled to stick to it if you think a different format will work better for you. --Mel [0] Standard Operating Procedure - we used to call them HOWTOs, but I switched the term to be more consistent with the terminology other teams use (specifically, https://fedoraproject.org/wiki/Category:Infrastructure_SOPs). The HOWTO pages have been redirected/recategorized to reflect this. -- Fedora-marketing-list mailing list fedora-marketing-l...@redhat.com https://www.redhat.com/mailman/listinfo/fedora-marketing-list ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Voting app offline time?
On Tue, Dec 15, 2009 at 1:46 PM, Mike McGrath mmcgr...@redhat.com wrote: On Tue, 15 Dec 2009, Paul W. Frields wrote: On Mon, Dec 14, 2009 at 12:08:15PM -0500, Seth Vidal wrote: On Mon, 14 Dec 2009, Mike McGrath wrote: On Mon, 14 Dec 2009, Paul W. Frields wrote: On Mon, Dec 14, 2009 at 09:51:09AM -0500, Paul W. Frields wrote: On Tue, Dec 15, 2009 at 12:14:46AM +1000, Nigel Jones wrote: We had some the original DB downtime ~1-2 hours, then downtime due to fas not been around ~1-2 hours, plus a DB downtime of ~2-3 hours and a VPN downtime of about an hour. All in all, I think it meets the 1 day extension criteria. Yup, top-side estimates add up to 8 hours, so we should extend the voting by one day as agreed. Make it so! ;-) And... after making the announcement, it's come to light that there are some DNS issues outstanding which could potentially have presented a problem to voters. This one's been difficult to confirm. We know there were some issues that have been resolved and so far slicehost is the only one having issues, which is almost certainly on us but so far no one has actually complained about any issues which is surprising to me but does point to a not widly spread issue. It could also mean they are trying to email about the problems but cannot b/c their mailer won't let them email to a domain they cannot find. :( This is how I noticed the problem this morning from slicehost. What's the status on the DNS problems? Were they resolved, and if so, approximately when? That will tell me what we need to do for a further extension for voting. Thanks to all of you who worked on this problem and the rest of the move. AFAIK, some people had issues for an hour or two. But having said that.. I'm begining to think more and more this is a bind bug because I still can't resolve fedoraproject.org from my slicehost: dig @67.207.128.4 fedoraproject.org I'd think if this was a wide spread problem people would be screaming about it. For what it is worth I think the one day extension with the previous extension is plenty but adding another just to be safe isn't going to upset anyone I don't think. So if you think just to be cautious it would be good to extend the voting through the 17th I don't think anyone would object. John ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Voting app offline time?
On Tue, Dec 15, 2009 at 01:51:15PM -0600, inode0 wrote: On Tue, Dec 15, 2009 at 1:46 PM, Mike McGrath mmcgr...@redhat.com wrote: On Tue, 15 Dec 2009, Paul W. Frields wrote: On Mon, Dec 14, 2009 at 12:08:15PM -0500, Seth Vidal wrote: On Mon, 14 Dec 2009, Mike McGrath wrote: On Mon, 14 Dec 2009, Paul W. Frields wrote: On Mon, Dec 14, 2009 at 09:51:09AM -0500, Paul W. Frields wrote: On Tue, Dec 15, 2009 at 12:14:46AM +1000, Nigel Jones wrote: We had some the original DB downtime ~1-2 hours, then downtime due to fas not been around ~1-2 hours, plus a DB downtime of ~2-3 hours and a VPN downtime of about an hour. All in all, I think it meets the 1 day extension criteria. Yup, top-side estimates add up to 8 hours, so we should extend the voting by one day as agreed. Make it so! ;-) And... after making the announcement, it's come to light that there are some DNS issues outstanding which could potentially have presented a problem to voters. This one's been difficult to confirm. We know there were some issues that have been resolved and so far slicehost is the only one having issues, which is almost certainly on us but so far no one has actually complained about any issues which is surprising to me but does point to a not widly spread issue. It could also mean they are trying to email about the problems but cannot b/c their mailer won't let them email to a domain they cannot find. :( This is how I noticed the problem this morning from slicehost. What's the status on the DNS problems? Were they resolved, and if so, approximately when? That will tell me what we need to do for a further extension for voting. Thanks to all of you who worked on this problem and the rest of the move. AFAIK, some people had issues for an hour or two. But having said that.. I'm begining to think more and more this is a bind bug because I still can't resolve fedoraproject.org from my slicehost: dig @67.207.128.4 fedoraproject.org I'd think if this was a wide spread problem people would be screaming about it. For what it is worth I think the one day extension with the previous extension is plenty but adding another just to be safe isn't going to upset anyone I don't think. So if you think just to be cautious it would be good to extend the voting through the 17th I don't think anyone would object. We'll just stick with what we announced and planned, to make things easier. -- Paul W. Frieldshttp://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Mea Culpa and Apology
On Tue, 15 Dec 2009, Stephen John Smoogen wrote: One of the big problems with the move from PHX1 to PHX2 has been the renaming of hosts. This was a big mistake on my part and made life very difficult for the Fedora people who worked over the weekend to get it working and running into constant headaches. I apologize and owe everyone a lot. Stephen, don't sweat it too much. It happens to everyone. In this case the lesson is: If you are making a big change, do not add any other big changes (or small changes) if you can avoid it. All change is bad. More change is MORE bad. Thanks, -sv ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Mea Culpa and Apology
@Stephen, No matter how experienced or how good one is, shit happens to everybody from time to time. Cheer up. On Tue, 2009-12-15 at 15:08 -0500, Seth Vidal wrote: On Tue, 15 Dec 2009, Stephen John Smoogen wrote: One of the big problems with the move from PHX1 to PHX2 has been the renaming of hosts. This was a big mistake on my part and made life very difficult for the Fedora people who worked over the weekend to get it working and running into constant headaches. I apologize and owe everyone a lot. Stephen, don't sweat it too much. It happens to everyone. In this case the lesson is: If you are making a big change, do not add any other big changes (or small changes) if you can avoid it. All change is bad. More change is MORE bad. Thanks, -sv ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Bastion
I've updated the bastion.fedoraproject.org ns record to point to bastion3. We still have a lot of work to do but we'll go ahead and stick with this ssh key: bastion,bastion.fedoraproject.org,209.132.182.51 ssh-rsa B3NzaC1yc2EBIwAAAQEAuy7kT2nhJRYyi4Af28KzHJIgFzD84TE9lzDhGB6a9XvCpdI5n0RGZ2bznXR+SRbLjYrseVD+S7ScDFB6+TrXwZpWSM3aJiTJPCzGpI5SRZIaJ4IfAsY9t6n4K3QMmXh+BPGYrUb7ohwtyZADy+7/4E4YOc4cEMxzBEML2D06ERSmCybX01rAHw81Rx/4KN5oH05ynqAcfjF9f1tesDZIZ/NOMkVtMLBWwLf9iKkOKj8dAGBJy/uhidOkVm8bfI9X3NyKwjXC/V8RCeCv/n77y40yvEKYUy8iKOyHQHCVXmTck+6bX1Ity9XYFlljOLaoXPmrO2r6VlyzL0tDKbqtkw== -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Marketing Infrastructure projects, update 2009-12-15
Since a substantial portion of what Marketing is working on between now and Alpha (March 2) is getting infrastructure/tools up so we can do our work efficiently for the rest of the cycle, here's a quick update on our two big marketing-related infrastructure projects, also sent to Logistics for cross-team goodness (and mayyybe packaging help? maybe?). Project 1: Zikula This is for https://fedoraproject.org/wiki/Fedora_Insight (which includes FWN, and will be largely reusable for Docs), and progress is being tracked on the logistics list, https://admin.fedoraproject.org/mailman/listinfo/logistics. We are currently (still) on publictest6. We want to have soft-launched on production by Alpha (March 3). Simon has an overview of the remaining work that needs to be done before we can be puppetized (http://lists.fedoraproject.org/pipermail/logistics/2009-December/000303.html). He's also updated the how to set up a zikula sandbox instructions, https://fedoraproject.org/wiki/How_to_set_up_a_Zikula_sandbox. We are blocked by final fixes to the theme (design/CSS help needed!) which will then need to be packaged, as well as licensing issues with Scribite (http://lists.fedoraproject.org/pipermail/logistics/2009-December/000301.html) but it looks like the interaction between Fedora and Zikula folks is beginning to snowball (http://lists.fedoraproject.org/pipermail/logistics/2009-December/000302.html)... I'd like to see if we can get packagers + upstream on IRC at the same time to just Work Out these licensing issues at one go, because asynch communication is Very Slow. After those blockers are out, I *think* we'll be go for puppetizing. Project 2: Limesurvey This is for https://fedoraproject.org/wiki/Marketing_research#Infrastructure, and seems like it should be relatively straightforward (but then again, that's what I thought for zikula...) Robyn Bergeron and Ryan Rix are driving this from the Marketing side of things, but I'm temporarily stepping in for Ryan since he's afk for exams this week. As far as I know, there's only one package involved in this (limesurvey) which is still under review (https://bugzilla.redhat.com/show_bug.cgi?id=508817). It *sounds* from the comments like it should be done and ready for approval, but a reviewer (*coughIancough* is needed). In the meantime, I tried installing the latest SRPM from Eric (Sparks) on pt6, and failboated, details here: https://fedoraproject.org/wiki/How_to_set_up_a_limesurvey_sandbox#Install_the_package. Once the package is approved and installed on pt6, I *think* that's all we'll need (we might want to change the logo, but that's trivial and I really can't think of anything else atm). In the meantime-meantime, because we need to get the FUDCon survey up yesterday, we're giving limeservice $10 to host a 250-response survey for us so we can start administering that survey tomorrow. That means we'll have a db ready to migrate whenever our limesurvey instance goes up, plus people with familiarity administering limesurvey software via the web interface (some of us, myself included, have used it before and know the rudiments). Questions? Comments? Am I missing something? (Is this helpful?) --Mel ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Post Move Stuff
So the move itself is over but there is still a lot of work to be done. At the moment some of our normally redundant services (koji, vpn) aren't redundant. Also I'm pretty sure puppet is still failing on some hosts but that we can fix at our own pace. Also just a note, smooge and I are likely going to be in recovery mode the next couple of days. So if something seems broken please open a ticket. So what is left? - We started renaming everything, we'll need to finish that (involves renames, re-keying things, etc) - New network map. We went from essentially having a single network to having 3 networks: - A build network - A storage network - A public network - Training on one of our new server types, bxen*. These hosts are dedicated to build and releng activities. This was done for a couple of reasons, most of which are organizational. It will also allow us to more easily predict growth needs for the buildsystem in the future. - Figure out what to do about proxy servers in PHX2. We can go a load balancer route or a heartbeat route but I'm not totally convinced we need two proxy servers in PHX2 like we had in PHX1 though, because of the way network routing still works we'll have to figure out something HA - QA - The new QA boxes are ready to be configured, I'll be working with jlaska on this. It's the first kind of hosted by Infrastructure but not really run by infrastructure set of boxes. Similar to how the releng boxes work but the QA team is less close to Infrastructure then release engineering is. This will involve training and some new policies. - Host certification - this is something I've been working on but not enacted yet. Mostly a solid lookover everything based on a recent CSI doc. http://infrastructure.fedoraproject.org/csi/host-lifecycle-policy/en-US/html-single/#HostLifecycle-Host-Recertification In some organizations this the certification process will help bring about accountability. For us it's more about knowing what's going on, it's not like if someone accidentally certifies a box wrong they'll be in trouble but in our case a second pair of eyes will help. Even in the first trial run I did with smooge he discovered something I missed. There's also a lot of little things to do, especially with verifying things like IPTables and monitoring. -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
