Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?

2013-01-16 Thread Jesse Noller 


On Wednesday, January 16, 2013 at 10:05 AM, Stephan Deibel wrote:

 M.-A. Lemburg wrote:
  I've been able to recover the pages from archive.org (http://archive.org) 
  and have also
  tried Google cache (which failed due to limits on the number of
  allowed requests) and Yahoo/Bing cache. The latter worked, but
  only returns a small fraction of the pages we have had in the wiki -
  about 300+ pages. They are more recent than the archive.org 
  (http://archive.org) ones,
  though, so I'm trying to merge the Yahoo archive ones back into the
  archive.org (http://archive.org) recovery.
  
  I recovered around 4500 pages from archive.org (http://archive.org)... in 
  HTML. Reimar
  has a tool to convert them back into wiki markup, which we'll
  try to use to prepare an import.
  
  Meanwhile I'm also trying to see whether we can still extract some
  data from the broken VM image. It does show traces of the wiki
  file contents, so the data still exists on the image in some
  form. Noah already tried extundelete with no success. I'm going
  to give some of the other tools a try as well, e.g. ext4magic
  or PhotoRec.
 
 
 
 Phew, sounds like fun... thanks for everyone's work on this!
 
 Can someone explain (to PSF members list) how it ended up that there 
 were no backups? I'm not trying to put anyone on the spot, just trying 
 to (a) understand how this happened, making it so hard to recover, and 
 (b) make sure that python.org (http://python.org) and other important 
 resources _are_ being 
 backed up in a way that prevents this kind of thing from taking down 
 services for a long time.
 
 Thanks,
 
 - Stephan

Noah can expand on this as Infrastructure lead, but the short version is this - 
last year we got some beefy donations and hosting form OSU/OSL - this allows us 
to run our own VM infrastructure and isolate/spin up new servers at will (which 
is great). We've been slowly migrating the old services to the new systems.

Our backups are currently handled via donated services to Tummy.com - in the 
transition, one of the things which had to be done was update those backups to 
point to the new virtual machines. This happened for some of the more mission 
critical virtual machines, but unfortunately one of the machines which fell 
through the cracks was the wiki machine, which hosts not just one Moin instance 
- but every single wiki the PSF hosts (including the members wiki, etc). 

Due to this, when the server was compromised, and the data deleted sometime 
around the 28th of december due to a 0 day exploit in Moin Moin, we lost all 
data from the move to OSU. 

We have coordinated with Noah, Sean at Tummy, etc to ensure all VMs hosted at 
the new setup are on a vigorous backup regime (offsite via Tummy). In addition 
to this, Noah is deploying an on site backup system / coordinating with OSU to 
ensure we have secondary / on site backups of everything.

This ultimately comes down to a miscommunication/miss on our part, and we are 
examining ways to backfill our volunteer team with paid services and leveraging 
the services OSU offers to ensure we have good backups, support and other 
things we may lack today.

Thanks go out to Noah for identifying and triaging the issue as best as 
possible and for Marc-Andre and others for looking to recover what they can 
from the compromised virtual machine and web archives.

All of our infrastructure is managed by Chef 
(https://github.com/coderanger/psf-chef/tree/master/roles) and Ganeti at OSU. 

Currently being backed up are:

virt-l4es2w.psf.osuosl.org
virt-gwhg4e.psf.osuosl.org
virt-wdiwcy.psf.osuosl.org
virt-sxw5uy.psf.osuosl.org
virt-oku3tm.psf.osuosl.org
virt-h669vt.psf.osuosl.org
virt-wzmlmm.psf.osuosl.org
virt-ys0nco.psf.osuosl.org
virt-7yvsjn.psf.osuosl.org
virt-k4b2sa.psf.osuosl.org
virt-ozvw2q.psf.osuosl.org
virt-8joqck.psf.osuosl.org
virt-et2yi0.psf.osuosl.org



This also includes non PSF assets such as PyPy assets we are now hosting for 
free. As I said, this is both a combination of communication issues and 
volunteer load. The board is examining paid backup/leads where needed and/or 
leveraging OSU's services and administration.

Jesse Noller
Director, Python Software Foundation
Chair, PyCon 2013 - http://us.pycon.org
jnol...@gmail.com / jnol...@python.org
+1 617-877-9135



___
pydotorg-www mailing list
pydotorg-www@python.org
http://mail.python.org/mailman/listinfo/pydotorg-www

Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?

2013-01-16 Thread Jesse Noller 


On Wednesday, January 16, 2013 at 11:13 AM, Stephan Deibel wrote:

 Jesse Noller wrote:
  Noah can expand on this as Infrastructure lead, but the short version is 
  this - last year we got some beefy donations and hosting form OSU/OSL - 
  this allows us to run our own VM infrastructure and isolate/spin up new 
  servers at will (which is great). We've been slowly migrating the old 
  services to the new systems.
  ...
  This also includes non PSF assets such as PyPy assets we are now hosting 
  for free. As I said, this is both a combination of communication issues and 
  volunteer load. The board is examining paid backup/leads where needed 
  and/or leveraging OSU's services and administration.
 
 
 
 Great, thanks. I figured you were already on top of looking at what the 
 PSF can do, but it seemed worth bringing up.
 
 Would it make sense to develop an infrastructure policy with a set of 
 requirements for infrastructure? Then the PSF could pay someone (or 
 appoint someone) to review everything periodically to make sure there 
 are working audited backups, security patches, security scans, and 
 whatever else is required by the policy. I don't know if that's too 
 bureaucratic but I'd support it as a way to use PSF funds.
 
 - Stephan 
Already working on a policy/job description/whatever you might call it. Just 
got side swiped with the Flu. 

___
pydotorg-www mailing list
pydotorg-www@python.org
http://mail.python.org/mailman/listinfo/pydotorg-www

Re: [pydotorg-www] [PSF-Board-Public] League of Awesome page is updated

2012-03-15 Thread Jesse Noller 
On Thu, Mar 15, 2012 at 11:43 AM, M.-A. Lemburg m...@egenix.com wrote:
 Noah Kantrowitz wrote:
 http://www.python.org/psf/league/ has been updated with our latest sponsors.

 Hi Noah,

 thanks for the update. I was a bit surprised that the PSF now
 sports a new page linking firms who have donated hardware or
 services without the board being involved.

 While I don't think that anyone from the board would object to
 having such a page, I do think that the board has to approve such
 important changes to the PSF website. Looking at the website
 repository, Martin added the original infrastructure page
 on 2012-02-23. Perhaps Martin could shed some light on this ?!

 I also noticed that we now have two directories mentioning the
 infrastructure sponsors:

 http://www.python.org/psf/infrastructure/
 http://www.python.org/psf/league/

 and that the League entry in the left hand menu only shows up in
 the PSF top-level

 http://www.python.org/psf/

 and not in the lower level entries, e.g.

 http://www.python.org/psf/about/

Steve Holden and I both gave Noah the nod for this. So blame us.
___
pydotorg-www mailing list
pydotorg-www@python.org
http://mail.python.org/mailman/listinfo/pydotorg-www

[pydotorg-www] Fwd: wiki.python.org

2011-10-28 Thread Jesse Noller 
FYI; from a user

-- Forwarded message --
From: Donald Stufft donald.stu...@gmail.com
Date: Fri, Oct 28, 2011 at 9:04 AM
Subject: wiki.python.org
To: jnol...@python.org


Hello!
I had asked your wife if you had anything to do with the python.org
website and she said you did and directed me to email you.
On wiki.python.org the logo in the top left should take you back to
http://python.org/. There doesn't currently appear to be an easy way
to get back, and there is a Front Page link already in the
navigation for going to the front page of the wiki. Alternatively some
sort of Return to Python.org Link in the same approximate location
would also suffice.
Thanks for your Time!
Donald Stufft
___
pydotorg-www mailing list
pydotorg-www@python.org
http://mail.python.org/mailman/listinfo/pydotorg-www