Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
[PSF list removed] On 2013-01-18, M.-A. Lemburg wrote: In other words, the backdoor will likely have been open for several months. My thanks to all the work put in by volunteers. Has there been any consideration given to using different wiki software? It's my impression that MoinMoin has a quite poor record with regard to security: http://moinmo.in/SecurityFixes The abundance of past holes doesn't predict future ones but in general there seems to be a correlation. Whatever software we use, keeping the wiki separated (e.g. in its own VM) is definitely a good idea. Anytime you allow remote users to create content the risks are high. Regards, Neil ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
On 18.01.2013 19:59, Neil Schemenauer wrote: [PSF list removed] On 2013-01-18, M.-A. Lemburg wrote: In other words, the backdoor will likely have been open for several months. My thanks to all the work put in by volunteers. Has there been any consideration given to using different wiki software? It's my impression that MoinMoin has a quite poor record with regard to security: http://moinmo.in/SecurityFixes The abundance of past holes doesn't predict future ones but in general there seems to be a correlation. I think that's a misinterpretation. MoinMoin is used in a *lot* of places and so finding vulnerabilities becomes more attractive than for other similar software. I agree, though, that a security audit would probably not hurt :-) Perhaps they should have one of their GSoC students run such an audit this summer. Whatever software we use, keeping the wiki separated (e.g. in its own VM) is definitely a good idea. Anytime you allow remote users to create content the risks are high. True. Let's not overreact :-) Without the incident we would still be under the assumption that we have backups for everything... It also shows that we have to make a few enhancement to the way we do logging; but that's going to be a new thread. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 18 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2013-01-22: Python Meeting Duesseldorf ... 4 days to go : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
On Fri, Jan 18, 2013 at 3:51 PM, Paul Boddie p...@boddie.org.uk wrote: M.-A. Lemburg wrote: On 18.01.2013 19:59, Neil Schemenauer wrote: [PSF list removed] On 2013-01-18, M.-A. Lemburg wrote: In other words, the backdoor will likely have been open for several months. My thanks to all the work put in by volunteers. Has there been any consideration given to using different wiki software? It's my impression that MoinMoin has a quite poor record with regard to security: http://moinmo.in/SecurityFixes The abundance of past holes doesn't predict future ones but in general there seems to be a correlation. I think that's a misinterpretation. MoinMoin is used in a *lot* of places and so finding vulnerabilities becomes more attractive than for other similar software. Agreed. Just because the MoinMoin project has openly published advisories (and fixed vulnerabilities) doesn't mean that it has a poor record, or at least a record that is poorer than other software. I happen to be subscribed to notifications for MediaWiki, for example, and advisories are regularly published exhorting users to upgrade in order to fix various issues. We could spend substantial effort migrating to something else without any guarantee of improved security and with substantial inconvenience incurred. As I noted on a rather tiresome thread on the PSF list, throwing everything out in order to do things some other, supposedly better way is an unfortunate Python community tendency that we shouldn't indulge. I also think that using people's software and then abandoning it (and them) when we find something we don't like about it, instead of offering to improve it, is counterproductive if not a betrayal of those people. Speaking of improving it: on Wednesday, the PSF approved a grant to expedite development efforts that the MoinMoin team is putting in to using passlib for their password handling. ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
On Friday 18 January 2013 22:50:10 Brian Curtin wrote: Speaking of improving it: on Wednesday, the PSF approved a grant to expedite development efforts that the MoinMoin team is putting in to using passlib for their password handling. This is a most welcome development. Although there may be people who argue that usage of this library is overdue, any effort or initiative that can encourage more sharing and collaboration amongst Python Web projects and revive channels like the Web SIG, so that best practices can be propagated and projects may look after each other instead of justifying factionalism through the idea that there must be winners and losers, is an initiative worth supporting. Thanks for keeping us informed! Paul P.S. Personally, I'd either not heard of passlib or had forgotten about its existence, but then again I'm not doing password handling myself on a day-to-day basis. ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
M.-A. Lemburg wrote: I've been able to recover the pages from archive.org and have also tried Google cache (which failed due to limits on the number of allowed requests) and Yahoo/Bing cache. The latter worked, but only returns a small fraction of the pages we have had in the wiki - about 300+ pages. They are more recent than the archive.org ones, though, so I'm trying to merge the Yahoo archive ones back into the archive.org recovery. I recovered around 4500 pages from archive.org... in HTML. Reimar has a tool to convert them back into wiki markup, which we'll try to use to prepare an import. Meanwhile I'm also trying to see whether we can still extract some data from the broken VM image. It does show traces of the wiki file contents, so the data still exists on the image in some form. Noah already tried extundelete with no success. I'm going to give some of the other tools a try as well, e.g. ext4magic or PhotoRec. Phew, sounds like fun... thanks for everyone's work on this! Can someone explain (to PSF members list) how it ended up that there were no backups? I'm not trying to put anyone on the spot, just trying to (a) understand how this happened, making it so hard to recover, and (b) make sure that python.org and other important resources _are_ being backed up in a way that prevents this kind of thing from taking down services for a long time. Thanks, - Stephan ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
On Wednesday, January 16, 2013 at 10:05 AM, Stephan Deibel wrote: M.-A. Lemburg wrote: I've been able to recover the pages from archive.org (http://archive.org) and have also tried Google cache (which failed due to limits on the number of allowed requests) and Yahoo/Bing cache. The latter worked, but only returns a small fraction of the pages we have had in the wiki - about 300+ pages. They are more recent than the archive.org (http://archive.org) ones, though, so I'm trying to merge the Yahoo archive ones back into the archive.org (http://archive.org) recovery. I recovered around 4500 pages from archive.org (http://archive.org)... in HTML. Reimar has a tool to convert them back into wiki markup, which we'll try to use to prepare an import. Meanwhile I'm also trying to see whether we can still extract some data from the broken VM image. It does show traces of the wiki file contents, so the data still exists on the image in some form. Noah already tried extundelete with no success. I'm going to give some of the other tools a try as well, e.g. ext4magic or PhotoRec. Phew, sounds like fun... thanks for everyone's work on this! Can someone explain (to PSF members list) how it ended up that there were no backups? I'm not trying to put anyone on the spot, just trying to (a) understand how this happened, making it so hard to recover, and (b) make sure that python.org (http://python.org) and other important resources _are_ being backed up in a way that prevents this kind of thing from taking down services for a long time. Thanks, - Stephan Noah can expand on this as Infrastructure lead, but the short version is this - last year we got some beefy donations and hosting form OSU/OSL - this allows us to run our own VM infrastructure and isolate/spin up new servers at will (which is great). We've been slowly migrating the old services to the new systems. Our backups are currently handled via donated services to Tummy.com - in the transition, one of the things which had to be done was update those backups to point to the new virtual machines. This happened for some of the more mission critical virtual machines, but unfortunately one of the machines which fell through the cracks was the wiki machine, which hosts not just one Moin instance - but every single wiki the PSF hosts (including the members wiki, etc). Due to this, when the server was compromised, and the data deleted sometime around the 28th of december due to a 0 day exploit in Moin Moin, we lost all data from the move to OSU. We have coordinated with Noah, Sean at Tummy, etc to ensure all VMs hosted at the new setup are on a vigorous backup regime (offsite via Tummy). In addition to this, Noah is deploying an on site backup system / coordinating with OSU to ensure we have secondary / on site backups of everything. This ultimately comes down to a miscommunication/miss on our part, and we are examining ways to backfill our volunteer team with paid services and leveraging the services OSU offers to ensure we have good backups, support and other things we may lack today. Thanks go out to Noah for identifying and triaging the issue as best as possible and for Marc-Andre and others for looking to recover what they can from the compromised virtual machine and web archives. All of our infrastructure is managed by Chef (https://github.com/coderanger/psf-chef/tree/master/roles) and Ganeti at OSU. Currently being backed up are: virt-l4es2w.psf.osuosl.org virt-gwhg4e.psf.osuosl.org virt-wdiwcy.psf.osuosl.org virt-sxw5uy.psf.osuosl.org virt-oku3tm.psf.osuosl.org virt-h669vt.psf.osuosl.org virt-wzmlmm.psf.osuosl.org virt-ys0nco.psf.osuosl.org virt-7yvsjn.psf.osuosl.org virt-k4b2sa.psf.osuosl.org virt-ozvw2q.psf.osuosl.org virt-8joqck.psf.osuosl.org virt-et2yi0.psf.osuosl.org This also includes non PSF assets such as PyPy assets we are now hosting for free. As I said, this is both a combination of communication issues and volunteer load. The board is examining paid backup/leads where needed and/or leveraging OSU's services and administration. Jesse Noller Director, Python Software Foundation Chair, PyCon 2013 - http://us.pycon.org jnol...@gmail.com / jnol...@python.org +1 617-877-9135 ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
Jesse Noller wrote: Noah can expand on this as Infrastructure lead, but the short version is this - last year we got some beefy donations and hosting form OSU/OSL - this allows us to run our own VM infrastructure and isolate/spin up new servers at will (which is great). We've been slowly migrating the old services to the new systems. ... This also includes non PSF assets such as PyPy assets we are now hosting for free. As I said, this is both a combination of communication issues and volunteer load. The board is examining paid backup/leads where needed and/or leveraging OSU's services and administration. Great, thanks. I figured you were already on top of looking at what the PSF can do, but it seemed worth bringing up. Would it make sense to develop an infrastructure policy with a set of requirements for infrastructure? Then the PSF could pay someone (or appoint someone) to review everything periodically to make sure there are working audited backups, security patches, security scans, and whatever else is required by the policy. I don't know if that's too bureaucratic but I'd support it as a way to use PSF funds. - Stephan ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
On Wednesday, January 16, 2013 at 11:13 AM, Stephan Deibel wrote: Jesse Noller wrote: Noah can expand on this as Infrastructure lead, but the short version is this - last year we got some beefy donations and hosting form OSU/OSL - this allows us to run our own VM infrastructure and isolate/spin up new servers at will (which is great). We've been slowly migrating the old services to the new systems. ... This also includes non PSF assets such as PyPy assets we are now hosting for free. As I said, this is both a combination of communication issues and volunteer load. The board is examining paid backup/leads where needed and/or leveraging OSU's services and administration. Great, thanks. I figured you were already on top of looking at what the PSF can do, but it seemed worth bringing up. Would it make sense to develop an infrastructure policy with a set of requirements for infrastructure? Then the PSF could pay someone (or appoint someone) to review everything periodically to make sure there are working audited backups, security patches, security scans, and whatever else is required by the policy. I don't know if that's too bureaucratic but I'd support it as a way to use PSF funds. - Stephan Already working on a policy/job description/whatever you might call it. Just got side swiped with the Flu. ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
Re: [pydotorg-www] [PSF-Members] [Infrastructure] Wiki news?
On 16.01.2013 09:26, M.-A. Lemburg wrote: Meanwhile I'm also trying to see whether we can still extract some data from the broken VM image. It does show traces of the wiki file contents, so the data still exists on the image in some form. Noah already tried extundelete with no success. I'm going to give some of the other tools a try as well, e.g. ext4magic or PhotoRec. Update on the last bit: The tools were not able to recover the deleted files in the file structure, but were able to reconstruct a large number of files from the unallocated parts of the disk. Given that moin saves all revisions of a wiki page in the file system, with the file name being the only indication of the revision, those files may be useful in important cases, but there's no way to use them as input for automatic processing. The tools did also recover a number of log files that had been deleted, which allowed for a better analysis of what was used for the attack. Unfortunately, the logs for the important Dec 28 appear to have been overwritten by some other files, so I can't tell for sure whether the same attack as for the Debian wiki was used, but it is highly likely: http://wiki.debian.org/DebianWiki/SecurityIncident2012 The moinexec.py action plugin mentioned there was used on our wiki VM as well. In the course of this, the IP address from which the rm -r * originated turned up and we've contacted the ISP for more information. Several others played with the URLs as well, but only did harmless stuff. The attacker must have been in the know about the fact that wiki.python.org was also running the Jython wiki, since the availability via python.org and jython.org were checked after the rm run. Reimar is working on the conversion of the archive.org page dump to wiki format. I'll try to transmogrify the first Yahoo dump I ran into a suitable format for him to use tomorrow (the later runs returned fewer pages, which indicates that these caches can really only be used for short periods of time). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 17 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2013-01-22: Python Meeting Duesseldorf ... 5 days to go : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ pydotorg-www mailing list pydotorg-www@python.org http://mail.python.org/mailman/listinfo/pydotorg-www
