Re: [Python-Dev] python-dev summary for 2005-07-01 to 2005-07-15 [draft]
On 7/27/05, Steven Bethard <[EMAIL PROTECTED]> wrote: > Here's the draft for the first half of July. Thanks! This is looking great! (Although I can't help you with the GCC/C++ thread -- I've been avoiding that one like the plague myself. :-) To all the contributors, great job guys! -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP: Migrating the Python CVS to Subversion
I'd like to see the Python source be stored in Subversion instead of CVS, and on python.org instead of sf.net. To facilitate discussion, I have drafted a PEP describing the rationale for doing so, and the technical procedure to be performed. This is for discussion on python-dev and eventual BDFL pronouncement; if you see a reason not to make such a change, or if you would prefer a different procedure, please speak up. Encouragement and support is welcome, as well :-) Regards, Martin PEP: XXX Title: Migrating the Python CVS to Subversion Version: $Revision $ Last-Modified: $Date$ Author: Martin v. Löwis <[EMAIL PROTECTED]> Discussions-To: Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 14-Jul-2004 Post-History: 14-Jul-2004 Abstract The Python source code is currently managed in a CVS repository on sourceforge.net. This PEP proposes to move it to a subversion repository on svn.python.org. Rationale = This change has two aspects: moving from CVS to subversion, and moving from SourceForge to python.org. For each, a rationale will be given. Moving to Subversion CVS has a number of limitations that have been elimintation by Subversion. For the development of Python, the most notable improvements are: - ability to rename files and directories, and to remove directories, while keeping the history of these files. - support for change sets (sets of correlated changes to multiple files) through global revision numbers. - support for offline diffs, which is useful when creating patches. Moving to python.org SourceForge has kindly provided an important infrastructure for the past years. Unfortunately, the attention that SF received has also caused repeated overload situations in the past, to which the SF operators could not always respond in a timely manner. In particular, for CVS, they had to reduce the load on the primary CVS server by introducing a second, read-only CVS server for anonymous access. This server is regularly synchronized, but behind the the read-write CVS repository between synchronizations. As a result, users without commit access can see recent changes to the repository only with a delay. On python.org, it would be possible to make the repository accessible for anonymous access. Migration Procedure === To move the Python CVS repository, the following steps need to be executed. The steps are elaborated in more detail in the next sections. 1. Assign passwords for all current committers for use on svn.python.org. User names on SF and svn.python.org should be identical, unless some committer requests a different user name. 2. At the beginning of the migration, announce that the repository on SourceForge closed. 3. 24 hours after the last commit, download the CVS repository. 4. Convert the CVS repository into two subversion repositories, one for distutils and one for Python. 5. Publish the repositories for write access for all committers, and anonymous read access. 6. Disable CVS access on SF. Assign Passwords Currently, access to Subversion on svn.python.org uses WebDAV over https, using basic authentication. For this to work, authorized users need to provide a password. This mechanism should be used, atleast initially, for the Python CVS as well, since various committers also have a username/password pair for the www SVN repository already. Alternatives to password-based access include: - Subversion over SSH, using SSH key pairs. This would require to give committers accounts on the machine, which currently is ruled out by the administration policy of svn.python.org. - Subversion over WebDAV, using SSL client certificates. This would work, but would require to administrate a certificate authority. Downloading the CVS Repository == The CVS repository can be downloaded from http://cvs.sourceforge.net/cvstarballs/python-cvsroot.tar.bz2 Since this tarball is generated only once a day, some time after the repository freeze must pass before the tarball can be picked up. It should be verified that the last commit, as recorded on the python-commits mailing list, is indeed included in the tarball. Converting the CVS Repository = The Python CVS repository contains two modules: distutils and python. Keeping them together will produce quite long repository URLs, so it is more convenient if the Python CVS and the distutils CVS are converted into two separate repositories. As the repository format, fsfs should be used (requires Subversion 1.1). fsfs has the advantage of being more backup-friendly, as it allows to backup a repository incrementally, without requiring to run any dump commands. The conversion should be done using cvs2svn utility, available e.g. in the cvs2svn Debian package. The command for converting the Python repository is cvs2svn -q --encoding=latin1 --force-branch=cnri-16-start
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On 7/28/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > I'd like to see the Python source be stored in Subversion instead > of CVS, and on python.org instead of sf.net. To facilitate discussion, > I have drafted a PEP describing the rationale for doing so, and > the technical procedure to be performed. In principle I'm +1 on this (both aspects). I don't know enough to understand all the subtleties. I hope we're correctly estimating the effort required to manage the server and the users. Managing users is especially important -- if a user is compromised (as has happened in the past for python.org users) the whole repository is compromised. Now this could happen to SF users too, but I'm not sure that we know all the tricks in the book to prevent attacks; SF has been doing this for years and that's an aspect of SF that I trust (I think I've heard that they have even modified their SSH server to be stricter). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, Jul 28, 2005 at 10:00:00PM +0200, "Martin v. L?wis" wrote: > I'd like to see the Python source be stored in Subversion instead > of CVS, and on python.org instead of sf.net. +1, +1. > CVS has a number of limitations that have been elimintation by > Subversion. For the development of Python, the most notable improvements > are: > - ability to rename files and directories, and to remove directories, > while keeping the history of these files. > - support for change sets (sets of correlated changes to multiple > files) through global revision numbers. > - support for offline diffs, which is useful when creating patches. -- transactional operation - a changeset is either committed or rolled back at once; -- very effective (both in terms of speed and memory) tagging and branching; tags and branches are very easy to understand and use in SVN. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Jul 28, 2005, at 4:20 PM, Guido van Rossum wrote: > Managing users is especially important -- if a > user is compromised (as has happened in the past for python.org users) > the whole repository is compromised. Now this could happen to SF users > too, but I'm not sure that we know all the tricks in the book to > prevent attacks; SF has been doing this for years and that's an aspect > of SF that I trust (I think I've heard that they have even modified > their SSH server to be stricter). If you use the fsfs storage mechanism for subversion, it is somewhat simpler to verify that the repository is not compromised. Each commit is represented as a separate file, and thus old commits are never modified. Only new files are appended to the directory. If you have a filesystem that allows "append-only" permissions on a directory, you can enforce this directly. Additionally, it is possible in your backup script to verify that only new files were added and nothing else changed. Then at least you know how much you need to examine instead of having to treat the entire repository as possibly contaminated. James ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] installation of python on a Zaurus
Aahz wrote: > On Sun, Jul 24, 2005, Chuck Robey wrote: > >>I'm trying to get Python installed on a Zaurus, running OpenBSD. > > > While python-dev can be a good place to get questions like this > answered, many more people read comp.lang.python, and you should ask > there, too. Thanks ... I found my problem, once I'd put in enough prints into setup.py to look at what what happening (I do wish I knew how to use the python debugging features better than I do) and read the library code, I realized that it seemed to be finding the libraries by relying on the Linux habit of softlinking libraries like libtcl85.so.1.0 to libtcl85.so. This isn't done on the zaurus (until I forced it for the tcl libraries), and clearing this, got all my python 2.4.1 installed. I've played a bit with graphics already. so I guess I did it right. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, 2005-07-28 at 16:00, "Martin v. Löwis" wrote: > I'd like to see the Python source be stored in Subversion instead > of CVS +1 > , and on python.org instead of sf.net. +0 I know that SF has promised svn access to projects for a long time now, but I haven't heard anything from them in a long time. It's listed under their "Strategic Projects" but the last update to that news item was back in April. Question: do we wait for SF to make the service available (after getting more up-to-date status and a realistic timetable), or do we go straight to svn.python.org? > This is for discussion on python-dev and eventual BDFL pronouncement; > if you see a reason not to make such a change, or if you would prefer > a different procedure, please speak up. Encouragement and support is > welcome, as well :-) Thanks for writing this PEP Martin! > 1. Assign passwords for all current committers for use on svn.python.org. >User names on SF and svn.python.org should be identical, unless some >committer requests a different user name. We've been going with firstname.lastname (with some exceptions -- hi Aahz! :) for the current svn access. Is it better to stay with that convention or to maintain consistency with SF's CVS committer names? Maybe the latter for revision history consistency. > 2. At the beginning of the migration, announce that the repository on >SourceForge closed. We can probably play games with the various CVS hooks to disable all checkins during this time. We might also want to disable the u/i access to CVS at the same time. > 4. Convert the CVS repository into two subversion repositories, >one for distutils and one for Python. Do we also want to split off nondist and encodings? IWBNI the Python source code proper weren't buried too deep in the directory structure. Note that we might want to provide different access permission to different parts of the repository (but I think we can do that even if we don't split those off into separate repos). > As the repository format, fsfs should be used (requires Subversion 1.1). > fsfs has the advantage of being more backup-friendly, as it allows to > backup a repository incrementally, without requiring to run any dump > commands. Definitely +1 on fsfs. Again, thanks Martin! -Barry signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, 2005-07-28 at 16:20, Guido van Rossum wrote: > I hope we're correctly estimating the effort required to manage the > server and the users. Yah, me too! ;) We are building some experience with this though, having moved many of the system files, and all of the web pages, to svn. So far, the management overhead has been almost nil (um, None :). We'll have a bit of ongoing work to add users, but the infrastructure team is also building up some community knowledge about how to do that. > Managing users is especially important -- if a > user is compromised (as has happened in the past for python.org users) > the whole repository is compromised. Now this could happen to SF users > too, but I'm not sure that we know all the tricks in the book to > prevent attacks; SF has been doing this for years and that's an aspect > of SF that I trust (I think I've heard that they have even modified > their SSH server to be stricter). James has a very interesting idea for mitigating this. Presumably , we'll have backups of everything. I'll feel better when we have coverage from maybe 6 admins spanning as many timezones as possible. -Barry signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, 2005-07-28 at 17:58, James Y Knight wrote: > If you use the fsfs storage mechanism for subversion, it is somewhat > simpler to verify that the repository is not compromised. Each commit > is represented as a separate file, and thus old commits are never > modified. Only new files are appended to the directory. If you have a > filesystem that allows "append-only" permissions on a directory, you > can enforce this directly. Additionally, it is possible in your > backup script to verify that only new files were added and nothing > else changed. > > Then at least you know how much you need to examine instead of having > to treat the entire repository as possibly contaminated. Would it buy us any additional piece of mind to checksum the transaction files as they're committed and store those checksums outside the repository? -Barry signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
[...] > Publish the Repositories > [...] > As an option, websvn (available > e.g. from the Debian websvn package) could be provided. Is there any reason that this should be an option, and not just done? For occasional source (particularly C source) lookups, I've found webcvs really useful (especially when on a machine without cvs or ssh). I presume that I'm not alone here. If there are issues with it (stability, security, whatever), then I could understand making it optional, but otherwise I think it would be great if the PEP just included it. =Tony.Meyer ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
> Do we also want to split off nondist and encodings? IWBNI > the Python source code proper weren't buried too deep in the > directory structure. +1 =Tony.Meyer ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On 7/28/05, Barry Warsaw <[EMAIL PROTECTED]> wrote: > On Thu, 2005-07-28 at 16:00, "Martin v. Löwis" wrote: > > I'd like to see the Python source be stored in Subversion instead > > of CVS > > +1 > +1 from me as well; single commit numbers for commits across multiple files will be wonderful. > > , and on python.org instead of sf.net. > > +0 > > I know that SF has promised svn access to projects for a long time now, > but I haven't heard anything from them in a long time. It's listed > under their "Strategic Projects" but the last update to that news item > was back in April. Question: do we wait for SF to make the service > available (after getting more up-to-date status and a realistic > timetable), or do we go straight to svn.python.org? > I say forget SF and we move it. Of course I won't be involved with the migration so me saying this doesn't mean too much. =) >> 1. Assign passwords for all current committers for use on svn.python.org. >>User names on SF and svn.python.org should be identical, unless some >> committer requests a different user name. > >We've been going with firstname.lastname (with some exceptions -- hi >Aahz! :) for the current svn access. Is it better to stay with that >convention or to maintain consistency with SF's CVS committer names? >Maybe the latter for revision history consistency. I say go with the first.last naming. While this might put committer names out of sync, we could keep a mapping of SF names to the new names in developers.txt for easy referencing. But it would be handy to have actual name references since I know I don't always remember who is whom on SF since some people go with nicks that are not based on their name at all. [SNIP] > > 4. Convert the CVS repository into two subversion repositories, > >one for distutils and one for Python. > > Do we also want to split off nondist and encodings? IWBNI the Python > source code proper weren't buried too deep in the directory structure. > Note that we might want to provide different access permission to > different parts of the repository (but I think we can do that even if we > don't split those off into separate repos). > Seems like a reasonable thing. Would make it easier for occasional committers as well as people who check out the code just for generating a patch. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
"Martin v. Löwis" wrote: > Converting the CVS Repository > = > > The Python CVS repository contains two modules: distutils and > python. Keeping them together will produce quite long repository > URLs, so it is more convenient if the Python CVS and the distutils > CVS are converted into two separate repositories. If I understand things correctly, one project/one repo creates a 'hard' barrier for moving code across projects (while retaining history, so done via an svn command). Is the 'long url' really the only argument for this, and is it significant enough? Instead of: https://svn.python.org/python https://svn.python.org/distutils you could have https://svn.python.org/main/python https://svn.python.org/main/distutils or something similar. It's an extra few chars, and it would give a convenient way to branch off pieces of the main code into their own subprojects in the future if needed. For more experimental things, you can always have other repos: https://svn.python.org/someotherrepo/... But maybe the issue of moving code isn't too important, I'm certainly no expert on svn. Cheers, f ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
I theory Subversion should allow you to be more secure. CVS has a very limited concept of security and for the most part you need to rely on SSH. Subversion makes use of Apache as one of its server options. Any authentication method you can use in Apache 2 you can use for Subversion. Once Apache does the authentication, Subversion allows fairly fine grained access control. SSL can be used for encrypting communication. Note that there are sites like Sourceforge that do provide Subversion hosting to open source projects. I don't know enough about them to be able to recommend any. -Chris On Thu, Jul 28, 2005 at 01:20:14PM -0700, Guido van Rossum wrote: > On 7/28/05, "Martin v. L?wis" <[EMAIL PROTECTED]> wrote: > > I'd like to see the Python source be stored in Subversion instead > > of CVS, and on python.org instead of sf.net. To facilitate discussion, > > I have drafted a PEP describing the rationale for doing so, and > > the technical procedure to be performed. > > In principle I'm +1 on this (both aspects). I don't know enough to > understand all the subtleties. > > I hope we're correctly estimating the effort required to manage the > server and the users. Managing users is especially important -- if a > user is compromised (as has happened in the past for python.org users) > the whole repository is compromised. Now this could happen to SF users > too, but I'm not sure that we know all the tricks in the book to > prevent attacks; SF has been doing this for years and that's an aspect > of SF that I trust (I think I've heard that they have even modified > their SSH server to be stricter). > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/lambacck%40computer.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Martin v. Löwis wrote: >Currently, access to Subversion on svn.python.org uses WebDAV over >https, using basic authentication. For this to work, authorized >users need to provide a password. This mechanism should be used, >atleast initially, for the Python CVS as well, since various committers >also have a username/password pair for the www SVN repository already. > >Alternatives to password-based access include: >- Subversion over SSH, using SSH key pairs. This would require > to give committers accounts on the machine, which currently is > ruled out by the administration policy of svn.python.org. >- Subversion over WebDAV, using SSL client certificates. This > would work, but would require to administrate a certificate > authority. > > I'm definitely positive to a migration to Subversion, but I'd be really concerned about using plain text authentication mechanisms. SSH obviously is much prefered, and clearly there are ways to limit the "accounts" on the svn.python.org, many other projects does this already for cvs. E.g thor (17:11) 350/0 $ ssh -l 'username' cvs.mozilla.org To use anonymous CVS install the latest version of CVS on your local machine. Then set your CVSROOT environment variable to the following value: [EMAIL PROTECTED]:/cvsroot Connection to cvs.mozilla.org closed. We should have enough man powers to come up with some secure solution here :). -- Leif ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
[Martin v. Löwis] > I'd like to see the Python source be stored in Subversion instead > of CVS, and on python.org instead of sf.net. To facilitate discussion, > I have drafted a PEP describing the rationale for doing so, and > the technical procedure to be performed. > > This is for discussion on python-dev and eventual BDFL pronouncement; > if you see a reason not to make such a change, or if you would prefer > a different procedure, please speak up. Encouragement and support is > welcome, as well :-) Encouragement and support on SVN, undecided on moving to python.org (don't know when SF intends to support SVN, don't have a feel for the state of-- or propsects for ongoing --python.org volunteerism). > ... > The conversion should be done using cvs2svn utility, available e.g. > in the cvs2svn Debian package. The command for converting the Python > repository is > > cvs2svn -q --encoding=latin1 --force-branch=cnri-16-start > --force-branch=descr-branch --force-branch=release152p1-patches > --force-tag=r16b1 --fs-type=fsfs -s py.svn.new python/python > > The command to convert the distutils repository is > > cvs2svn -q --encoding=latin1 --fs-type=fsfs -s dist.svn.new python/distutils > > Sample results of this conversion are available at > > http://www.dcl.hpi.uni-potsdam.de/python/ > http://www.dcl.hpi.uni-potsdam.de/distutils/ I'm sending this to Jim Fulton because he did the conversion of Zope Corp's code base to SVN. Unfortunately, Jim will soon be out of touch for several weeks. Jim, do you have time to summarize the high bits of the problems you hit? IIRC, you didn't find any conversion script at the time capable of doing the whole job as-is. If that's wrong, it would be good to know that too. Other than that, I'd just like to see an explicit mention in the PEP of a plan to preserve the pre-conversion CVS tarball forever. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thursday 28 July 2005 07:21 pm, Tim Peters wrote: > [Martin v. Löwis] > > > The conversion should be done using cvs2svn utility, available e.g. > > in the cvs2svn Debian package. The command for converting the Python > > repository is > > Sample results of this conversion are available at > > > > http://www.dcl.hpi.uni-potsdam.de/python/ > > http://www.dcl.hpi.uni-potsdam.de/distutils/ > > I'm sending this to Jim Fulton because he did the conversion of Zope > Corp's code base to SVN. Unfortunately, Jim will soon be out of touch > for several weeks. Jim, do you have time to summarize the high bits > of the problems you hit? IIRC, you didn't find any conversion script > at the time capable of doing the whole job as-is. If that's wrong, it > would be good to know that too. The conversion script isn't perfect and does fail on complex CVS arrangements or where there is extensive history to migate. However it appears above that Martin has already tried the script out, with success. BTW, re SSH access on python.org, using Apache's SSL support re https would provide as good of security without the risk of giving out shell accounts. SSL would encrypt the link and require a password or permit cert auth instead, same as SSH. Cert admin needn't be hard if only a single server cert is used, with client passwords, instead of client certs. -Jeff ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, 2005-07-28 at 20:15, Leif Hedstrom wrote: > I'm definitely positive to a migration to Subversion, but I'd be really > concerned about using plain text authentication mechanisms. We won't use plain text, but we may (or, we currently do) use basic auth over ssl. The security then is in the passwords, so we have to make sure they're generated securely. > We should have enough man powers to come up with some secure solution > here :). Volunteers (i.e. those with actual free time to give on implementing any solution) are definitely welcome. -Barry signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Jul 28, 2005, at 3:19 PM, Barry Warsaw wrote: > On Thu, 2005-07-28 at 20:15, Leif Hedstrom wrote: > > >> I'm definitely positive to a migration to Subversion, but I'd be >> really >> concerned about using plain text authentication mechanisms. >> > > We won't use plain text, but we may (or, we currently do) use basic > auth > over ssl. The security then is in the passwords, so we have to make > sure they're generated securely. Users can generate their hashed password with the htpasswd command locally, and send those to the server. That's what most subversion hosts seem to prefer. -bob ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Tim Peters wrote: > [Martin v. Löwis] >> The conversion should be done using cvs2svn utility, available e.g. >> in the cvs2svn Debian package. The command for converting the Python >> repository is [...] > I'm sending this to Jim Fulton because he did the conversion of Zope > Corp's code base to SVN. Unfortunately, Jim will soon be out of touch > for several weeks. Jim, do you have time to summarize the high bits > of the problems you hit? IIRC, you didn't find any conversion script > at the time capable of doing the whole job as-is. If that's wrong, it > would be good to know that too. If you hit any snags, you may be interested in contacting the admin for scipy.org. The scipy CVS repo choked cvs2svn pretty badly a while ago, but recent efforts eventually prevailed. This afternoon an email arrived from him: Original Message Subject: [SciPy-dev] SciPy CVS to Subversion migration Date: Thu, 28 Jul 2005 20:02:59 -0500 From: Joe Cooper <[EMAIL PROTECTED]> Reply-To: SciPy Developers List <[EMAIL PROTECTED]> To: SciPy Dev List <[EMAIL PROTECTED]> Hi all, The issues with converting our CVS repository to Subversion have been resolved, and so I'd like to make the switch tomorrow (Friday) afternoon. [...] I know Joe was in contact with the SVN devs to work on this, so perhaps he's using a patched version of cvs2svn, I simply don't know. But I mention it in case it proves useful to the python.org conversion. cheers, f ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
[Jeff Rush] > The conversion script isn't perfect and does fail on complex CVS > arrangements or where there is extensive history to migate. However it > appears above that Martin has already tried the script out, with success. I'd still like to hear from Jim, as I don't believe all serious problems were identified by eyeball at once. That said, the Python project has made relatively very little use of complex (for CVS) concepts like branching, but in Zopeland it wasn't unusual, over long stretches, for someone to make a new branch every day. Ah, before I forget, "single repository" has worked very well for Zope (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... projects): http://svn.zope.org/ Long URLs don't really get in the way in practice (rarely a need to type one after initial checkout; even "svn switch" is usually just a tail-end cmdline edit starting from a copy+paste of "svn info" output). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, 2005-07-28 at 22:14, Tim Peters wrote: > Ah, before I forget, "single repository" has worked very well for Zope > (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... > projects): > > http://svn.zope.org/ > > Long URLs don't really get in the way in practice (rarely a need to > type one after initial checkout; even "svn switch" is usually just a > tail-end cmdline edit starting from a copy+paste of "svn info" > output). It depends. In my use of svn, I do a lot of cross-branch merging and repo-side tagging. Those are done with urls and in those cases, long urls can suck. But we may not do a ton of that with the Python project, and besides it might not be important enough to split the directories. -Barry signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
[Tim] >> Ah, before I forget, "single repository" has worked very well for Zope >> (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... >> projects): >> >> http://svn.zope.org/ >> >> Long URLs don't really get in the way in practice (rarely a need to >> type one after initial checkout; even "svn switch" is usually just a >> tail-end cmdline edit starting from a copy+paste of "svn info" >> output). [Barry] > It depends. In my use of svn, I do a lot of cross-branch merging and > repo-side tagging. Yup, me too -- between the two of us, we don't have enough fingers to count how many trunks, branches, and tags of ZODB and Zope I have to fiddle with. > Those are done with urls and in those cases, long urls can suck. They're all still copy, paste, tail-edit for me, and-- indeed! --having them all in the same repository is what makes just-the-tail editing possible. Merges I do from the cmdline, but repo-side tagging I do with the TortoiseSVN GUI, and the latter gives easy-to-copy/paste/edit URL fields. So switch to Windows for that part ;-) > But we may not do a ton of that with the Python project, > and besides it might not be important enough to split the directories. Ya, in Python we make a branch about once per release, + once per 5 years when Jeremy underestimates how long it will take to finish a crusade . ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, Jul 28, 2005, Barry Warsaw wrote: > On Thu, 2005-07-28 at 22:14, Tim Peters wrote: >> >> Ah, before I forget, "single repository" has worked very well for Zope >> (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... >> projects): >> >> http://svn.zope.org/ >> >> Long URLs don't really get in the way in practice (rarely a need to >> type one after initial checkout; even "svn switch" is usually just a >> tail-end cmdline edit starting from a copy+paste of "svn info" >> output). > > It depends. In my use of svn, I do a lot of cross-branch merging and > repo-side tagging. Those are done with urls and in those cases, long > urls can suck. But we may not do a ton of that with the Python project, > and besides it might not be important enough to split the directories. Why can't you write a Python script to generate the URLs? <0.3 wink> -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ The way to build large Python applications is to componentize and loosely-couple the hell out of everything. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thu, 2005-07-28 at 22:59, Tim Peters wrote: > Yup, me too -- between the two of us, we don't have enough fingers to > count how many trunks, branches, and tags of ZODB and Zope I have to > fiddle with. I have a small inkling of your pain. > They're all still copy, paste, tail-edit for me, and-- indeed! > --having them all in the same repository is what makes just-the-tail > editing possible. Good point! > Ya, in Python we make a branch about once per release, + once per 5 > years when Jeremy underestimates how long it will take to finish a > crusade . So are you saying that moving to svn will let us do more long lived branches? Yay! -B signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Barry Warsaw wrote: > I know that SF has promised svn access to projects for a long time now, > but I haven't heard anything from them in a long time. It's listed > under their "Strategic Projects" but the last update to that news item > was back in April. Question: do we wait for SF to make the service > available (after getting more up-to-date status and a realistic > timetable), or do we go straight to svn.python.org? My proposal is to go straight to svn.python.org, for two reasons: - It might well be 2006 before they update the status and provide a realistic timetable. It's not that I lost faith into SF, but they seem to be continually overworked (as anybody), so anything that is not *really* essential to the operation of the service (such as support for subversion) has to wait "until we have time" -- which means it has to wait forever. Remember the plan to provide shell access to the project admins on the CVS server? - They currently have a separate anonymous access for CVS which is behind the real CVS, for load sharing reasons. They also had severe performance issues in the past. It might be that we also get performance problems, but we only need to support a single project (or two if you count pydotorg) on Subversion, not thousands of them. So our load evolution is much more predictable. >>1. Assign passwords for all current committers for use on svn.python.org. >> User names on SF and svn.python.org should be identical, unless some >> committer requests a different user name. > > > We've been going with firstname.lastname (with some exceptions -- hi > Aahz! :) for the current svn access. Is it better to stay with that > convention or to maintain consistency with SF's CVS committer names? > Maybe the latter for revision history consistency. It's also a convenience issue, and has social aspects. For example, firstname.lastname does not work quite well for me, either (martin.v.loewis or martin.von.loewis would work better; likewise guido.van.rossum?), other users prefer not to use their "true" real name (e.g. tim_one vs. tim.peters). Another issue is password assignment - currently, users cannot choose their passwords on svn.python.org, right? > We can probably play games with the various CVS hooks to disable all > checkins during this time. We might also want to disable the u/i access > to CVS at the same time. That should be tested in advance, of course. >>4. Convert the CVS repository into two subversion repositories, >> one for distutils and one for Python. > > > Do we also want to split off nondist and encodings? Well, encodings is empty right now, so that might be a good idea. Technically, I would just move the files in the CVS repository before conversion. As for nondist: how precisely would you do that? Make it a separate repository? If so, where? I could envision a "/projects" repository, that has the current nondist. > Note that we might want to provide different access permission to > different parts of the repository (but I think we can do that even if we > don't split those off into separate repos). I don't know how cvs2svn supports it, but one option would be to revert the trunk/ structure in /projects, so you get http://www.python.org/projects/peps/trunk http://www.python.org/projects/peps/tags http://www.python.org/projects/peps/branches http://www.python.org/projects/sandbox/trunk http://www.python.org/projects/sandbox/tags http://www.python.org/projects/sandbox/branches Then you could give different access to peps and to sandbox. Perhaps there isn't even a need for tags/branches on PEPs? Or we put everything in nondist into /projects, and then put tags + branches as siblings (so only people with write access to tags could create them)? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On Thursday 28 July 2005 20:07, Fernando Perez wrote: > or something similar. It's an extra few chars, and it would give a > convenient way to branch off pieces of the main code into their own > subprojects in the future if needed. More interestingly, keeping it in a single repository makes it easier to merge projects, or parts of projects, together, without losing the history. This would be useful when developing packages that may be considered for the standard library, but which also need to continue separate releases to support older versions of Python. We've found it very handy to keep multiple projects in a single repository for zope.org. -Fred -- Fred L. Drake, Jr. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Guido van Rossum wrote: > I hope we're correctly estimating the effort required to manage the > server and the users. Managing users is especially important -- if a > user is compromised (as has happened in the past for python.org users) > the whole repository is compromised. Now this could happen to SF users > too, but I'm not sure that we know all the tricks in the book to > prevent attacks; SF has been doing this for years and that's an aspect > of SF that I trust (I think I've heard that they have even modified > their SSH server to be stricter). There are three issues that I see: - management of access control. This is actually something that we do on SF as well; if the pydotorg admins get overloaded, I'm sure the current projects/python admins would be willing to help there. - assignment of passwords. This I don't like about the current pydotorg setup - there should be a way to chose your own password; perhaps without involving an administrator. I could imagine a web form for password change, and administrator interaction in case of a lost password. - compromised passwords. The only tricky question then is: was the repository altered? Fortunately, for Subversion, there should be an easy way to tell: in fsfs, files never change (only new files are added). So we could generate md5sums of all files in the repository, and download these to an offsite place. If the md5sum of an immutable file changes, we were compromised (there are, of course, a few files that do change regularly). Of course, we also need regular backups of the entire data so we can restore them if they got compromised. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Weekly Python Patch/Bug Summary
Patch / Bug Summary ___ Patches : 357 open ( +7) / 2885 closed ( +3) / 3242 total (+10) Bugs: 898 open ( +9) / 5144 closed ( +3) / 6042 total (+12) RFE : 191 open ( +2) / 178 closed ( +0) / 369 total ( +2) New / Reopened Patches __ shutil.copytree() quits too soon after an error. (2005-07-21) http://python.org/sf/1242454 opened by Dale Wilson bltinmodule.c whitespace normalization (2005-07-21) http://python.org/sf/1242579 opened by Ruslan Spivak repair typos (2005-07-22) CLOSED http://python.org/sf/1243081 opened by George Yoshida Faster output if message already has a boundary (2005-07-23) http://python.org/sf/1243654 opened by L. Peter Deutsch httplib gzip support (2005-07-23) http://python.org/sf/1243678 opened by Moonz Big speedup in email message parsing (2005-07-23) http://python.org/sf/1243730 opened by L. Peter Deutsch Patch for (Doc) #1243553 (2005-07-24) http://python.org/sf/1243910 opened by Peter van Kampen expat binding for XML_ParserReset (Bug #1208730) (2005-07-24) http://python.org/sf/1244208 opened by John Eikenberry Enable os.startfile and webbrowser.WindowsDefault on Cygwin (2005-07-25) http://python.org/sf/1244861 opened by Michael Hoffman hide tests from TestProgram (2005-07-25) http://python.org/sf/1244929 opened by Eric Huss Encoding alias "unicode-1-1-utf-7" (2005-07-26) http://python.org/sf/1245379 opened by Oren Tirosh weaklist (2001-11-30) http://python.org/sf/487738 reopened by fdrake Patches Closed __ optparse documentation bug fixes (2005-05-18) http://python.org/sf/1204347 closed by gward repair typos (2005-07-22) http://python.org/sf/1243081 closed by birkenfeld fileinput.py fixed clobbering behavior and added encodings (2004-10-15) http://python.org/sf/1048075 closed by cconnett Allow use of embedded Tcl without requiring Tk (2004-01-02) http://python.org/sf/869468 closed by gvanrossum New / Reopened Bugs ___ list(obj) can swallow KeyboardInterrupt (2005-07-22) http://python.org/sf/1242657 opened by Steve Alexander Incorrect documentation of re.UNICODE (2005-07-22) http://python.org/sf/1243192 opened by nhaldimann Misuse of "it's"(2005-07-22) CLOSED http://python.org/sf/1243288 opened by Joanne Bogart pydoc on cgi.escape lacks info that are in www docs (2005-07-23) http://python.org/sf/1243553 opened by Nikos Kouremenos Python function/method/constant names as HTML-tag IDs (2005-07-24) http://python.org/sf/1243945 opened by Chad Miller 2.4.1 build fails on OpenBSD 3.7 (2005-07-25) http://python.org/sf/1244610 opened by L. Peter Deutsch logging module needs to that it changed in 2.4 (2005-07-25) http://python.org/sf/1244683 opened by Alan Segfault in Python interpreter 2.3.5 (2005-07-25) http://python.org/sf/1244864 opened by Evil Mr Henry Time module is missing inverse of gmtime() (2005-07-26) http://python.org/sf/1245224 opened by Jeremy Fincher log() on a big number fails on powerpc64 (2005-07-26) http://python.org/sf/1245381 opened by Jiri Dluhos Segmentation fault when importing expat from xml.parser (2005-07-28) http://python.org/sf/1246405 opened by Jean-Pierre line numbers off by 1 (2005-07-27) http://python.org/sf/1246473 opened by Brett Cannon failure to build RPM on rhel 3 (2005-07-28) http://python.org/sf/1246900 opened by Patrick Wagstrom Bugs Closed ___ Misuse of "it's"(2005-07-22) http://python.org/sf/1243288 closed by birkenfeld ioctl has problem with -ive request codes (2005-07-01) http://python.org/sf/1231069 closed by mwh ioctl has problems on 64 bit machines (2005-01-31) http://python.org/sf/1112949 closed by birkenfeld ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Tony Meyer wrote: > Is there any reason that this should be an option, and not just done? Certainly: it's administrator load, which in turn is volunteer time. > For > occasional source (particularly C source) lookups, I've found webcvs really > useful (especially when on a machine without cvs or ssh). I presume that > I'm not alone here. Of course, so if there is a volunteer, to actually set it up, it should be done. For the PEP itself, somebody would have to contribute precise instructions how to setup websvn (what URLs, how does the Apache configuration read, etc). > If there are issues with it (stability, security, whatever), then I could > understand making it optional, but otherwise I think it would be great if > the PEP just included it. There are security issues in the setup if you want to have non-public repositories (which then shouldn't be public through websvn, either). The pydotorg repository itself is non-public, but I don't know whether this would be an issue. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
On 7/29/05, Fred L. Drake, Jr. <[EMAIL PROTECTED]> wrote: > On Thursday 28 July 2005 20:07, Fernando Perez wrote: > > or something similar. It's an extra few chars, and it would give a > > convenient way to branch off pieces of the main code into their own > > subprojects in the future if needed. > > More interestingly, keeping it in a single repository makes it easier to merge > projects, or parts of projects, together, without losing the history. This > would be useful when developing packages that may be considered for the > standard library, but which also need to continue separate releases to > support older versions of Python. We've found it very handy to keep multiple > projects in a single repository for zope.org. I thought I would chime in on this with some more experience. We use SVN (migated from CVS about 2 years ago) for everything, and we keep it all in one repository (even though there's several "products") for exactly this reason. The major "downside" that I, and some others, find is changeset explosion. When you start getting into 5 digit changeset numbers it can become a bit unwieldy to remember to type them all, but otherwise it works well. We also moved from BerkeleyDB-based to fsfs recently, and it seems to have fixed some intermittent problems we had had in the past. Another thing to look at would be Trac, which we are in the process of moving to from the horrendous nightmare of Bugzilla. It's integration with SVN as well as Wiki is quite amazing. Chris -- | Christopher Petrilli | [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Fred L. Drake, Jr. wrote: > More interestingly, keeping it in a single repository makes it easier to > merge > projects, or parts of projects, together, without losing the history. This > would be useful when developing packages that may be considered for the > standard library, but which also need to continue separate releases to > support older versions of Python. We've found it very handy to keep multiple > projects in a single repository for zope.org. So do you use project/trunk or trunk/project? If the former, I would need to get instructions on how to do the conversion from CVS. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Barry Warsaw wrote: > We won't use plain text, but we may (or, we currently do) use basic auth > over ssl. The security then is in the passwords, so we have to make > sure they're generated securely. That (sort of) *is* plain text passwords. Somebody who took over svn.python.org can get the password. In public-key or digest authentication, this won't be possible. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
Tim Peters wrote: > Ah, before I forget, "single repository" has worked very well for Zope > (which includes top-level Zope2, Zope3, ZODB, ZConfig, zdaemon, ... > projects): > > http://svn.zope.org/ > > Long URLs don't really get in the way in practice (rarely a need to > type one after initial checkout; even "svn switch" is usually just a > tail-end cmdline edit starting from a copy+paste of "svn info" > output). That would indeed give conversion problems: cvs2svn automatically generates tags/trunk/branches in the root, with the original CVS modules below; there is a single space for tags and branches (as is in the original CVS repository). I would see two possible conversion procedures: 1. convert the modules one-by-one, adding to the same repository. I.e. python would get all the small revision numbers, then peps, then sandbox, and so on. Cross-module tags would not be supported (but likely don't exist in the first place), and the revision number would not increase in historical order. 2. convert the entire CVS, then rearrange things through svn move. This would be tedious to do (atleast for tags/branches), and it would cause all files to be renamed right from the scratch (some svn clients fail to display history past moves). So for all who would prefer to see a single repository, could somebody please 1. state how precisely the repository should be organized (with exact URLs on svn.python.org - eg. which of the non-dist directories becomes toplevel, which ones get their own tags/branches/trunk, etc). 2. state how the conversion should be performed Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion
"Martin v. Löwis" wrote: > Fred L. Drake, Jr. wrote: >> More interestingly, keeping it in a single repository makes it easier to >> merge >> projects, or parts of projects, together, without losing the history. This >> would be useful when developing packages that may be considered for the >> standard library, but which also need to continue separate releases to >> support older versions of Python. We've found it very handy to keep multiple >> projects in a single repository for zope.org. > > So do you use project/trunk or trunk/project? If the former, I would > need to get instructions on how to do the conversion from CVS. For ipython, which recently went through cvs2svn, I found that moving over to a project/trunk structure was a few minutes worth of work. Since svn has moving commands, it was just a matter of making the extra project/ directory and moving things one level down the hierarchy. Even if cvs2svn doesn't quite create things the way you want them in the long run, svn is flexible enough that a few manual tweaks should be quite easy to perform. Cheers, f ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
