[Python-Dev] Community buildbots and Python release quality metrics
Today on planetpython.org, Doug Hellman announced the June issue of Python magazine. The cover story this month is about Pybots, "the fantastic automation system that has been put in place to make sure new releases of Python software are as robust and stable as possible". Last week, there was a "beta" release of Python which, according to the community buildbots, cannot run any existing python software. Normally I'd be complaining here about Twisted, but in fact Twisted is doing relatively well right now; only 80 failing tests. Django apparently cannot even be imported. The community buildbots have been in a broken state for months now[1]. I've been restraining myself from whinging about this, but now that it's getting close to release, it's time to get these into shape, or it's time to get rid of them. There are a few separate issues here. One is, to what extent should Python actually be compatible between releases? There's not much point in saying "this release is compatible with python 2.5" if all python 2.5 programs need to be modified in order to run on python 2.6. Of course, it's quite likely that there are some 3rd-party Python programs that continue to work, but python.org provides links to 2 projects that don't and none that do. Another way to phrase this question is, "whose responsibility is it to make Python 2.5 programs run on Python 2.6"? Or, "what happens when the core team finds out that a change they have made has broken some python software 'in the wild'"? Here are a couple of ways to answer these questions: 1) It's the python core team's responsibility. There should be Python buildbots for lots of different projects and they should all be green before the code can be considered [one of: allowed in trunk, alpha, beta, RC]. 2) It's the python core team's responsibility as long as the major projects are using public APIs. The code should not be considered [alpha, beta, RC] if there are any known breakages in public APIs. 3) It's only the python core team's responsibility to notify projects of incompatible changes of which they are aware, which may occur in public or private APIs. Major projects with breakages will be given a grace period before a [beta, RC, final] to release a forward-compatible version. 4) The status quo: every new Python release can (and will, at least speaking in terms of 2.6) break lots of popular software, with no warning aside from the beta period. There are no guarantees. For obvious reasons, I'd prefer #1, but any of the above is better than #4. I don't think #4 is an intentional choice, either; I think it's the result of there being no clear consequence for community buildbots failing. Or, for that matter, of core buildbots failing. Investigating the history is tricky (I see a bunch of emails from Barry here, but I'm not sure which is the final state) but I believe the beta went out with a bunch of buildbots offline or failing. A related but independent issue is: should the community buildbots continue to exist? An automated test is only as good as the consequences of its failure. As it stands, the community buildbots seem to be nothing but an embarrassment. I don't mean this to slam Grig, or the Python team - I mean, literally, if there is no consequence of their failure then the only use I can see for the data the buildbots currently provide (i.e. months and months of failures) is to embarrass the Python core developers. python.org's purpose should not be to provide negative press for Python ;), so if this continues to be the case, they probably shouldn't be linked. This isn't just an issue with changes to Python breaking stuff; the pybots' configuration is apparently not well maintained, since the buildbots which say "2.5" aren't passing either, and that's not a problem with the code, it's just that the slaves are actually running against [EMAIL PROTECTED] Ultimately these tools only exist to ensure the quality of Python releases. The really critical question here is, what does it mean to have a Python release that is high-quality? There are some obvious things: it shouldn't crash, it should conform to its own documentation. Personally I think "it passes all of its own tests" and "it runs existing Python code" are important metrics too, but the most important thing I'm trying to get across here is that there should be a clear understanding of which goals the release / QA / buildbot / test process is trying to accomplish. For me, personally, it really needs to be clear when I can say "you guys screwed up and broke stuff", and when I just have to suck it up and deal with the consequences of a new version of Python in Twisted. It's definitely bad for all of us if the result is that new releases of Python just break everything. Users don't care where the responsibility lies, they just know that stuff doesn't work, so we should decide on a process which allows
Re: [Python-Dev] Community buildbots and Python release quality metrics
Too verbose, Glyph. :-) It needs to be decided case-by-case. The beta is called beta because, well, it may break stuff and we may want to fix it. But there are also likely many frameworks that use undefined behavior. I'm not particularly impressed by statistics like "all tests are red" -- this may all be caused by a single issue. For example, I'd much rather read an explanation about *why* Django cannot even be imported than a blanket complaint that this is a disgrace. So why is it? --Guido On Thu, Jun 26, 2008 at 8:18 AM, <[EMAIL PROTECTED]> wrote: > Today on planetpython.org, Doug Hellman announced the June issue of Python > magazine. The cover story this month is about Pybots, "the fantastic > automation system that has been put in place to make sure new releases of > Python software are as robust and stable as possible". > > Last week, there was a "beta" release of Python which, according to the > community buildbots, cannot run any existing python software. Normally I'd > be complaining here about Twisted, but in fact Twisted is doing relatively > well right now; only 80 failing tests. Django apparently cannot even be > imported. > > The community buildbots have been in a broken state for months now[1]. I've > been restraining myself from whinging about this, but now that it's getting > close to release, it's time to get these into shape, or it's time to get rid > of them. > > There are a few separate issues here. One is, to what extent should Python > actually be compatible between releases? There's not much point in saying > "this release is compatible with python 2.5" if all python 2.5 programs need > to be modified in order to run on python 2.6. Of course, it's quite likely > that there are some 3rd-party Python programs that continue to work, but > python.org provides links to 2 projects that don't and none that do. > > Another way to phrase this question is, "whose responsibility is it to make > Python 2.5 programs run on Python 2.6"? Or, "what happens when the core > team finds out that a change they have made has broken some python software > 'in the wild'"? > > Here are a couple of ways to answer these questions: > > 1) It's the python core team's responsibility. There should be Python > buildbots for lots of different projects and they should all be green before > the code can be considered [one of: allowed in trunk, alpha, beta, RC]. > 2) It's the python core team's responsibility as long as the major projects > are using public APIs. The code should not be considered [alpha, beta, RC] > if there are any known breakages in public APIs. > 3) It's only the python core team's responsibility to notify projects of > incompatible changes of which they are aware, which may occur in public or > private APIs. Major projects with breakages will be given a grace period > before a [beta, RC, final] to release a forward-compatible version. > 4) The status quo: every new Python release can (and will, at least > speaking in terms of 2.6) break lots of popular software, with no warning > aside from the beta period. There are no guarantees. > > For obvious reasons, I'd prefer #1, but any of the above is better than #4. > I don't think #4 is an intentional choice, either; I think it's the result > of there being no clear consequence for community buildbots failing. Or, > for that matter, of core buildbots failing. Investigating the history is > tricky (I see a bunch of emails from Barry here, but I'm not sure which is > the final state) but I believe the beta went out with a bunch of buildbots > offline or failing. > > A related but independent issue is: should the community buildbots continue > to exist? An automated test is only as good as the consequences of its > failure. As it stands, the community buildbots seem to be nothing but an > embarrassment. I don't mean this to slam Grig, or the Python team - I mean, > literally, if there is no consequence of their failure then the only use I > can see for the data the buildbots currently provide (i.e. months and months > of failures) is to embarrass the Python core developers. python.org's > purpose should not be to provide negative press for Python ;), so if this > continues to be the case, they probably shouldn't be linked. This isn't > just an issue with changes to Python breaking stuff; the pybots' > configuration is apparently not well maintained, since the buildbots which > say "2.5" aren't passing either, and that's not a problem with the code, > it's just that the slaves are actually running against [EMAIL PROTECTED] > > Ultimately these tools only exist to ensure the quality of Python releases. > The really critical question here is, what does it mean to have a Python > release that is high-quality? There are some obvious things: it shouldn't > crash, it should conform to its own documentation. Personally I think "it > passes all of its own tests" and "it runs existing Python code" are > important metrics too, but the mos
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] wrote: Today on planetpython.org, Doug Hellman announced the June issue of Python magazine. The cover story this month is about Pybots, "the fantastic automation system that has been put in place to make sure new releases of Python software are as robust and stable as possible". Last week, there was a "beta" release of Python which, according to the community buildbots, cannot run any existing python software. Normally I'd be complaining here about Twisted, but in fact Twisted is doing relatively well right now; only 80 failing tests. Django apparently cannot even be imported. beta 1 has some trouble running *our* test suite - I'd be fairly surprised if the community buildbots were in significantly better shape. The community buildbots have been in a broken state for months now[1]. Continuously running community buildbots on the maintenance trees makes sense, since those trees should always be in a releasable state. For the trunk, they're really only interesting when the Python core buildbots are reporting all green, but some of the community buildbots are reporting red. One of the problems is what the term "beta" means to different groups - for us, this first beta was really about saying "zero new features from here on, focus on making what we have now work properly". The relatively late landing of a couple of major PEPs (371, 3108) also didn't do any favours for trunk stability. If the community buildbots aren't largely green by the time beta 2 comes out, that's when I'll agree we have a problem - they should definitely be green by the time first release candidate comes out. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 03:33 pm, [EMAIL PROTECTED] wrote: Too verbose, Glyph. :-) Mea culpa. "Glyph" might be a less appropriate moniker than "Altogether too many glyphs". It needs to be decided case-by-case. If certain tests are to be ignored on a case-by-case basis, why not record that decision by disabling the test in the code? Otherwise, the decision inevitably gets boiled down to "it's okay to release the code with a bunch of tests failing, but I don't know which ones". As it was on Twisted when we used to make case-by-case decisions about failures, and as it is here now. The beta is called beta because, well, it may break stuff and we may want to fix it. That's also why the alpha is called an alpha. My informal understanding is that a beta should have no (or at least very few) known issues, with a medium level of confidence that no further ones will be found. An RC would have absolutely no known issues with a fairly high level of confidence that no further ones will be found. This would include the community buildbots basically working for the most part; I would not be surprised if there were a few tests that still had issues. But clearly the reality does not meet my informal expectations, so it would be nice to have something written down to check against. Still, I'm bringing this up now because it _is_ a beta, and I think it will be too late to talk about dealing with persistent test failures after the RCs start coming out. (Of course, I'm just being sneaky. I don't actually care if it's clearly documented, I just care that I stop having problems with incompatibility. But I believe having it clearly spelled out would actually prevent a lot of the problems that I've been having, since I don't think anyone would *intentionally* select a policy where every release breaks at least one major dependent project.) I'm not particularly impressed by statistics like "all tests are red" -- this may all be caused by a single issue. The issue, for me, is not specifically that tests are red. It's that there's no clear policy about what to do about that. Can a release go out with some of the tests being red? If so, what are the extenuating circumstances? Does this have to be fixed? If not, why not? Why are we talking about this now? Shouldn't the change which caused Django to become unimportable have been examined at the time, rather than months later? (In other words, if it *is* a single issue, great, it's easy to fix: revert that single issue.) If not, then shouldn't someone in Django-land have been notified so they could cope with the change? Sorry that there are so many questions here; if I had fewer, I'd use fewer words to ask them. For example, I'd much rather read an explanation about *why* Django cannot even be imported than a blanket complaint that this is a disgrace. So why is it? I don't know. JP is already addressing the issues affecting Twisted in another thread (incompatible changes in the private-but-necessary-to- get-any-testing-done API of the warnings module). But I really think that whoever made the change which broke it should be the one investigating it, not me. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 03:42 pm, [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: beta 1 has some trouble running *our* test suite - I'd be fairly surprised if the community buildbots were in significantly better shape. That's another problem, yes :) The community buildbots have been in a broken state for months now[1]. If the community buildbots aren't largely green by the time beta 2 comes out, that's when I'll agree we have a problem - they should definitely be green by the time first release candidate comes out. I have mixed feelings about this: on the one hand, if this were a matter of public record, I would have known that this was too early to start complaining, and we could have saved everyone a lot of time reading these messages. That would be good; I would prefer to just let everyone work without bothering about process. On the other hand, it's much easier to deal with test failures as they arise than in one giant chunk of work at the end of the development process. I spoke to a few core developers at PyCon who thought that the buildbots should always be green and any change that broke them should be reverted. They may remain nameless unless they wish to raise their hands :-) but that's definitely how I feel about it. Continuously running community buildbots on the maintenance trees makes sense, since those trees should always be in a releasable state. For the trunk, they're really only interesting when the Python core buildbots are reporting all green, but some of the community buildbots are reporting red. (... which should be all the time ...) One of the problems is what the term "beta" means to different groups - for us, this first beta was really about saying "zero new features from here on, focus on making what we have now work properly". The relatively late landing of a couple of major PEPs (371, 3108) also didn't do any favours for trunk stability. A big part of why I wrote this message is that I wanted a clear understanding of the relationship between the definition of "alpha", "beta" and "RC" and the state of various buildbots. If that relationship exists already, just linking to it from http://python.org/download/releases/2.6/ would be good. By the way, that page still says these are "alpha" releases. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: > an explanation about *why* Django cannot even be imported than a > blanket complaint that this is a disgrace. So why is it? > File "/home/ralf/pediapress/appserver/django/db/models/options.py", line 198, in _many_to_many self._fill_m2m_cache() File "/home/ralf/pediapress/appserver/django/db/models/options.py", line 221, in _fill_m2m_cache cache[field] = None File "/home/ralf/pediapress/appserver/django/utils/datastructures.py", line 75, in __setitem__ super(SortedDict, self).__setitem__(key, value) TypeError: unhashable type: 'ManyToManyField' same for sqlalchemy: egg/sqlalchemy/schema.py", line 113, in __call__ return type.__call__(self, name, metadata, *args, **kwargs) File "/home/ralf/py26/lib/python2.6/site-packages/SQLAlchemy-0.5.0beta1-py2.6.egg/sqlalchemy/schema.py", line 207, in __init__ self.primary_key = PrimaryKeyConstraint() File "/home/ralf/py26/lib/python2.6/site-packages/SQLAlchemy-0.5.0beta1-py2.6.egg/sqlalchemy/schema.py", line 294, in _set_primary_key self.constraints.add(pk) TypeError: unhashable type: 'PrimaryKeyConstraint' and already discussed: http://mail.python.org/pipermail/python-dev/2008-April/078421.html - Ralf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
Ralf Schmitt wrote: TypeError: unhashable type: 'ManyToManyField' TypeError: unhashable type: 'PrimaryKeyConstraint' and already discussed: http://mail.python.org/pipermail/python-dev/2008-April/078421.html Following the discussion to it's conclusions leads one to a tracker issue [1] that was supposed to be a blocker for the beta but (I assume) was forgotten about. Clearly this should be promoted to being a blocker again and some decision made. [1] http://bugs.python.org/issue2235 -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 04:42 pm, [EMAIL PROTECTED] wrote: On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: an explanation about *why* Django cannot even be imported than a blanket complaint that this is a disgrace. So why is it? and already discussed: http://mail.python.org/pipermail/python-dev/2008-April/078421.html Following that trail of breadcrumbs, I ended up here: http://bugs.python.org/issue2235 with a message from some guy named "Barry Warsaw" (anyone know him?) that says: """ Guido, can you comment on Amaury's latest patch? I'm going to bump this back to critical so as not to hold up 2.6 alpha, but it should be marked as a release blocker for the first beta. """ I don't know if this "Barry" guy has the appropriate permissions on the bugtracker to increase priorities, so I've taken the liberty of upgrading it as a release blocker for the _second_ beta ... ;-). So, at least there's been one productive consequence of this discussion. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 9:21 AM, <[EMAIL PROTECTED]> wrote: > On 03:33 pm, [EMAIL PROTECTED] wrote: >> It needs to be decided case-by-case. > > If certain tests are to be ignored on a case-by-case basis, why not record > that decision by disabling the test in the code? Otherwise, the decision > inevitably gets boiled down to "it's okay to release the code with a bunch > of tests failing, but I don't know which ones". As it was on Twisted when > we used to make case-by-case decisions about failures, and as it is here > now. No, I just meant that we need to figure out for each 3rd party test that fails whether the failure is our fault (too incompatibile) or theirs (relied on undefined behavior) and what the best fix is (change our code or theirs -- note that even if it's there fault there are cases where the best fix is to change our code. >> The beta is called beta because, well, it may break stuff and we may want >> to fix it. > > That's also why the alpha is called an alpha. My informal understanding is > that a beta should have no (or at least very few) known issues, with a > medium level of confidence that no further ones will be found. An RC would > have absolutely no known issues with a fairly high level of confidence that > no further ones will be found. This would include the community buildbots > basically working for the most part; I would not be surprised if there were > a few tests that still had issues. I guess we differ in interpretation. I see alphas as early outreach to the community. I see betas as a line in the sand where we stop adding/changing functionality -- but certainly not the moment where we expect everything to be perfect. During the beta period we fix bugs only -- those could be problems with new features, but those could also be problems where we accidentally introduced a backwards incompatibility. I'm sorry if your interpretation of the terminology is different, but this is mine and this is what we've always used, and it's not likely to change. (At least not for the 2.6/3.0 release.) > But clearly the reality does not meet my informal expectations, so it would > be nice to have something written down to check against. Still, I'm > bringing this up now because it _is_ a beta, and I think it will be too late > to talk about dealing with persistent test failures after the RCs start > coming out. Absolutely correct. The RCs are hoped to be as good as the final release. *Now* is the time to bring up issue. But please bring up specific issues -- I don't want to have an extended discussion about process or quality or expectations. I just want the code to be fixed. > (Of course, I'm just being sneaky. I don't actually care if it's clearly > documented, I just care that I stop having problems with incompatibility. > But I believe having it clearly spelled out would actually prevent a lot of > the problems that I've been having, since I don't think anyone would > *intentionally* select a policy where every release breaks at least one > major dependent project.) And here we seem to be parting our ways. We have a large amount of process already. I don't want more. >> I'm not particularly impressed by statistics like "all tests are red" -- >> this >> may all be caused by a single issue. > > The issue, for me, is not specifically that tests are red. It's that > there's no clear policy about what to do about that. Can a release go out > with some of the tests being red? If so, what are the extenuating > circumstances? Does this have to be fixed? If not, why not? Why are we > talking about this now? Shouldn't the change which caused Django to become > unimportable have been examined at the time, rather than months later? (In > other words, if it *is* a single issue, great, it's easy to fix: revert that > single issue.) If not, then shouldn't someone in Django-land have been > notified so they could cope with the change? > > Sorry that there are so many questions here; if I had fewer, I'd use fewer > words to ask them. If you're talking about community buildbots (which I presume are testing 3rd party packages against the core release) being red, that's out of scope for the core developers. We have enough trouble keeping our own (core) buildbots green, as you may have noticed. Some of the core buildbots are red because, well, frankly, they run on a cranky old version of an OS that few people care about. And then sometimes we leave them red, or turn them off. I see the core buildbots as a tool for developers to have their changes tested on a wide variety of platforms. They are an internal tool that gives us guidance about whether we're ready. I hope the community buildbots can be used the same way: a red bot means someone needs to look into an issue. The issue could be with the core or with the 3rd party package being tested. I don't think a policy like "no community buildbots should be red" makes any sense. >> For example, I'd much rather read an explanation about
Re: [Python-Dev] Community buildbots and Python release quality metrics
And just to make my position perfectly clear, I've unassigned it, since I don't foresee to be able to give this issue the quality time it clearly needs. Mind you, I agree it's a release blocker. But I don't have time to personally investigate it. Sorry. On Thu, Jun 26, 2008 at 9:54 AM, <[EMAIL PROTECTED]> wrote: > On 04:42 pm, [EMAIL PROTECTED] wrote: >> >> On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <[EMAIL PROTECTED]> >> wrote: >>> >>> an explanation about *why* Django cannot even be imported than a >>> blanket complaint that this is a disgrace. So why is it? > >> and already discussed: >> http://mail.python.org/pipermail/python-dev/2008-April/078421.html > > Following that trail of breadcrumbs, I ended up here: > > http://bugs.python.org/issue2235 > > with a message from some guy named "Barry Warsaw" (anyone know him?) that > says: > > """ > Guido, can you comment on Amaury's latest patch? I'm going to bump this > back to critical so as not to hold up 2.6 alpha, but it should be marked > as a release blocker for the first beta. > """ > > I don't know if this "Barry" guy has the appropriate permissions on the > bugtracker to increase priorities, so I've taken the liberty of upgrading it > as a release blocker for the _second_ beta ... ;-). So, at least there's > been one productive consequence of this discussion. > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> That's also why the alpha is called an alpha. My informal understanding > is that a beta should have no (or at least very few) known issues No, that's not the purpose. Instead, it is a promise that no further features will be added, i.e. that the code is stable from a feature point of view. It certainly will contain bugs. The final release will certainly contain bugs, just look at the long list of open bug reports. > The issue, for me, is not specifically that tests are red. It's that > there's no clear policy about what to do about that. Can a release go > out with some of the tests being red? If so, what are the extenuating > circumstances? The final release, no. The beta release: sure (in particular if it's the first beta release). Also make a difference between the Python buildbots, and the community buildbots. I think few if any core committers ever look at the status of the community buildbots - the community has to bring breakage to our attention (as you just did). > Does this have to be fixed? If not, why not? Here, I'm confused what specifically you talk about. That Django doesn't import? It is certainly not part of the release procedure to verify that Django can still be imported. > Why are > we talking about this now? Shouldn't the change which caused Django to > become unimportable have been examined at the time, rather than months > later? Somebody should have pointed it out when it happened. Unfortunately, nobody did, so apparently, the community doesn't really care. > I don't know. JP is already addressing the issues affecting Twisted in > another thread (incompatible changes in the private-but-necessary-to- > get-any-testing-done API of the warnings module). But I really think > that whoever made the change which broke it should be the one > investigating it, not me. How could they have known that they broke it? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> A big part of why I wrote this message is that I wanted a clear > understanding of the relationship between the definition of "alpha", > "beta" and "RC" and the state of various buildbots. If that > relationship exists already, just linking to it from > http://python.org/download/releases/2.6/ would be good. By the way, > that page still says these are "alpha" releases. I think its significantly simpler to answer specific questions on the mailing list, to educate the specific set of people who have this question, than to put an official answer on the web page. So I'm skeptical that anybody will change the web page - just continue asking questions as you encounter them. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 6:54 PM, <[EMAIL PROTECTED]> wrote: > On 04:42 pm, [EMAIL PROTECTED] wrote: >> >> On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <[EMAIL PROTECTED]> >> wrote: >>> >>> an explanation about *why* Django cannot even be imported than a >>> blanket complaint that this is a disgrace. So why is it? > >> and already discussed: >> http://mail.python.org/pipermail/python-dev/2008-April/078421.html > > Following that trail of breadcrumbs, I ended up here: > > http://bugs.python.org/issue2235 > > with a message from some guy named "Barry Warsaw" (anyone know him?) that > says: > > """ > Guido, can you comment on Amaury's latest patch? I'm going to bump this > back to critical so as not to hold up 2.6 alpha, but it should be marked > as a release blocker for the first beta. > """ > > I don't know if this "Barry" guy has the appropriate permissions on the > bugtracker to increase priorities, so I've taken the liberty of upgrading it > as a release blocker for the _second_ beta ... ;-). So, at least there's > been one productive consequence of this discussion. this patch even make things worse for me. now py.test also dies. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt <[EMAIL PROTECTED]> wrote: > this patch even make things worse for me. now py.test also dies. Please add details to the tracker. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 7:55 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt <[EMAIL PROTECTED]> wrote: >> this patch even make things worse for me. now py.test also dies. > > Please add details to the tracker. > Well, I think most probably the patch is correct and it just catches another class which defines __eq__ but does not define __hash__. So nothing to add there. The question is if this behaviour should be kept or not. I guess there's no bug tracking that... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 11:07 AM, Ralf Schmitt <[EMAIL PROTECTED]> wrote: > On Thu, Jun 26, 2008 at 7:55 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: >> On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt <[EMAIL PROTECTED]> wrote: >>> this patch even make things worse for me. now py.test also dies. >> >> Please add details to the tracker. > Well, I think most probably the patch is correct and it just catches > another class which defines __eq__ but does not define __hash__. So > nothing to add there. So you're saying py.test needs to be fixed? Fine with me, but then I don't understand why you bothered bringing it up here instead of fixing it (or reporting to the py.test maintainers, I don't know if you're one or not). > The question is if this behaviour should be kept or not. I guess > there's no bug tracking that... The best place is that very same issue. If this feature is really too annoying something needs to be done. Detailed reports about real code that gets broken without a good cause help making that case. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
I do tend to ramble on, so here's an executive summary of my response: I want python developers to pay attention to the community buildbots and to treat breakages of existing projects as a serious issue. However, I don't think that maintaining those projects is the core team's job, so all I'm asking for is for core developers to: * treat breakages of 3rd party packages as a potentially serious issue, * if possible (i.e. if they find out about the breakage soon enough, which should be the case in any pybots failure) revert the change that caused the problem until the problem can be fixed, and * notify 3rd party maintainers when it's decided that the breakage will not be fixed. This only applies to breakages that the core developers find out about, which for all practical purposes means the ones on the community builders page. Those of you looking for point-by-point responses and some repetition of the above points, enjoy :). On 05:03 pm, [EMAIL PROTECTED] wrote: On Thu, Jun 26, 2008 at 9:21 AM, <[EMAIL PROTECTED]> wrote: On 03:33 pm, [EMAIL PROTECTED] wrote: It needs to be decided case-by-case. (misunderstanding) No, I just meant that we need to figure out for each 3rd party test that fails whether the failure is our fault (too incompatibile) or theirs (relied on undefined behavior) and what the best fix is (change our code or theirs -- note that even if it's there fault there are cases where the best fix is to change our code. This is basically fine, as far as I'm concerned. I would like to suggest, however, that these issues be dealt with as soon as possible, rather than waiting for the release process to begin. A lot of decisions are made on this mailing list about the supposed properties of "average" python code, without any actual survey of said code. Sometimes the results of that survey can be really surprising. The end goal of any particular compatibility policy, of a distinction between "public" and "private" APIs, and so on, is to keep code working. I'm sorry if your interpretation of the terminology is different, but this is mine and this is what we've always used, and it's not likely to change. (At least not for the 2.6/3.0 release.) I have no problem with your definitions of these terms. I think that they should probably be in PEP 101 though. Would you accept a patch that added an edited / expanded version of this paragraph? Still, I'm bringing this up now because it _is_ a beta, Absolutely correct. The RCs are hoped to be as good as the final release. *Now* is the time to bring up issue. Well, that's good, at least :) But please bring up specific issues -- I don't want to have an extended discussion about process or quality or expectations. I just want the code to be fixed. Well, one specific issue has been bumped in priority as a result of this thread, and others are under discussion. The code is getting fixed. (I just care that I stop having problems with incompatibility.) And here we seem to be parting our ways. We have a large amount of process already. I don't want more. Looking at it from my perspective, I'm proposing a reduction in process. Under the current process, if a buildbot goes red, the developer makes a judgment call, the release manager makes a judgment call, there's discussion on a ticket, a ticket gets filed, it gets promoted, it gets demoted, the RM forgets to re-promote it... My suggestion is that the process be, simply: if a buildbot (community or otherwise) goes red, the change that broke it gets reverted. No questions asked! It's still there in the revision history, ready to be re-applied once the issues get worked out. Discussion can then take place and case-by-case judgments can be applied. If you're talking about community buildbots (which I presume are testing 3rd party packages against the core release) being red, that's out of scope for the core developers. I don't necessarily think that keeping the community buildbots red is the core developers' responsibility, but I don't think it should be entirely out of scope, either. The python test suite is, frankly, poor - and I hope I'm not surprising you by saying that. It's full of race conditions, tends to fail intermittently, and is frequently ignored. Not only that, but it is quite often changed, so tests for issues that affect real code are quite often removed. So, the community buildbots are not just making sure that 3rd-party code still works, they are an expanded, independently developed test suite to make sure that *python itself* still works. Sometimes they will not fill that role particularly well, but they are worth paying attention to. If python had a good, exhaustive regression test suite that was immutable between major versions, I'd probably feel differently. But that's not the world we live in. Right now, apparently, the *default* policy is that if the community buildbots go red, later, before a release, s
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 05:14 pm, [EMAIL PROTECTED] wrote: I don't know. JP is already addressing the issues affecting Twisted in another thread (incompatible changes in the private-but-necessary-to- get-any-testing-done API of the warnings module). But I really think that whoever made the change which broke it should be the one investigating it, not me. How could they have known that they broke it? Because the relevant community buildbot turned red with that revision of trunk. Keep in mind I'm not talking about every piece of Python code in the universe; just the ones selected for the community buildbots. It would be nice if there were a dozen or so projects on there, but paying attention to the two builders that do actually run right now would be a good start. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] schrieb: Another way to phrase this question is, "whose responsibility is it to make Python 2.5 programs run on Python 2.6"? Or, "what happens when the core team finds out that a change they have made has broken some python software 'in the wild'"? Here are a couple of ways to answer these questions: 1) It's the python core team's responsibility. There should be Python buildbots for lots of different projects and they should all be green before the code can be considered [one of: allowed in trunk, alpha, beta, RC]. 2) It's the python core team's responsibility as long as the major projects are using public APIs. The code should not be considered [alpha, beta, RC] if there are any known breakages in public APIs. > 3) It's only the python core team's responsibility to notify projects of incompatible changes of which they are aware, which may occur in public or private APIs. Major projects with breakages will be given a grace period before a [beta, RC, final] to release a forward-compatible version. 4) The status quo: every new Python release can (and will, at least speaking in terms of 2.6) break lots of popular software, with no warning aside from the beta period. There are no guarantees. Python's backwards compatibility has never extended to non-public APIs. That rules out 1). 2) would be nice, but to be honest, 2) and 3) are unrealistic given the number of Python core developers and the tasks already at hand. 4) is not acceptable either. However, you seem to be overlooking that there's a 3.5) in there: it's the Python core team's responsibility as long as the major projects are using public APIs; but to reduce the workload every project should watch its own buildbots and notify the core team using the issue tracker in order to get the issue resolved. At no time will a policy "the community buildbots must be green" be useful: the tests that run on these buildbots are not under our control, so if the tests test things we deem non-public we can't do anything about it. (And we may have a hard time convincing project authors to change the tests if we promised to make them run anyway.) Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] schrieb: On 03:42 pm, [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: beta 1 has some trouble running *our* test suite - I'd be fairly surprised if the community buildbots were in significantly better shape. That's another problem, yes :) The community buildbots have been in a broken state for months now[1]. If the community buildbots aren't largely green by the time beta 2 comes out, that's when I'll agree we have a problem - they should definitely be green by the time first release candidate comes out. I have mixed feelings about this: on the one hand, if this were a matter of public record, I would have known that this was too early to start complaining, and we could have saved everyone a lot of time reading these messages. That would be good; I would prefer to just let everyone work without bothering about process. The problem is that not enough people care about the community buildbots, and this isn't likely to improve since we have not enough manpower to do it. On the other hand, it's much easier to deal with test failures as they arise than in one giant chunk of work at the end of the development process. I spoke to a few core developers at PyCon who thought that the buildbots should always be green and any change that broke them should be reverted. They may remain nameless unless they wish to raise their hands :-) but that's definitely how I feel about it. I think no core developer will not tell you that the (core) buildbots should be green all the time. As for reverting changes that break, I'd support this only for changes that break *all* of them. For example, I only use one platform to develop on (and I guess it's the same for many others), having the buildbots go red on another platform means I can try to fix the issue. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
To me (and I'm an outsider not a direct developer), it's Python developers responsibility to handle the Python problems and the Python build bots. The community build bots are the responsibility of their authors. If JP is handling the Twisted one then great. It's got a gatekeeper. If no one is handling the Django build bot that's not the Python Dev Team's problem but is a problem on the Django side and someone involved in that should be tasked with monitoring. Having said all that, I agree that the community bots ought to at least be looked at. If I check in a patch in and I notice that a community bot went from green to red then I probably should double check my code. The problem is that, as you said, the community bots haven't been well maintained... They've been red for awhile. So asking the developers to then go do the leg work to find the original error, fix it and then back check everything between then and now that might have ALSO caused a problem is alot of effort. It's not the Bank's responsibility to balance my check book. They give me the tools to do it and then I have to check. A similar principle applies here, I believe. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, 26 Jun 2008 21:46:48 +0200, Georg Brandl <[EMAIL PROTECTED]> wrote: [snip] As for reverting changes that break, I'd support this only for changes that break *all* of them. For example, I only use one platform to develop on (and I guess it's the same for many others), having the buildbots go red on another platform means I can try to fix the issue. BuildBot has two ways to let you run your code on all builders before you commit it to trunk. You can force a build on a branch or you can try a build with a patch. I don't know if these options are enabled on Python's buildmaster. If they are, then if you want, you can use them to make sure your code works on all platforms before you put it into trunk, where it may cause problems for someone else. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> I want python developers to pay attention to the community buildbots I don't think that goal is realistic. Instead, somebody who has actual interest in this matter should pay this attention, and then bring it up on python-dev when something breaks. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] wrote: to what extent should Python actually be compatible between releases? As I understand things from years of observation, the following are fair game to changed in ways possibly backward-incompatible for specific code: bugs, detailed float behavior (which may be system-specific anyway), error messages, private interfaces, non-enforcement of documented rules, and defined implementation-dependent behavior. But Guido has been somewhat conservative about this (least so for bug fixes, I think) and sometimes put off changes even when the fault lies with questionable usages. One fly in the ointment is that bugs (a discrepancy between doc and code) may be fixed either by changing the doc or the code, according to which embodies the design intention. So neither can be taken as gospel. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
Georg Brandl wrote: [EMAIL PROTECTED] schrieb: Another way to phrase this question is, "whose responsibility is it to make Python 2.5 programs run on Python 2.6"? Or, "what happens when the core team finds out that a change they have made has broken some python software 'in the wild'"? Here are a couple of ways to answer these questions: 1) It's the python core team's responsibility. There should be Python buildbots for lots of different projects and they should all be green before the code can be considered [one of: allowed in trunk, alpha, beta, RC]. 2) It's the python core team's responsibility as long as the major projects are using public APIs. The code should not be considered [alpha, beta, RC] if there are any known breakages in public APIs. > 3) It's only the python core team's responsibility to notify projects of incompatible changes of which they are aware, which may occur in public or private APIs. Major projects with breakages will be given a grace period before a [beta, RC, final] to release a forward-compatible version. 4) The status quo: every new Python release can (and will, at least speaking in terms of 2.6) break lots of popular software, with no warning aside from the beta period. There are no guarantees. Python's backwards compatibility has never extended to non-public APIs. That rules out 1). 2) would be nice, but to be honest, 2) and 3) are unrealistic given the number of Python core developers and the tasks already at hand. 4) is not acceptable either. However, you seem to be overlooking that there's a 3.5) in there: it's the Python core team's responsibility as long as the major projects are using public APIs; but to reduce the workload every project should watch its own buildbots and notify the core team using the issue tracker in order to get the issue resolved. At no time will a policy "the community buildbots must be green" be useful: the tests that run on these buildbots are not under our control, so if the tests test things we deem non-public we can't do anything about it. (And we may have a hard time convincing project authors to change the tests if we promised to make them run anyway.) If, however, the community buildbot goes red because of a regression or backwards incompatibility the developers should, when notified, at least undertake to verify that the breakage is intentional. Unfortunately having a buildbot is like installing a firewall. Some people assume that the very presence of the buildbot is a guard against version incompatibilities, whereas in truth there is always the messy business of communication required to ensure the information gets to where it is useful. The bottom line, of course, is that if project X can't be bothered to tell the core development team when changes break their build it's nobody's fault but their own. If the report it and the core developers ignore the reports, that's something that should be rectified. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> BuildBot has two ways to let you run your code on all builders before you > commit it to trunk. You can force a build on a branch or you can try a > build with a patch. I don't know if these options are enabled on Python's > buildmaster. If they are, then if you want, you can use them to make sure > your code works on all platforms before you put it into trunk, where it may > cause problems for someone else. Even if that's possible, it adds a burden. As likely not all people agree that such a procedure would be a good thing, it's necessary that somebody establishes a policy. Guido has already indicated that he is not interested in further policies; Barry (the canonical other person to set policies) has indicated in the past that he would like such a policy, but deems it unrealistic at this point. He gives other procedural changes higher priority, so that realistically, such a policy might not be established within the next two years. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 07:44 pm, [EMAIL PROTECTED] wrote:
At no time will a policy "the community buildbots must be green" be
useful: the tests that run on these buildbots are not under our
control,
so if the tests test things we deem non-public we can't do anything
about it. (And we may have a hard time convincing project authors to
change the tests if we promised to make them run anyway.)
That's not what I'm suggesting.
If there is a legitimate disagreement between Python developers and
developers of a project about whether an API should continue to be
supported, then clearly, the Python developers get to win. Welcome to
open source.
However, I believe that the core team is not paying attention to other
projects breaking. I'm not saying that you want to make breaking
changes, or that bug reports are not dealt with, but the problem is that
dealing with these problems after the fact makes it _much_ more painful.
When you get to the release, and you have 30 bug reports due to other
projects breaking, they get triaged, some get left in, and some features
of lots of different projects are left broken. And many projects do not
bother to test with the beta.
If developers were simply presented with the results of their changes
immediately ("you broke twisted, django doesn't import, zope segfaults
and mercurial destroys your repository") then perhaps they would weigh
the benefits of compatibility differently, and do the trivial, obvious
fix immediately, rather than waiting 6 months and having to figure out
what the heck change caused the bug. I accept the fact that python core
development is done differently than i.e. Java development, and some
backward compatibility may simply be broken.
Case in point: changes to the warnings module being discussed in a
separate thread break a significant number of Twisted's tests, because
they remove functionality which is not public but is required to test
code that uses the warnings module. Would Brett have taken this into
account if he knew about it immediately when revision 62303 was
committed? Maybe not, but now that the code is written and done,
there's significantly less motivation to go back and fix it. At the
time it might have only been a few minutes' work to continue supporting
the use-case which Twisted needs. Brett wouldn't even have necessarily
needed to do the work: it would have been easier to convince a Twisted
developer to do the work it to keep the community buildbot green rather
than to make it a bit less red. Now, though, it's much easier to say
"oh well, that's private, you lose". I don't ascribe this to malice -
it really *would* be much harder to fix it now, for us as well as for
him.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> I don't ascribe this to malice - > it really *would* be much harder to fix it now, for us as well as for him. I think I disagree. It's easier to fix it now than it was to fix it back then. Fixing it back then would have meant to constantly observe the buildbots, and keep them running, so it would be a huge effort to maintain the infrastructure just to find a the few changes that unintentially broke something. Looking at them now is a lot of effort, also. But this effort is feasible, once the root cause is identified, the patch causing it might just get backed out. Maintaining the community buildbots has proven infeasible. It's unfortunate that many package authors don't understand that not all breakage is deliberate, and that their only chance to get undesired breakage reverted is to report bugs NOW. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] schrieb:
On 07:44 pm, [EMAIL PROTECTED] wrote:
At no time will a policy "the community buildbots must be green" be
useful: the tests that run on these buildbots are not under our
control,
so if the tests test things we deem non-public we can't do anything
about it. (And we may have a hard time convincing project authors to
change the tests if we promised to make them run anyway.)
That's not what I'm suggesting.
Sure, but I wanted to say that nevertheless :)
If there is a legitimate disagreement between Python developers and
developers of a project about whether an API should continue to be
supported, then clearly, the Python developers get to win. Welcome to
open source.
However, I believe that the core team is not paying attention to other
projects breaking.
Quite true. It is their duty to bring the breakage to our attention, if
they want to see results.
I'm not saying that you want to make breaking
changes, or that bug reports are not dealt with, but the problem is that
dealing with these problems after the fact makes it _much_ more painful.
When you get to the release, and you have 30 bug reports due to other
projects breaking, they get triaged, some get left in, and some features
of lots of different projects are left broken. And many projects do not
bother to test with the beta.
I thought we're talking about the projects that *do* use the buildbots?
If developers were simply presented with the results of their changes
immediately ("you broke twisted, django doesn't import, zope segfaults
and mercurial destroys your repository") then perhaps they would weigh
the benefits of compatibility differently, and do the trivial, obvious
fix immediately, rather than waiting 6 months and having to figure out
what the heck change caused the bug.
This sounds very nice in theory, but it must be implemented in a way that
does not add a burden to the developer, as Martin said.
> Case in point: changes to the warnings module being discussed in a
> separate thread break a significant number of Twisted's tests, because
> they remove functionality which is not public but is required to test
> code that uses the warnings module. Would Brett have taken this into
> account if he knew about it immediately when revision 62303 was
> committed? Maybe not, but now that the code is written and done,
> there's significantly less motivation to go back and fix it. At the
> time it might have only been a few minutes' work to continue supporting
> the use-case which Twisted needs. Brett wouldn't even have necessarily
> needed to do the work: it would have been easier to convince a Twisted
> developer to do the work it to keep the community buildbot green rather
> than to make it a bit less red. Now, though, it's much easier to say
> "oh well, that's private, you lose". I don't ascribe this to malice -
> it really *would* be much harder to fix it now, for us as well as for
> him.
I must say that I'm increasingly surprised by this discussion since it seems
that not only the Python core devs neglect the community buildbots.
Shouldn't the project authors have at least the same interest that their
code runs on the next major Python version? And while they don't necessarily
have more time to watch the buildbots than we have, the amount of what they
need to keep an eye on.
Georg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
Terry Reedy schrieb: [EMAIL PROTECTED] wrote: to what extent should Python actually be compatible between releases? As I understand things from years of observation, the following are fair game to changed in ways possibly backward-incompatible for specific code: bugs, detailed float behavior (which may be system-specific anyway), error messages, private interfaces, non-enforcement of documented rules, and defined implementation-dependent behavior. But Guido has been somewhat conservative about this (least so for bug fixes, I think) and sometimes put off changes even when the fault lies with questionable usages. Which is the source of many unresolved open issues in the tracker -- everyone agrees that "it" is a bug and should be fixed, perhaps there's even a patch, but people may be using it this way, so nothing is done forever. (And these bugs usually are the kind of things we don't want to change in 3.0 either since they are subtle things.) However, I don't have a solution here, so I won't complain about it anymore. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jun 26, 2008, at 11:42 AM, Nick Coghlan wrote: If the community buildbots aren't largely green by the time beta 2 comes out, that's when I'll agree we have a problem - they should definitely be green by the time first release candidate comes out. Just FTR, I'll be looking at the Python buildbots and Roundup tracker to decide whether I think beta2 is in shape to be released or not. I definitely won't be tracking the non-core buildbots, so if something troublesome comes up there, you will have to submit a bug report to the Python tracker. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGQEu3EjvBPtnXfVAQIp6QP/bOU+8vImlRkfKCXazYhi3yEXcNwCPpWI e3AoiOlgKbOhSPraNTyUBjC+OjuqHQttyMJz8TTpexSwhpG/erDIqUjRbkW0Yaas attvFZBYbJhF3qvHyLzmSp9U3oXY6VyWayL8ZdzIGcukEW7g8DgQNQyoGOE4kEeX i6+4RIRf+7A= =X1lh -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jun 26, 2008, at 12:54 PM, [EMAIL PROTECTED] wrote: On 04:42 pm, [EMAIL PROTECTED] wrote: On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: an explanation about *why* Django cannot even be imported than a blanket complaint that this is a disgrace. So why is it? and already discussed: http://mail.python.org/pipermail/python-dev/2008-April/078421.html Following that trail of breadcrumbs, I ended up here: http://bugs.python.org/issue2235 with a message from some guy named "Barry Warsaw" (anyone know him?) that says: """ Guido, can you comment on Amaury's latest patch? I'm going to bump this back to critical so as not to hold up 2.6 alpha, but it should be marked as a release blocker for the first beta. """ I don't know if this "Barry" guy has the appropriate permissions on the bugtracker to increase priorities, so I've taken the liberty of upgrading it as a release blocker for the _second_ beta ... ;-). So, at least there's been one productive consequence of this discussion. Glyph, you did the right thing. I wish there was a way of marking a bug "not-release-blocker-for-now" and have it automatically update back to blocker at a certain time or after a certain event. I think it's valid for this issue to block beta2. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGQF0XEjvBPtnXfVAQJwiQP/T9f33lNtAQ9/eooKEyVCDms7NXJE7mIf QPOQtXuTZdsfUl51OxkxewVj6ERZasHPwIB2c13HYuHRrMrmrE9EYStdbmMxh0BI 7EhsO4SfDI3ZnYFJvAEMrTvgY8Vz4v817LhzWNZ7RWxq/yOHG8C/ZgubPJIa8mnd EGWUVoLg77E= =OJHh -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 12:35 PM, <[EMAIL PROTECTED]> wrote: > > On 05:14 pm, [EMAIL PROTECTED] wrote: >>> >>> I don't know. JP is already addressing the issues affecting Twisted in >>> another thread (incompatible changes in the private-but-necessary-to- >>> get-any-testing-done API of the warnings module). But I really think >>> that whoever made the change which broke it should be the one >>> investigating it, not me. >> >> How could they have known that they broke it? > > Because the relevant community buildbot turned red with that revision of > trunk. Keep in mind I'm not talking about every piece of Python code in the > universe; just the ones selected for the community buildbots. It would be > nice if there were a dozen or so projects on there, but paying attention to > the two builders that do actually run right now would be a good start. Sorry, this is an unacceptable burden on the core developers. The core developers aren't going to look at the community buildbots (unless voluntarily) and they aren't going to roll back changes just because some community buildbot goes red. In general someone outside the core developer group needs to bring the issue to the core developers' attention and then a fix will be created if appropriate. Rollbacks are generally reserved for accidental checkins or checkins against the process rules (e.g. during a code freeze). Heck, we don't even roll back if one of our own buildbots goes red. I'm fine with requesting that the core developers pay serious attention to reports about 3rd party code being broken. The community buildbots are a fine tool to find out about this. But any policy that requires an automatic rollback because a buildbot (community or core) goes red is unacceptable. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
Barry Warsaw schrieb: I don't know if this "Barry" guy has the appropriate permissions on the bugtracker to increase priorities, so I've taken the liberty of upgrading it as a release blocker for the _second_ beta ... ;-). So, at least there's been one productive consequence of this discussion. Glyph, you did the right thing. I wish there was a way of marking a bug "not-release-blocker-for-now" and have it automatically update back to blocker at a certain time or after a certain event. I think it's valid for this issue to block beta2. I just added a "deferred blocker" priority -- that implements one half of your wish. Mass issue updating must be done by someone who knows Roundup better than me, I'm afraid. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 1:06 PM, <[EMAIL PROTECTED]> wrote:
> On 07:44 pm, [EMAIL PROTECTED] wrote:
>>
>> At no time will a policy "the community buildbots must be green" be
>> useful: the tests that run on these buildbots are not under our control,
>> so if the tests test things we deem non-public we can't do anything
>> about it. (And we may have a hard time convincing project authors to
>> change the tests if we promised to make them run anyway.)
>
> That's not what I'm suggesting.
>
> If there is a legitimate disagreement between Python developers and
> developers of a project about whether an API should continue to be
> supported, then clearly, the Python developers get to win. Welcome to open
> source.
>
> However, I believe that the core team is not paying attention to other
> projects breaking. I'm not saying that you want to make breaking changes,
> or that bug reports are not dealt with, but the problem is that dealing with
> these problems after the fact makes it _much_ more painful. When you get to
> the release, and you have 30 bug reports due to other projects breaking,
> they get triaged, some get left in, and some features of lots of different
> projects are left broken. And many projects do not bother to test with the
> beta.
Well, sorry, that's life. We're not going to deal with breakage in 3rd
party code on a "drop all other work" basis.
You seem to have a different idea about how to do development than the
core developers. Tough luck. You can't tell us how to do our work.
We're doing the best we can, and we're happy to listen to suggestions
you're making, but the set of suggestions you're making in this thread
are an unacceptable burden, and it's not going to happen.
> If developers were simply presented with the results of their changes
> immediately ("you broke twisted, django doesn't import, zope segfaults and
> mercurial destroys your repository") then perhaps they would weigh the
> benefits of compatibility differently, and do the trivial, obvious fix
> immediately, rather than waiting 6 months and having to figure out what the
> heck change caused the bug. I accept the fact that python core development
> is done differently than i.e. Java development, and some backward
> compatibility may simply be broken.
>
> Case in point: changes to the warnings module being discussed in a separate
> thread break a significant number of Twisted's tests, because they remove
> functionality which is not public but is required to test code that uses the
> warnings module. Would Brett have taken this into account if he knew about
> it immediately when revision 62303 was committed? Maybe not, but now that
> the code is written and done, there's significantly less motivation to go
> back and fix it.
I disagree. It's broken and should be fixed. Beta 1 just came out so
this is the perfect time to file a bug.
> At the time it might have only been a few minutes' work to
> continue supporting the use-case which Twisted needs. Brett wouldn't even
> have necessarily needed to do the work: it would have been easier to
> convince a Twisted developer to do the work it to keep the community
> buildbot green rather than to make it a bit less red.
Why? That sounds like it's a problem with the psychology of the
Twisted developers.
> Now, though, it's
> much easier to say "oh well, that's private, you lose". I don't ascribe
> this to malice - it really *would* be much harder to fix it now, for us as
> well as for him.
I don't believe this.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jun 26, 2008, at 5:20 PM, Georg Brandl wrote: Barry Warsaw schrieb: I don't know if this "Barry" guy has the appropriate permissions on the bugtracker to increase priorities, so I've taken the liberty of upgrading it as a release blocker for the _second_ beta ... ;-). So, at least there's been one productive consequence of this discussion. Glyph, you did the right thing. I wish there was a way of marking a bug "not-release-blocker-for-now" and have it automatically update back to blocker at a certain time or after a certain event. I think it's valid for this issue to block beta2. I just added a "deferred blocker" priority -- that implements one half of your wish. Mass issue updating must be done by someone who knows Roundup better than me, I'm afraid. Cool, thanks! I should have thought of that before. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGQNm3EjvBPtnXfVAQIy1wP/QPFwklmnay8VPPovHA6H4XA6tW+ziWPV hbeyAZX/MMxEYv/98Z4nOQU7M4AczlXHZks80UvDdKLwPe2wTwfOIgmCTeb+vciI 20GpxrHpWQNHy/XKeawEBxyLHJZ4qNGIAFmHwoiUusM6erTeVb9kWa0czG31En6r Z4MV0YKfu+A= =Zxh4 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> I just added a "deferred blocker" priority -- that implements one half > of your wish. Mass issue updating must be done by someone who knows > Roundup better than me, I'm afraid. I doubt true mass update will be necessary. If you remind me that I should convert all "deferred blocker" issues to some other priority, I'll do that manually in a matter of minutes (using a query to find all issues with that priority). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 8:45 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > So you're saying py.test needs to be fixed? Fine with me, but then I > don't understand why you bothered bringing it up here instead of > fixing it (or reporting to the py.test maintainers, I don't know if > you're one or not). > no, I'm not. Just wanted to point out, that this ticket/patch does not solve the django issue. (this is what I first thought it would do). - Ralf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jun 26, 2008, at 5:54 PM, Martin v. Löwis wrote: I just added a "deferred blocker" priority -- that implements one half of your wish. Mass issue updating must be done by someone who knows Roundup better than me, I'm afraid. I doubt true mass update will be necessary. If you remind me that I should convert all "deferred blocker" issues to some other priority, I'll do that manually in a matter of minutes (using a query to find all issues with that priority). Martin, I think that will work great. The idea being that after each release, we always bump deferred blockers back to release blocker. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGQZbHEjvBPtnXfVAQIwWQQAqWoIoqV0S0O8ZpkXpWd3LO4Fo7MBfjRT LL5QgIPBWdQdZ4RR68LSfdAhiHGnZCZorpNksGaeIxP55qgS1z31fy3JF39UUbVR 3ojymtF6Is0qCB5lmkJIx9Na/2RUEOUWTQoQP3dG851oRRuSVTXb4uRaBlNlQErY oN9JxG5xQR0= =z9nN -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 09:17 pm, [EMAIL PROTECTED] wrote: On Thu, Jun 26, 2008 at 12:35 PM, <[EMAIL PROTECTED]> wrote: Because the relevant community buildbot turned red with that revision of trunk. Keep in mind I'm not talking about every piece of Python code in the universe; just the ones selected for the community buildbots. It would be nice if there were a dozen or so projects on there, but paying attention to the two builders that do actually run right now would be a good start. Sorry, this is an unacceptable burden on the core developers. The core developers aren't going to look at the community buildbots (unless voluntarily) and they aren't going to roll back changes just because some community buildbot goes red. OK, fair enough. Taking a step back, I was pushing this really hard because to *me*, it seems like dealing with the influx of bug reports after the fact is an unreasonable amount of additional work, whereas immediate reverts are just an occasional, temporary inconvenience. Based on my experience, "We'll deal with the reports as they come in" sounds like "no". In general someone outside the core developer group needs to bring the issue to the core developers' attention and then a fix will be created if appropriate. Rollbacks are generally reserved for accidental checkins or checkins against the process rules (e.g. during a code freeze). Heck, we don't even roll back if one of our own buildbots goes red. I think since the last time we had a similar conversation, other Twisted developers have been fairly diligent in reporting bugs. (In the time they've been reporting Python bugs, I've mostly been planning a wedding. Others who have survived it tell me that eventually, this process ends... and then I should be participating more directly.) I'll try to step up that feeback loop and get other projects involved. I hope I am wrong about generating an unreasonable amount of work. I'm fine with requesting that the core developers pay serious attention to reports about 3rd party code being broken. The community buildbots are a fine tool to find out about this. But any policy that requires an automatic rollback because a buildbot (community or core) goes red is unacceptable. Thanks. I appreciate it; an increased emphasis on 3rd party code being broken is really what I was looking for. This is fine with me. I mean, whatever your decision is, I'm going to have to live with it :), but in either case, we have to be looking for bugs and helping to investigate them. On my end of things I guess it's not going to make much difference. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 26 Jun, 09:24 pm, [EMAIL PROTECTED] wrote: On Thu, Jun 26, 2008 at 1:06 PM, <[EMAIL PROTECTED]> wrote: On 07:44 pm, [EMAIL PROTECTED] wrote: Well, sorry, that's life. We're not going to deal with breakage in 3rd party code on a "drop all other work" basis. For the record, "automatic revert" does not mean "drop all other work". The changeset's still there. It's still in the revision history. It can easily be applied to anybody's working copy. It can easily be put into a branch. It can be automatically re-merged later. I do all of these things all the time, and I was *not* intending to suggest that any 3rd-party breakage should cause a code freeze or anything even remotely like that. Case in point: changes to the warnings module I disagree. It's broken and should be fixed. Beta 1 just came out so this is the perfect time to file a bug. I'll go back over the recent conversation and work out the specifics of the bug (if JP doesn't, or hasn't already beaten me to it). (...) it would have been easier to convince a Twisted developer to do the work it to keep the community buildbot green rather than to make it a bit less red. Why? That sounds like it's a problem with the psychology of the Twisted developers. I hardly think it's unique to us. TDD test runners typically only know 2 colors and 2 states: "passed" and "fails". Once you're in the "fail" state, you tend to accumulate more failures; there's a much bigger bang for your buck. Better tools with nicer interfaces would let you easily mark individual tests as "usually intermittent" or something and make it a "slightly dirty green" or something, but we don't have those. The move from "red" to "green" is much more psychologically significant to just about anyone I know than the move from "red but 14 failures" to "red but 12 failures". Despite the idea's origins in a now-highly-controversial book on criminology, this is often referred to in XP discussion circles (wikis and the like) as the "fix broken windows" collaboration pattern. I notice this in lots of other areas of my life; a little pile of papers tends to become a big pile of papers, the first dent in a car makes the driver a little less careful, and so on. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] wrote: I do tend to ramble on, so here's an executive summary of my response: I want python developers to pay attention to the community buildbots and to treat breakages of existing projects as a serious issue. Counter-proposal: * Interested developers or users of the major third party projects tested by the community buildbots should monitor the community buildbots, and start filing bug reports with the appropriate party as soon as the upcoming Python release hits 2.Xa1 (i.e. first alpha). * If the failure is due to an incompatibility between Python 2.X and 2.X-1, then the bug report should be filed against Python. While these issues may or may not be addressed before the first beta, they *must* be addressed before the first release candidate. * If the failure is due to an incompatibility between Python 2.X and 2.X-2 that was properly deprecated in 2.X-1, then the bug report should be filed against the third party project. Prioritising these is a question for the developers of that project. * Before filing a bug report against Python for a community buildbot failure, check if the relevant regression is also causing failures of the core buildbots. If it is, skip the bug report until the core buildbots are passing again. It's currently a fact of life that we do NOT keep the trunk in an always-releasable state. We just don't. It might be nice if we did, there are lots of reasons why that's a good way to run a project, but at this point in time it isn't the case with Python. Reacting every time a community buildbot goes red would be a serious waste of effort. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 3:22 PM, Ralf Schmitt <[EMAIL PROTECTED]> wrote: > On Thu, Jun 26, 2008 at 8:45 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: >> >> So you're saying py.test needs to be fixed? Fine with me, but then I >> don't understand why you bothered bringing it up here instead of >> fixing it (or reporting to the py.test maintainers, I don't know if >> you're one or not). >> > > no, I'm not. Just wanted to point out, that this ticket/patch does not > solve the django issue. > (this is what I first thought it would do). I'm still missing details. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 5:33 PM, <[EMAIL PROTECTED]> wrote: > OK, fair enough. Taking a step back, I was pushing this really hard because > to *me*, it seems like dealing with the influx of bug reports after the fact > is an unreasonable amount of additional work, whereas immediate reverts are > just an occasional, temporary inconvenience. Based on my experience, "We'll > deal with the reports as they come in" sounds like "no". No, that's not how it was intended. I'd rather hear you tell us about your pain than telling us how to fix it. > I think since the last time we had a similar conversation, other Twisted > developers have been fairly diligent in reporting bugs. (In the time > they've been reporting Python bugs, I've mostly been planning a wedding. > Others who have survived it tell me that eventually, this process ends... > and then I should be participating more directly.) I'll try to step up that > feeback loop and get other projects involved. I hope I am wrong about > generating an unreasonable amount of work. Again, I'd much rather see you generate a huge pile of work for us in the form of specific bug reports than trying to tell us how to change our process. > Thanks. I appreciate it; an increased emphasis on 3rd party code being > broken is really what I was looking for. This is fine with me. I mean, > whatever your decision is, I'm going to have to live with it :), but in > either case, we have to be looking for bugs and helping to investigate them. > On my end of things I guess it's not going to make much difference. Right. The beta period is the time to pay attention to 3rd party breakage. While we won't make any specific promises, we're taking 3rd party code breakage very seriously and, depending on the specifics, it can hold up a release. It just won't be automatic; many 3rd party breakages are most easily fixed by making small adjustments to the 3rd party code involved. (Imagine a 3rd party package still using string exceptions.) . . . On Thu, Jun 26, 2008 at 6:13 PM, <[EMAIL PROTECTED]> wrote: > For the record, "automatic revert" does not mean "drop all other work". The > changeset's still there. It's still in the revision history. It can easily > be applied to anybody's working copy. It can easily be put into a branch. > It can be automatically re-merged later. I do all of these things all the > time, and I was *not* intending to suggest that any 3rd-party breakage > should cause a code freeze or anything even remotely like that. Still, a revert often holds up other work that depends on the reverted change. I expect that in most cases, finding the problem and applying a fix is much less of a burden for the core developer team as a whole than rolling back, working on a fix, and re-applying. I also expect that the policy of semi-automated merges from 2.6 into 3.0 would make a revert even more work, and more likely to cause second-order problem in the 3.0 branch. >>> (...) it would have been easier to >>> convince a Twisted developer to do the work it to keep the community >>> buildbot green rather than to make it a bit less red. >> >> Why? That sounds like it's a problem with the psychology of the >> Twisted developers. > > I hardly think it's unique to us. TDD test runners typically only know 2 > colors and 2 states: "passed" and "fails". Once you're in the "fail" state, > you tend to accumulate more failures; there's a much bigger bang for your > buck. Better tools with nicer interfaces would let you easily mark > individual tests as "usually intermittent" or something and make it a > "slightly dirty green" or something, but we don't have those. The move from > "red" to "green" is much more psychologically significant to just about > anyone I know than the move from "red but 14 failures" to "red but 12 > failures". Still sounds like perhaps you're too much focused on the "green bar". I know this is encouraged by XP, but I don't want to have a religious debates about whether XP is the right development model for Python. (The time would be better spent on improving the buildbot reporting!) > Despite the idea's origins in a now-highly-controversial book on > criminology, this is often referred to in XP discussion circles (wikis and > the like) as the "fix broken windows" collaboration pattern. I notice this > in lots of other areas of my life; a little pile of papers tends to become a > big pile of papers, the first dent in a car makes the driver a little less > careful, and so on. For me, with that first dent comes the realization that it's really not worth it to obsess over scratches, and that makes me a more relaxed and hence *better* (because less stressed) driver. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail
Re: [Python-Dev] Community buildbots and Python release quality metrics
Guido van Rossum wrote: On Thu, Jun 26, 2008 at 3:22 PM, Ralf Schmitt <[EMAIL PROTECTED]> wrote: On Thu, Jun 26, 2008 at 8:45 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: So you're saying py.test needs to be fixed? Fine with me, but then I don't understand why you bothered bringing it up here instead of fixing it (or reporting to the py.test maintainers, I don't know if you're one or not). no, I'm not. Just wanted to point out, that this ticket/patch does not solve the django issue. (this is what I first thought it would do). I'm still missing details. I'm going to spend some time looking into this this weekend (issue assigned appropriately. Getting it working again shouldn't be too bad (just revert the relevants bits to their 2.5 equivalents) - the tricky part is going to be keeping the warning of the associated change to the semantics in 3.0. (I'll also take a look at 3.0 to make sure the relevant test case from the tracker issue works properly there) Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Thu, Jun 26, 2008 at 8:18 AM, <[EMAIL PROTECTED]> wrote: > Today on planetpython.org, Doug Hellman announced the June issue of Python > magazine. The cover story this month is about Pybots, "the fantastic > automation system that has been put in place to make sure new releases of > Python software are as robust and stable as possible". > > Last week, there was a "beta" release of Python which, according to the > community buildbots, cannot run any existing python software. Normally I'd > be complaining here about Twisted, but in fact Twisted is doing relatively > well right now; only 80 failing tests. Django apparently cannot even be > imported. > > The community buildbots have been in a broken state for months now[1]. I've > been restraining myself from whinging about this, but now that it's getting > close to release, it's time to get these into shape, or it's time to get rid > of them. Hi all, Sorry for not replying sooner, I was on vacation when this thread started and I only got back in town yesterday. To bring my $0.02 to this discussion: the Pybots 'community buildbots' turned out largely to be a failure. Why? Because there was never really a 'community' around it, especially a community of project leaders who would be interested in the state of their projects' tests. All the machines donated for the Pybots farm belong to people who just happen to be interested in given projects, but are not really the leaders of those projects. The only project who constantly stayed on top of the buildbot status was Twisted, represented by JP Calderone (although even there the tests were running on my machine, and not on a machine contributed by the Twisted folks.) I still haven't given up, and I hope this thread will spur project leaders into donating time, or resources, to the Pybots project. It has been my bitter observation about the Open Source world that people just LOVE to get stuff for free. As soon as you mention more involvement from them in the form of time, money, hardware resources, etc., the same brave proponents of cool things to be done are nowhere to be found. To come back to this thread, I don't think it's reasonable to expect the Python core developers to be that interested in the status of the community buildbots. It is again up to the project leaders to step up to the plate, donate machines to Pybots, and stay on top of any breakages that result from Python core checkins. It seems to me that the Python core developers have always responded promptly and favorably to reports of breakages coming from the Pybots farm. Grig ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 01:02 am, [EMAIL PROTECTED] wrote: To bring my $0.02 to this discussion: the Pybots 'community buildbots' turned out largely to be a failure. Let's not say it's a failure. Let's instead say that it hasn't yet become a success :-). I still haven't given up, and I hope this thread will spur project leaders into donating time, or resources, to the Pybots project. It has been my bitter observation about the Open Source world that people just LOVE to get stuff for free. As soon as you mention more involvement from them in the form of time, money, hardware resources, etc., the same brave proponents of cool things to be done are nowhere to be found. I think this list is the wrong place to go to reach the people who need to be reached. It's python core developers and other people already involved in and aware of core development. That said I'm not sure what the *right* place is; I think your blog is syndicated on the unofficial planet python, so maybe that's a good place to start. Sadly, the right thing to do in terms of drumming up support is to get someone interested in PR and have them go to each project individually, but that might be more effort than setting up the buildbots themselves, at least initially... However, let's say that this were tremendously successful, and lots of people start paying attention. I think pybots.org needs to be updated to say exactly what a participant interested in python testing needs to do, beyond "here's how you set up a buildbot" (a page that is actually a daunting-looking blog post which admits it may be somewhat outdated), because setting up a buildbot might not be the only thing that the project needs. It's one thing to tell people that they need to be helping out (and I'm sure you're right) but it's much more useful to get the message out that "we really need people to do X, Y, and Z". One thing I will definitely commit to is that if you make a "cry for help" page, I'll blog about it to drive attention to it, and I'll encourage the other, perhaps better-read Python bloggers I know to do so as well. My personal interest at the moment is to get all of the irrelevant red off of the community builders page. Whether or not you believe in an XP "green bar" philosophy, the large number of spurious failures is distracting. Who is it that is capable of making appropriate changes? Is there something I could do to help with that? Note that I'm committing to say that I can do *that*, but, at least you could shut me up by making it my fault ;-). (I'd also like to improve the labels of the build slaves. What exactly is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) It would be good to remove the perception that it's somebody else's problem as much as possible. Right now, all these dead buildbots suggest to the various communities, "oh, I guess that guy who runs that buildbot needs to fix it". The dead bots should just be killed off, and their projects removed from the list, so that if someone wants to get involved and set up a bot for lxml, they're not put off of it by the fact that it might be rude to the guy who is currently (allegedly) running it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] writes: > On 01:02 am, [EMAIL PROTECTED] wrote: > >To bring my $0.02 to this discussion: the Pybots 'community buildbots' > >turned out largely to be a failure. > > Let's not say it's a failure. Let's instead say that it hasn't yet > become a success :-). +1 > >I still haven't given up, and I hope this thread will spur project > >leaders into donating time, or resources, to the Pybots project. > I think this list is the wrong place to go to reach the people who need > to be reached. It's python core developers and other people already > involved in and aware of core development. That said I'm not sure what > the *right* place is; I think your blog is syndicated on the unofficial > planet python, so maybe that's a good place to start. Sadly, the right > thing to do in terms of drumming up support is to get someone interested > in PR and have them go to each project individually, but that might be > more effort than setting up the buildbots themselves, at least > initially... Exactly, and that's why nobody should be "bitter" about it. The problem is that the while overall the effort and rewards look to be balanced in favor of the rewards, the startup costs for individuals are quite high. I think this *is* the place to start, though. The project leaders "should" be, and probably generally are, "here". They have the strongest interest in any individual 'bot, while Guido is quite correct in saying python-dev can't afford to have strong interest in all the bots. > However, let's say that this were tremendously successful, and lots of > people start paying attention. I think pybots.org needs to be updated > to say exactly what a participant interested in python testing needs to > do, beyond "here's how you set up a buildbot" (a page that is actually a > daunting-looking blog post which admits it may be somewhat outdated), > because setting up a buildbot might not be the only thing that the > project needs. It's one thing to tell people that they need to be > helping out (and I'm sure you're right) but it's much more useful to get > the message out that "we really need people to do X, Y, and Z". One > thing I will definitely commit to is that if you make a "cry for help" > page, I'll blog about it to drive attention to it, and I'll encourage > the other, perhaps better-read Python bloggers I know to do so as > well. Two suggestions in this vein: First, I think it's established that some but not all "red community bots" *are* of interest to Python core development. While I'm not aware of the technical details, I estimate that triaging the community 'bot failures is probably similar to reviewing bugs in the Python tracker. Perhaps Martin van Loewis and others who have offered the 5-for-1 review deal would be willing to extend the definition of "review" to include initial bug reports based on a red community bot (ie, you review the community 'bot failure and decide it is something that should concern Python core development). Perhaps that's not appropriate, but a similar system could be set up. Second, something intermediate between the occasional half-hour of triaging bugs and a full-blown PR campaign at the projects would be documenting the criteria for reporting a failure on a community 'bot to the Python tracker as a bug, etc. This would also serve as a basis for talking to project lurker who might have the odd half-hour to do some "red bot" triaging. (By criteria I mean the kinds of things that Python core considers necessary breakage in new versions that downstream must address in their own code, vs. regressions in a x.y.z patchlevel, etc. The kind of thing that glyph and Guido were discussing earlier.) > It would be good to remove the perception that it's somebody else's > problem as much as possible. To the extent that a 'bot is running prerelease project against prerelease Python, this is probably not very doable. If Python is stable and the project version is prerelease, it's the project's bug until proven otherwise, and vice versa. If both are stable, again some expertise is probably needed for triage. I guess that means that one important task is to classfy the bots in a two-by-two matrix according to stability of project and Python respectively. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> (I'd also like to improve the labels of the build slaves. What exactly > is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) It seems like you would like to edit the master configuration file. That can be arranged fairly easily. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Sun, Jul 6, 2008 at 8:46 AM, <[EMAIL PROTECTED]> wrote: > > However, let's say that this were tremendously successful, and lots of > people start paying attention. I think pybots.org needs to be updated to > say exactly what a participant interested in python testing needs to do, > beyond "here's how you set up a buildbot" (a page that is actually a > daunting-looking blog post which admits it may be somewhat outdated), > because setting up a buildbot might not be the only thing that the project > needs. It's one thing to tell people that they need to be helping out (and > I'm sure you're right) but it's much more useful to get the message out that > "we really need people to do X, Y, and Z". One thing I will definitely > commit to is that if you make a "cry for help" page, I'll blog about it to > drive attention to it, and I'll encourage the other, perhaps better-read > Python bloggers I know to do so as well. I have posted 'cries for help' repeatedly on my blog, with generally little success. See http://agiletesting.blogspot.com/search?q=pybots . But I will post again. If you can include here a paragraph of what your vision of the 'X, Y and Z' above is, that'd help too. I think I've been pretty clear about the benefits that the Pybots farm can bring to a given project, so all project leaders on this list should be aware of them IMO. If not, I'd be happy to rehash them. But the home page of pybots.org is pretty self-explanatory I think. > > My personal interest at the moment is to get all of the irrelevant red off > of the community builders page. Whether or not you believe in an XP "green > bar" philosophy, the large number of spurious failures is distracting. Who > is it that is capable of making appropriate changes? Is there something I > could do to help with that? Note that I'm committing to say that I can do > *that*, but, at least you could shut me up by making it my fault ;-). > I'll send a message to the pybots mailing list asking people whose buildbots are turned off if they're still interested in running them. Negative or no answers will mean we can remove them from the farm. > (I'd also like to improve the labels of the build slaves. What exactly is > "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) > It's not only a question of changing a static label in this case. A given buildslave can run the tests for multiple projects, in which case it becomes tricky to change the label on the fly accordingly. As an aside, the slave you mention was running on my machine, and I used it to run the Twisted tests, but I shut it down a while ago because the buildbot process was taking too many resources. If the Twisted project can donate a machine, I'd be happy to include it in the Pybots farm ASAP. > It would be good to remove the perception that it's somebody else's problem > as much as possible. Right now, all these dead buildbots suggest to the > various communities, "oh, I guess that guy who runs that buildbot needs to > fix it". The dead bots should just be killed off, and their projects > removed from the list, so that if someone wants to get involved and set up a > bot for lxml, they're not put off of it by the fact that it might be rude to > the guy who is currently (allegedly) running it. As I said, I'll see what the current owners have to say, and then I'll report back to this list. Thanks for offering your help! Grig ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> It's not only a question of changing a static label in this case. A > given buildslave can run the tests for multiple projects, in which > case it becomes tricky to change the label on the fly accordingly. I think you could set up different builders for a single slave in that case (use a slave lock to make them run sequentially). > As > an aside, the slave you mention was running on my machine, and I used > it to run the Twisted tests, but I shut it down a while ago because > the buildbot process was taking too many resources. If the Twisted > project can donate a machine, I'd be happy to include it in the Pybots > farm ASAP. Please talk to Trent Nelson. He has a Windows machine that he donated precisely for that kind of activity. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Sun, Jul 6, 2008 at 2:09 PM, Grig Gheorghiu <[EMAIL PROTECTED]> wrote: > I'll send a message to the pybots mailing list asking people whose > buildbots are turned off if they're still interested in running them. > Negative or no answers will mean we can remove them from the farm. > OK, I posted a message to the pybots mailing list and I removed 2 slaves. Out of the 6 remaining, 4 are currently active, and one will hopefully soon be active starting next week. This leaves just one unanswered for so far. I also got an email from another person volunteering a buildslave, so we'll soon have 7 machines. As I said, if anybody else wants to participate in the Pybots project, please let me know! I'll also post a blog entry on this soon. Grig ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 6 Jul, 05:29 pm, [EMAIL PROTECTED] wrote: (I'd also like to improve the labels of the build slaves. What exactly is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) It seems like you would like to edit the master configuration file. That can be arranged fairly easily. How shall we arrange it, then? :) Whoever is interested, I've got a recent SSH key on https://launchpad.net/~glyph/+sshkeys ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 6 Jul, 09:09 pm, [EMAIL PROTECTED] wrote: On Sun, Jul 6, 2008 at 8:46 AM, <[EMAIL PROTECTED]> wrote: It's one thing to tell people that they need to be helping out (and I'm sure you're right) but it's much more useful to get the message out that "we really need people to do X, Y, and Z". I have posted 'cries for help' repeatedly on my blog, with generally little success. See http://agiletesting.blogspot.com/search?q=pybots . But I will post again. If you can include here a paragraph of what your vision of the 'X, Y and Z' above is, that'd help too It looks to me like the main thing that Pybots needs is help with maintenance. Getting someone to set up an individual buildbot is one thing, but keeping it working is the important bit and it looks like people are not doing that. The project would also be greatly aided by having dedicated people diagnose errors, report bugs against Python core if they're real and report bugs against Pybots if they're spurious. It would be good to have this effort be centralized and directed because it would otherwise be too easy to file duplicate bug reports, or to assume "oh, this has been failing for months, someone must have filed a bug already". It's not only a question of changing a static label in this case. A given buildslave can run the tests for multiple projects, in which case it becomes tricky to change the label on the fly accordingly. I'm a bit confused about how the projects being tested changes on the fly... but then, this level of specifics is probably best left to the pybots mailing list. Hopefully sometime soon I'll have the time to add yet another subscription. Thanks for cleaning up the buildbots though! I can see a lot more tests actually running now :). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Tue, Jul 8, 2008 at 12:56 PM, <[EMAIL PROTECTED]> wrote: > It looks to me like the main thing that Pybots needs is help with > maintenance. Getting someone to set up an individual buildbot is one thing, > but keeping it working is the important bit and it looks like people are not > doing that. The project would also be greatly aided by having dedicated > people diagnose errors, report bugs against Python core if they're real and > report bugs against Pybots if they're spurious. > > It would be good to have this effort be centralized and directed because it > would otherwise be too easy to file duplicate bug reports, or to assume "oh, > this has been failing for months, someone must have filed a bug already". I agree with all you're saying here. As usual, the devil is in the details. Finding those 'dedicated people' and also people who would act as the central point of contact for bug reports etc. turns out to be very hard in practice. If you have any ideas, I'd be glad to hear them. Grig ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
