Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Nick Coghlan 

On 26/05/10 13:51, Stephen J. Turnbull wrote:

People have been asking "what's special about this module, to violate
the BCP principle?"  There's nothing special about the fact that
several people would use a "robust and debugged" futures module if it
were in the stdlib.  That's true of *every* module that is worth a
PEP.


I actually wrote a reply to that question earlier in the week, but 
failed at using gmail's web interface correctly and only sent it to 
Steve Holden.


===
The trick with futures and executor pools is that they're a *better* way 
of programming with threads in many cases.


However, given the choices of:
- hack together something simple with some worker Threads and a Queue 
(or two)

- write my own futures and executor infrastructure
- download a futures module from PyPI and live with the additional 
dependency


I'll choose the first option every time, and my programs will be the 
worse for it.


Put the capability to use futures and an executor into the stdlib, and 
it becomes something I can reach for without having to worry about 
additional dependencies beyond specifying a minimal Python version. It 
provides a higher level API that can be more readily switched between 
threading and multiprocessing back ends. It becomes something that can 
be taught as a standard Python technique for enabling concurrency in 
imperative code.


This is something that is irrelevant to me as a module on PyPI, but has 
the potential to significantly affect my programming in the future as a 
standard library module. Even in the near term, backports of future 
standard library modules are often perceived differently when being 
discussed as potential additional dependencies for an application (i.e. 
I believe it would be worthwhile for a backport of the module to earlier 
Python versions than 3.2 to be made available on PyPI).

===

Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Stephen J. Turnbull 
Nick Coghlan writes:

 > Those that say "just put it on PyPI"

Nobody is saying that, AFAICS.  Nobody is saying that *some* futures
module shouldn't *eventually* go into the stdlib.

The question is whether *this* futures module should go into the
stdlib *now*.  And it is clear that more time on PyPI would provide
valuable information.  This is a general principle that has served us
well: put best current practice backed up by actual widespread usage
in the stdlib, not theoretical wins based on the developer's
experience.  PyPI is a way to broaden usage to determine BCP, not an
end in itself.

People have been asking "what's special about this module, to violate
the BCP principle?"  There's nothing special about the fact that
several people would use a "robust and debugged" futures module if it
were in the stdlib.  That's true of *every* module that is worth a
PEP.  But remember, in the case of ipaddr it was the people who wanted
some such module badly who were also the most vocal opponents, because
they could see offhand that it was going to serve their use cases
badly.  (It turned out that this was equally trivial to fix despite a
week of hot debate, and everyone lived happily ever after.  But that
was smiling Luck, not inexorable Fate.)  For this module, three people
have said "I 'would have' used it if it were available," but none of
you has announced that you've started refactoring and the PEP 3148 API
meets all expectations.  I call that "damning with faint praise".

OTOH, Glyph has changed from "why not more time on PyPI?" to "let's
see if we can improve this a bit, then let's do it".  He has published
code (showing how to turn futures into Twisted Deferreds), and argues
that based on download stats to date and the nature of the use cases
it would take a lot of time on PyPI to demonstrate a BCP.  Those are
good arguments for an exception, IMHO.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Nick Coghlan 

On 24/05/10 20:46, Stephen J. Turnbull wrote:

Cameron Simpson writes:

  >  There's a lot to be said for a robust implementation of a well defined
  >  problem. Brian's module, had it been present and presuming it robust and
  >  debugged, would have been quite welcome.

That, of course, is the consensus view, both in general and with
respect to this particular module.

The difference is over what constitutes sufficient evidence for your
presumption of "robust and debugged" from the point of view of the
users of the stdlib.


At the very least, we'll be offering a promise to be "more robust and 
more debugged than what you came up with in that coding marathon last 
night" ;)


Having a decent test suite that is regularly executed on multiple 
platforms (which will be the case for any accepted module by the time it 
is included in a Python release) also places anything we release a cut 
above a *lot* of in-house code.


Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Nick Coghlan 

On 26/05/10 12:29, Lennart Regebro wrote:

On Wed, May 26, 2010 at 02:10, Nick Coghlan  wrote:

Those that say "just put it on PyPI" may not recognise the additional ...


Just a note, so we don't get sidelined by misunderstandings: I don't
think anybody said that. ;-)


Nah, that pseudo-quote wasn't from this discussion in particular. It's a 
reference to the ongoing tension between the "batteries included" 
advocates and the "make the standard library as streamlined as possible" 
crowd. Both sides have valid points, so the "included battery" vs 
"optional download" question needs to be decided on a case-by-case basis.



There are two issues here, one generic and one specific:

Generic: Modules should go on PyPI first, for a time, to stabilize
(and so they can be used in earlier versions of Python) before they
end up in stdlib. I suspect everyone actually agrees on that (but I
could be wrong).


That's the point I'm disagreeing with. For most modules it makes sense 
to do things that way, but for some low-level infrastructure elements, 
it is going to be less effective (because people will quickly throw 
together their own solutions instead of adding a new dependency for 
something "simple").


Other times we'll invent a new module because *we* need it for something 
(e.g. runpy).


Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Lennart Regebro 
On Wed, May 26, 2010 at 02:10, Nick Coghlan  wrote:
> Those that say "just put it on PyPI" may not recognise the additional ...

Just a note, so we don't get sidelined by misunderstandings: I don't
think anybody said that. ;-)

There are two issues here, one generic and one specific:

Generic: Modules should go on PyPI first, for a time, to stabilize
(and so they can be used in earlier versions of Python) before they
end up in stdlib. I suspect everyone actually agrees on that (but I
could be wrong).

Specific:Has futures been long enough on PyPI, and is it stable? I'm
staying out of that discussion. :-)

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Jesse Noller 
On Tue, May 25, 2010 at 7:54 PM, Nick Coghlan  wrote:
> On 23/05/10 22:47, Antoine Pitrou wrote:
>>
>> On Sun, 23 May 2010 08:34:22 -0400
>> Jesse Noller  wrote:
>>>
>>> Brian has already agreed to name spacing it to "concurrent.futures" -
>>> this means it will be a small part to a much larger concurrent.*
>>> implementation ala Java.
>>
>> What I would question here is what other things will be part
>> of the "concurrent" package, and who will implement them. Are there
>> plans for that? (or even tracker issues open?)
>
> I'm not sure it is called out explicitly in the PEP, but the specific
> example that came up in the previous discussions was something like
> "concurrent.pool" to hold a thread vs process agnostic worker pool interface
> based on the existing Pool interface in multiprocessing (with concrete
> implementations for both threading and multiprocessing).
>

Nick is correct - there's plenty of things in multiprocessing which
belong in a more abstract package as they're useful for more things
than just multiprocessing. I don't think they need to be called out as
part of the PEP though.

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documenting [C]Python's Internals

2010-05-25 Thread Nick Coghlan 

On 20/05/10 08:13, Yaniv Aknin wrote:

Hi,

I wanted to let python-dev know about a series of articles about
CPython's internals I'm publishing under the collective title "Guido's
Python"* (http://tech.blog.aknin.name/tag/guidos-python/).


A resource that may be useful to you is a 2.5 focused manuscript I put 
together a few years ago trying to bridge the gap between the library 
reference and the language reference:

http://svn.python.org/projects/sandbox/trunk/userref/

It's obviously a little dated in some areas and doesn't delve as deeply 
into the source code as you apparently plan to, but hopefully it may 
prove useful as a resource (I still have vague intentions of exporting 
that document to ReST markup and updating it to current Python, but that 
doesn't look like actually happening any time soon)


Cheers,
Nick.

P.S. For the record, the relevant URL is now 
http://tech.blog.aknin.name/tag/pythons-innards/



--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dead modules

2010-05-25 Thread Nick Coghlan 
(Sending again - I didn't mean to drop python-dev from the cc list when 
I originally sent this via the gmail web interface)


On Sun, May 23, 2010 at 9:00 PM, Dirkjan Ochtman > wrote:


   Right, it wasn't intended as that harsh... but it does come with a
   rather impressive set of constraints in terms of what you can do with
   the API.

True, but in some cases (especially low level infrastructure), it is 
worth accepting those constraints in order to achieve other aims (such 
as standardisation of techniques). Things like itertools, collections, 
functools, unittest owe their existence largely to the choice of gains 
in standardisation over flexibility of API updates.
Besides, popular PyPI modules don't have that much more freedom than the 
stdlib when it comes to API changes. The only real difference is that 
the 18-24 month release cycle for the stdlib is a lot *slower* than that 
of many PyPI packages, so feedback on any changes we make is 
correspondingly delayed. Hence the existence of projects like distutils2 
and unittest2 to enable that faster feedback cycle to inform the updates 
passed back into the more slowly evolving stdlib modules, as well as the 
desire to copy prior art wherever it makes sense to do so (whether that 
is other languages, existing PyPI modules or the internal code bases of 
large corporate contributors).


Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Nick Coghlan 

On 23/05/10 21:56, Lennart Regebro wrote:

On Sun, May 23, 2010 at 13:29, Brian Quinlan  wrote:

Parts of it, yes. Just like I can replace most operations in os.path and
urlparse with a few lines of code.


Yeah, but "parts of" is not the question. I've read the PEP, and I do
*not* know how to implement it. That means it's not a trivial module,
so that argument doesn't hold up here, even if we accept it as valid
(which I actually don't). I don't think any module in the stdlib is
entirely trivial. Yes, even parsing an URL is non-trivial, as shown by
the fact that the urlparse module apparently has a bug in it for urls
like svn+ssh://foo.bar/frotz. ;-)


In this case, the "trivial" refers to being able to write something that 
will get the job done for a specific task or application, but that isn't 
as solid from a reliability/maintainability/portability/scalability 
point of view.


By providing solid infrastructure in the standard library, we can remove 
that choice between "do it fast" vs "do it right", by providing 
ready-made robust infrastructure.


Those that say "just put it on PyPI" may not recognise the additional 
overhead that can be involved in identifying, obtaining approval to use 
and ongoing management of additional dependencies in a corporate 
environment that is actually performing appropriate due diligence in 
regards to IP licensing. This overhead can be especially significant 
(and, depending on licence and contract details, even a dealbreaker) for 
companies with specific IP licensing provisions in their contracts with 
their clients. It doesn't matter *how* easy we make it to download PyPI 
packages, we can't do anything about such company IP management policies 
(except for making it easier for programmers to violate them 
thoughtlessly, of course).


To use myself as an example, I have several utilities that I probably 
would have written differently if the futures module had been available 
in the standard library at the time I wrote them. As it is, they work 
well enough, but their usage of the threading module is fairly ad hoc 
(and migrating them to use multiprocessing would be a fairly complex 
task, and making that decision run-time selectable even more complicated).


In the near-term, backports of future standard library modules are much 
easier to get through a corporate review process as the licensing is 
typically similar to the PSF license (or is even the PSF license itself) 
and the modules come with a clear roadmap for eliminating the dependency 
(i.e. once the baseline Python version employed by the company includes 
the module in the standard library, the external dependency is no longer 
needed).


Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3148 ready for pronouncement

2010-05-25 Thread Nick Coghlan 

On 23/05/10 22:47, Antoine Pitrou wrote:

On Sun, 23 May 2010 08:34:22 -0400
Jesse Noller  wrote:


Brian has already agreed to name spacing it to "concurrent.futures" -
this means it will be a small part to a much larger concurrent.*
implementation ala Java.


What I would question here is what other things will be part
of the "concurrent" package, and who will implement them. Are there
plans for that? (or even tracker issues open?)


I'm not sure it is called out explicitly in the PEP, but the specific 
example that came up in the previous discussions was something like 
"concurrent.pool" to hold a thread vs process agnostic worker pool 
interface based on the existing Pool interface in multiprocessing (with 
concrete implementations for both threading and multiprocessing).


Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com