Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Nick Coghlan]

On Fri, Mar 25, 2011 at 11:29 PM, Antoine Pitrou  wrote:
>> My model for the suggestion is the context objects in the decimal
>> module. They offer a constrained way to affect the way the entire
>> decimal module goes about its business, and through judicious use of
>> thread local storage and context managers, allow this to be done
>> without distorting the public API of the decimal objects themselves.
>
> Making an API TLS-based means your code breaks mysteriously if you
> start offloading tasks to separate threads (which is quite common with
> network-related tasks).

Ah, true - and, as you say, libraries are far more likely to farm
networking operations out to threads than they are decimal
calculations. Such situations also cause problems for any approaches
based on temporary monkey-patching.

Still, exploring such ideas and their downsides would be one of the
purposes of writing a PEP on this topic. Sprinkling factory function
pixie dust everywhere may turn out to be the right thing to do, but
the various options should be explored before settling specifically on
that one.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Antoine Pitrou]

On Fri, 25 Mar 2011 21:10:08 +1000
Nick Coghlan  wrote:
> On Fri, Mar 25, 2011 at 12:22 PM, Glenn Linderman  
> wrote:
> > On 3/24/2011 4:25 PM, Nick Coghlan wrote:
> >
> > As an example of the last point, perhaps rather than modifying all the
> > *clients* of the socket module, it may make more sense to have tools
> > in the socket module itself to temporarily customise the socket
> > creation process in the current thread. The advantage of such an
> > approach is that it would then work for 3rd party libraries that
> > create sockets, without requiring any further modification.
> >
> > Would be easier to implement that way, not requiring changes to every client
> > of the socket library, but in some circles that would be called "action at a
> > distance", and harder to understand.
> 
> Oh, it is definitely action at a distance, and quite deliberately so.
> 
> My model for the suggestion is the context objects in the decimal
> module. They offer a constrained way to affect the way the entire
> decimal module goes about its business, and through judicious use of
> thread local storage and context managers, allow this to be done
> without distorting the public API of the decimal objects themselves.

Making an API TLS-based means your code breaks mysteriously if you
start offloading tasks to separate threads (which is quite common with
network-related tasks).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Nick Coghlan]

On Fri, Mar 25, 2011 at 12:22 PM, Glenn Linderman  wrote:
> On 3/24/2011 4:25 PM, Nick Coghlan wrote:
>
> As an example of the last point, perhaps rather than modifying all the
> *clients* of the socket module, it may make more sense to have tools
> in the socket module itself to temporarily customise the socket
> creation process in the current thread. The advantage of such an
> approach is that it would then work for 3rd party libraries that
> create sockets, without requiring any further modification.
>
> Would be easier to implement that way, not requiring changes to every client
> of the socket library, but in some circles that would be called "action at a
> distance", and harder to understand.

Oh, it is definitely action at a distance, and quite deliberately so.

My model for the suggestion is the context objects in the decimal
module. They offer a constrained way to affect the way the entire
decimal module goes about its business, and through judicious use of
thread local storage and context managers, allow this to be done
without distorting the public API of the decimal objects themselves.

There's no reason a solution along those lines wouldn't work for the
socket module, or any other API that would similarly benefit from
providing a mechanism for applications to indirectly control library
behaviour without resorting to monkey-patching.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Xavier Morel]

On 2011-03-25, at 10:22 , Simon Cross wrote:
> On Fri, Mar 25, 2011 at 1:25 AM, Nick Coghlan  wrote:
>> As an example of the last point, perhaps rather than modifying all the
>> *clients* of the socket module, it may make more sense to have tools
>> in the socket module itself to temporarily customise the socket
>> creation process in the current thread. The advantage of such an
>> approach is that it would then work for 3rd party libraries that
>> create sockets, without requiring any further modification.
> 
> -2.  That wouldn't allow one to use normal sockets in one 3rd party
> library and special sockets in another 3rd party library.
> 
> Schiavo
> Simon

Or even in "first-party" code (in the stdlib) to set different timeouts on 
different APIs (say, an xmlrpclib.ServerProxy and an IMAP4 client).

For instance, currently as far as I can tell setting a socket timeout on an 
xmlrpclib.ServerProxy without setting a global timeout involves:

* subclassing all of xmlrpclib.Serverproxy, xmlrpclib.Transport, httplib.HTTP 
and httplib.HTTPConnection
* overloading __init__ on the ServerProxy subclass (and on Transport if the 
timeout is to be a parameter)
* overloading make_connection on the Transport subclass (in order to use the 
HTTP subclass and propagate the timeout)
* overloading _connection_class on the HTTP subclass
* overloading connect on the HTTPConnection class

This *could* be solved by wrapping a socket-related thread-local context 
manager around each call resulting in the creation of a socket, but these call 
sites may not even be under control.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Nick Coghlan]

On Fri, Mar 25, 2011 at 7:22 PM, Simon Cross
 wrote:
> On Fri, Mar 25, 2011 at 1:25 AM, Nick Coghlan  wrote:
>> As an example of the last point, perhaps rather than modifying all the
>> *clients* of the socket module, it may make more sense to have tools
>> in the socket module itself to temporarily customise the socket
>> creation process in the current thread. The advantage of such an
>> approach is that it would then work for 3rd party libraries that
>> create sockets, without requiring any further modification.
>
> -2.  That wouldn't allow one to use normal sockets in one 3rd party
> library and special sockets in another 3rd party library.

Uh, yes it would. One possible implementation is to use exactly the
same techniques that are used to implement contexts in the decimal
module.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Simon Cross]

On Fri, Mar 25, 2011 at 1:25 AM, Nick Coghlan  wrote:
> As an example of the last point, perhaps rather than modifying all the
> *clients* of the socket module, it may make more sense to have tools
> in the socket module itself to temporarily customise the socket
> creation process in the current thread. The advantage of such an
> approach is that it would then work for 3rd party libraries that
> create sockets, without requiring any further modification.

-2.  That wouldn't allow one to use normal sockets in one 3rd party
library and special sockets in another 3rd party library.

Schiavo
Simon
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Glenn Linderman]

On 3/24/2011 4:25 PM, Nick Coghlan wrote:

As an example of the last point, perhaps rather than modifying all the
*clients*  of the socket module, it may make more sense to have tools
in the socket module itself to temporarily customise the socket
creation process in the current thread. The advantage of such an
approach is that it would then work for 3rd party libraries that
create sockets, without requiring any further modification.


Would be easier to implement that way, not requiring changes to every 
client of the socket library, but in some circles that would be called 
"action at a distance", and harder to understand.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Eugene Toder]

> 2. The level at which the dependency injection works (function
> arguments, class attributes, module globals) needs to be decided

+1. The scope of parameter needs to be specified. If the scope is
global, this can be achieved pretty easily -- declare some of the
imports in particular modules to be a part of the public APIs of that
modules. Being part of the public API means allowing to replace that
module with another one with compatible implementations. In other
words, bless monkey-patching of those imports.
The advantages of this is simplicity and no uglification of the code
(no changes even). However, this should really not be used for more
than just testing. For other purposes the scope should be smaller
(e.g. class) and Antoine's approach looks like the right thing.

Eugene
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Nick Coghlan]

On Fri, Mar 25, 2011 at 2:40 AM, Jesus Cea  wrote:
> What do you think?. Should I write a PEP? (I don't think so, but your
> opinion matters). I care, for instance, about how to garantee the API
> coverage actually needed for the new "socket-like" object. The idea is
> that your object should be "socket-like", but how much "socket-like" in
> the API sense?. Do you need to implement "getpeername()" for injecting
> in smtplib?.
>
> If you agree that this is a nice idea, what other libs do you think that
> could benefice of being "injected", beside "socket"?.

1. A PEP is definitely needed to thrash out how this API should work
in practice, as well as whether or not it should even be done at all

2. The level at which the dependency injection works (function
arguments, class attributes, module globals) needs to be decided

3. Alternative designs (such as standardised tools for thread-safe
monkey patching or other customisation of affected libraries) should
be considered

As an example of the last point, perhaps rather than modifying all the
*clients* of the socket module, it may make more sense to have tools
in the socket module itself to temporarily customise the socket
creation process in the current thread. The advantage of such an
approach is that it would then work for 3rd party libraries that
create sockets, without requiring any further modification.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Michael Foord]

On 24/03/2011 17:46, Andrew McNabb wrote:

On Thu, Mar 24, 2011 at 10:12:18AM -0700, Guido van Rossum wrote:

On Thu, Mar 24, 2011 at 9:46 AM, Benjamin Peterson  wrote:

I want to test the dev community interest in modifying the stdlib to
ease dependence injection.

I, for one, am -1. Code shouldn't be uglified for the purposes of
testing.

Well, the philosophy of dependency injection requires exactly that.
Personally I am on the fence; I've seen is used effectively but I've
also seen code, indeed, uglified with little benefits. (Same for other
extreme testing practices like TDD.)

I agree with the comments about uglification, but testing was only one
of the use cases that Jesus provided.  He also mentioned using
alternative network implementations, integrating with event loops, etc.
Another use case that I've run into before is setting socket options.


And some of the use cases for this seemed pretty good from the email 
thread he linked to.


Michael


--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Andrew McNabb]

On Thu, Mar 24, 2011 at 10:12:18AM -0700, Guido van Rossum wrote:
> On Thu, Mar 24, 2011 at 9:46 AM, Benjamin Peterson  
> wrote:
> >> I want to test the dev community interest in modifying the stdlib to
> >> ease dependence injection.
> >
> > I, for one, am -1. Code shouldn't be uglified for the purposes of
> > testing.
> 
> Well, the philosophy of dependency injection requires exactly that.
> Personally I am on the fence; I've seen is used effectively but I've
> also seen code, indeed, uglified with little benefits. (Same for other
> extreme testing practices like TDD.)

I agree with the comments about uglification, but testing was only one
of the use cases that Jesus provided.  He also mentioned using
alternative network implementations, integrating with event loops, etc.
Another use case that I've run into before is setting socket options.

--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Guido van Rossum]

On Thu, Mar 24, 2011 at 9:46 AM, Benjamin Peterson  wrote:
>> I want to test the dev community interest in modifying the stdlib to
>> ease dependence injection.
>
> I, for one, am -1. Code shouldn't be uglified for the purposes of
> testing.

Well, the philosophy of dependency injection requires exactly that.
Personally I am on the fence; I've seen is used effectively but I've
also seen code, indeed, uglified with little benefits. (Same for other
extreme testing practices like TDD.)

> It's also a slippery slope. Maybe we should add parameters
> for open() and io functions? What about sys and os? Those often need
> to be mocked.

There are existing solutions that work pretty well, so I'm thinking
there is no big problem that this solved.

But my main concern is about anything that proposed to make sweeping
changes to a large number of stdlib modules. This has almost always
resulted in reduced quality.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Antoine Pitrou]

On Thu, 24 Mar 2011 11:46:42 -0500
Benjamin Peterson  wrote:

> 2011/3/24 Jesus Cea :
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
> >
> > Hi, everybody.
> >
> > I want to test the dev community interest in modifying the stdlib to
> > ease dependence injection.
> 
> I, for one, am -1. Code shouldn't be uglified for the purposes of
> testing. It's also a slippery slope. Maybe we should add parameters
> for open() and io functions? What about sys and os? Those often need
> to be mocked.

A non-ugly way of doing "dependency injection" is to rely on
class/instance attributes instead. e.g.:

class FTP:
socket_factory = socket.create_connection

def open(self, host, port):
self.sock = self.socket_factory(host, port)
...


Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[James Y Knight]

On Mar 24, 2011, at 12:40 PM, Jesus Cea wrote:
> I want to test the dev community interest in modifying the stdlib to
> ease dependence injection.
> 
> The seminal idea was in
> .
> 
> A lot of stdlib modules use, deep inside, other modules to do their
> work. The most obvious case would be the socket library, used by ANY
> module involving the network.

Do you know about exocet (described by the last few blog entries on 
http://washort.twistedmatrix.com/search/label/exocet), a newly developed tool 
for doing dependency injection on arbitrary modules? It seems like it would be 
very well suited for the kinds of things you want to do.

James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for Python 3.3: dependence injection

[Benjamin Peterson]

2011/3/24 Jesus Cea :
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi, everybody.
>
> I want to test the dev community interest in modifying the stdlib to
> ease dependence injection.

I, for one, am -1. Code shouldn't be uglified for the purposes of
testing. It's also a slippery slope. Maybe we should add parameters
for open() and io functions? What about sys and os? Those often need
to be mocked.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Proposal for Python 3.3: dependence injection

[Jesus Cea]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi, everybody.

I want to test the dev community interest in modifying the stdlib to
ease dependence injection.

The seminal idea was in
.

A lot of stdlib modules use, deep inside, other modules to do their
work. The most obvious case would be the socket library, used by ANY
module involving the network.

This internal usage, deep inside, prevents programmers to "mess" with
the networking layer in an easy way. Some examples: transparent
encryption/compression, using alternative network implementations like
shared buffers if the traffic is local, integrating the socket in a
async event loop/greenlet/etc, network logging/sniffing, testing
(simulating a server, with no real network connection), etc. Testing
gets very easy if we can "mock" the internal socket, for instance.

So I propose to add an additional OPTIONAL parameter to quite a few
libraries. If the parameter is not present (compatible with client
current code), the library would instantiate a internal socket, just
like now. But if the parameter is present, it would inject a constructor
for the library to use instead of "socket".

Traditionally this "features" can be used in Python code using monkey
patching, but this approach is fragile (can break when upgrading), not
obvious for newbies and difficult to combine (for instance, trying to
apply multiple "monkey patches" over the socket module).

What do you think?. Should I write a PEP? (I don't think so, but your
opinion matters). I care, for instance, about how to garantee the API
coverage actually needed for the new "socket-like" object. The idea is
that your object should be "socket-like", but how much "socket-like" in
the API sense?. Do you need to implement "getpeername()" for injecting
in smtplib?.

If you agree that this is a nice idea, what other libs do you think that
could benefice of being "injected", beside "socket"?.

- -- 
Jesus Cea Avion _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
jabber / xmpp:j...@jabber.org _/_/_/_/  _/_/_/_/_/
.  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTYtz8plgi5GaxT1NAQJCzwQAkvtCwvKDh1QcQ+Qfbhe0qrKcQdI4wMfB
g4jpCmCX3UCkTvAjU1AsnxGGwVjvYZWdL2QVZS5+sJOP5GOmMzm96PBYjWZjVovk
YHIP+HFXU0BtbXlRJMV2+bjrwZf1g8CQFngyxqkQ69WdU+FMyxmDhFbcQ7y667HT
ldkINMLSuTk=
=rrkE
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com