Re: [Python-Dev] Breaking undocumented API

[James Y Knight]

On Nov 10, 2010, at 8:47 AM, Michael Foord wrote:
> How about making this explicit (either pep 8 or our developer docs):
> 
> If a module or package defines __all__ that authoritatively defines the 
> public interface. Modules with __all__ SHOULD still respect the naming 
> conventions (leading underscore for private members) to avoid confusing 
> users. Modules SHOULD NOT export private members in __all__.

I don't like the idea of the authoritative definition of a public interface 
being defined based on __all__, because that provides users almost no warning 
that they're using a private API: the __all__ attribute doesn't do anything if 
you aren't using import *. If there was some proposal to make it so that 
accessing an attribute not in __all__ did prevent or somehow warn users that 
they're doing something dangerous, that'd be different, but there isn't such a 
proposal, and I don't even know what such a proposal would look like...

On the other hand, if you make the primary mechanism to indicate privateness be 
a leading underscore, that's obvious to everyone.

James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Brett Cannon]

On Wed, Nov 10, 2010 at 05:47, Michael Foord  wrote:
> On 08/11/2010 22:07, Raymond Hettinger wrote:
>>
>> On Nov 8, 2010, at 11:58 AM, Brett Cannon wrote:
>>
>>> I think we need to, as a group, decide how to handle undocumented APIs
>>> that don't have a leading underscore: they get treated just the same
>>> as the documented APIs, or are they private regardless and thus we can
>>> change them at our whim?
>>
>> To start with, it doesn't hurt for a maintainer to add an __all__ entry
>> and to only document the parts of the API we think need to be exposed.  That
>> way, we can at least declare the parts that are intended to be public on a
>> go-forward basis.
>>
>> For the most part, the non-underscored parts of the API shouldn't be
>> changed "at our whim".  Some sense needs to be applied to the decision.
>>  Google's code search is great for showing how people actually have used a
>> module in real world code.  If that shows that people are accessing and/or
>> changing an attribute, it probably needs to remain exposed.   In the absence
>> of a code search, good guesses can be made about what someone might
>> reasonably and usefully be accessing (i.e. glob0 isn't likely).   The goal
>> is to improve the standard library while minimizing breakage, and that will
>> involve trade-offs depending on what is being changed.
>>
>> IIRC, we've been trying to get away from deprecations because they're so
>> disruptive.  For example, when the pprint rewrite is finally ready, if there
>> is an incompatible API change, I expect that a new clean class will be
>> offered, but that the old will be left in-place so that tons of existing
>> code won't break).  Likewise, with the unittest clean-ups, I'm expecting
>> that Michael will introduce aliases when fixing-up mis-named methods, rather
>> than break code that uses the existing names.
>>
>
> So it is obvious that we don't have a clearly stated policy for what defines
> the public API of standard library modules.
>
> How about making this explicit (either pep 8 or our developer docs):
>
> If a module or package defines __all__ that authoritatively defines the
> public interface. Modules with __all__ SHOULD still respect the naming
> conventions (leading underscore for private members) to avoid confusing
> users. Modules SHOULD NOT export private members in __all__.
>
> Names imported into a module a never considered part of its public API
> unless documented to be so or included in __all__.
>
> Methods / functions / classes and module attributes whose names begin with a
> leading underscore are private.
>
> If a class name begins with a leading underscore none of its members are
> public, whether or not they begin with a leading underscore.
>
> If a module name in a package begins with a leading underscore none of its
> members are public, whether or not they begin with a leading underscore.
>
> If a module or package doesn't define __all__ then all names that don't
> start with a leading underscore are public.
>
> All public members MUST be documented. Public functions, methods and classes
> SHOULD have docstrings. Private members may have docstrings.
>
>
> Where in the standard library this means that a module exports stuff that
> isn't helpful or shouldn't be part of the public API we need to migrate to
> private names and follow our deprecation process for the public names.

All sounds reasonable to me and what common practice out in the community is.

-Brett

>
> All the best,
>
>
> Michael Foord
>>
>> my-two-cents,
>>
>>
>> Raymond
>>
>>
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>
> --
>
> http://www.voidspace.org.uk/
>
> READ CAREFULLY. By accepting and reading this email you agree,
> on behalf of your employer, to release me from all obligations
> and waivers arising from any and all NON-NEGOTIATED agreements,
> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
> confidentiality, non-disclosure, non-compete and acceptable use
> policies (”BOGUS AGREEMENTS”) that I have entered into with your
> employer, its partners, licensors, agents and assigns, in
> perpetuity, without prejudice to my ongoing rights and privileges.
> You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Greg Ewing]

Stephen J. Turnbull wrote:

I don't really understand
what Tres is talking about when he writes "modules that expect to be
imported this way".  The *imported* module shouldn't care, no?  This
is an issue for the *importing* code to deal with.


I think he's talking about modules that add a prefix to all
of their exported names, such as Tkinter starting everything
with "Tk", on the expectation that import * will be the normal
way of using the module.

For very well-known modules with very well-known prefixes,
this probably doesn't do too much harm, since it's usually
fairly obvious where a given name is coming from. However,
it's probably best not encouraged, as it could lead people
who don't know better into bad habits.

There's also the downside that people who choose *not* to
use import *, and instead import the module itself and use
qualified references, end up with everything being prefixed
twice, e.g. 'import Tkinter as tk' leads to 'tk.TkWhatever'
everywhere.

On the other hand, when wrapping a C library there's a desire
to keep the Python names as close as possible to the C ones,
which usually come with prefixes to manage C's totally-global
namespace. So there's a bit of a double bind there.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 11/09/2010 11:12 PM, Stephen J. Turnbull wrote:
> Nick Coghlan writes:
> 
>  > > Module writers who compound the error by expecting to be imported
>  > > this way, thereby bogarting the global namespace for their own
>  > > purposes, should be fish-slapped. ;)
>  > 
>  > Be prepared to fish-slap all of python-dev then - we use precisely
>  > this technique to support optional acceleration modules. The pure
>  > Python versions of pairs like profile/_profile and heapq/_heapq
>  > include a try/except block at the end that does the equivalent of:
>  > 
>  >   try:
>  > from _accelerated import * # Allow accelerated overrides
>  >   except ImportError:
>  > pass # Use pure Python versions
> 
> But these identifiers will appear at the module level, not global, no?
> Otherwise this technique couldn't be used.  I don't really understand
> what Tres is talking about when he writes "modules that expect to be
> imported this way".  The *imported* module shouldn't care, no?  This
> is an issue for the *importing* code to deal with.

Right -- "private" star imports aren't the issue for me, because the
same user who creates them is responsible for the other end fo the
stick.  I was ranting about library authors who document star imports as
the expected usage pattern for their external users.

Note that I still wouldn't use star imports in the "private
acceleration" case myself.  I would prefer a pattern like:

- --- $< -
# spam.py

# Pure python API implementation
def foo(spat, blarg):
...

def bar(qux):
...

# Replace with accelearated C implemenataion
try:
import _spam
except ImportError:
pass # accelerated version not available
else:
foo = _spam.foo
bar = _spam.bar
- --- $< -

This explicit name remapping catches unintentional erros (e.g., _spam
renames a method) better than the star import.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzazykACgkQ+gerLs4ltQ5BHACfaAh2lVLZ8C+mdV/88UJ0JXTo
sqQAn2b2J9cZSQuz2xrwZX/JrvY3AaMh
=EIDa
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[R. David Murray]

On Wed, 10 Nov 2010 13:12:09 +0900, "Stephen J. Turnbull"  
wrote:
> Nick Coghlan writes:
> 
>  > > Module writers who compound the error by expecting to be imported
>  > > this way, thereby bogarting the global namespace for their own
>  > > purposes, should be fish-slapped. ;)
>  > 
>  > Be prepared to fish-slap all of python-dev then - we use precisely
>  > this technique to support optional acceleration modules. The pure
>  > Python versions of pairs like profile/_profile and heapq/_heapq
>  > include a try/except block at the end that does the equivalent of:
>  > 
>  >   try:
>  > from _accelerated import * # Allow accelerated overrides
>  >   except ImportError:
>  > pass # Use pure Python versions
> 
> But these identifiers will appear at the module level, not global, no?
> Otherwise this technique couldn't be used.  I don't really understand
> what Tres is talking about when he writes "modules that expect to be
> imported this way".  The *imported* module shouldn't care, no?  This
> is an issue for the *importing* code to deal with.

I think Tres was referring to certain packages (which shall remain
nameless since I don't feel like googling to find one) whose
documentation recommends the 'from  import *' methodology.

At least that's how I read "Module writers who..."  (that is, he's not
saying the *module* expects to be imported that way). [*]

--
R. David Murray  www.bitdance.com

[*] although reading that sentence literally, the thought of such a
module writer themselves being imported that way (a la Tron) has a
certain charm
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Michael Foord]

On 08/11/2010 22:07, Raymond Hettinger wrote:

On Nov 8, 2010, at 11:58 AM, Brett Cannon wrote:


I think we need to, as a group, decide how to handle undocumented APIs
that don't have a leading underscore: they get treated just the same
as the documented APIs, or are they private regardless and thus we can
change them at our whim?

To start with, it doesn't hurt for a maintainer to add an __all__ entry and to 
only document the parts of the API we think need to be exposed.  That way, we 
can at least declare the parts that are intended to be public on a go-forward 
basis.

For the most part, the non-underscored parts of the API shouldn't be changed "at our 
whim".  Some sense needs to be applied to the decision.  Google's code search is 
great for showing how people actually have used a module in real world code.  If that 
shows that people are accessing and/or changing an attribute, it probably needs to remain 
exposed.   In the absence of a code search, good guesses can be made about what someone 
might reasonably and usefully be accessing (i.e. glob0 isn't likely).   The goal is to 
improve the standard library while minimizing breakage, and that will involve trade-offs 
depending on what is being changed.

IIRC, we've been trying to get away from deprecations because they're so 
disruptive.  For example, when the pprint rewrite is finally ready, if there is 
an incompatible API change, I expect that a new clean class will be offered, 
but that the old will be left in-place so that tons of existing code won't 
break).  Likewise, with the unittest clean-ups, I'm expecting that Michael will 
introduce aliases when fixing-up mis-named methods, rather than break code that 
uses the existing names.



So it is obvious that we don't have a clearly stated policy for what 
defines the public API of standard library modules.


How about making this explicit (either pep 8 or our developer docs):

If a module or package defines __all__ that authoritatively defines the 
public interface. Modules with __all__ SHOULD still respect the naming 
conventions (leading underscore for private members) to avoid confusing 
users. Modules SHOULD NOT export private members in __all__.


Names imported into a module a never considered part of its public API 
unless documented to be so or included in __all__.


Methods / functions / classes and module attributes whose names begin 
with a leading underscore are private.


If a class name begins with a leading underscore none of its members are 
public, whether or not they begin with a leading underscore.


If a module name in a package begins with a leading underscore none of 
its members are public, whether or not they begin with a leading underscore.


If a module or package doesn't define __all__ then all names that don't 
start with a leading underscore are public.


All public members MUST be documented. Public functions, methods and 
classes SHOULD have docstrings. Private members may have docstrings.



Where in the standard library this means that a module exports stuff 
that isn't helpful or shouldn't be part of the public API we need to 
migrate to private names and follow our deprecation process for the 
public names.


All the best,


Michael Foord

my-two-cents,


Raymond




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Michael Foord]

On 09/11/2010 22:09, Nick Coghlan wrote:

On Wed, Nov 10, 2010 at 4:49 AM, Tres Seaver  wrote:

Outside an interactive prompt, anyone using "from foo import *" has set
themselves and their users up to lose anyway.

That syntax is the single worst misfeature in all of Python.  It impairs
readability and discoverability for *no* benefit beyond one-time typing
convenience.  Module writers who compound the error by expecting to be
imported this way, thereby bogarting the global namespace for their own
purposes, should be fish-slapped. ;)

Be prepared to fish-slap all of python-dev then - we use precisely
this technique to support optional acceleration modules. The pure
Python versions of pairs like profile/_profile and heapq/_heapq
include a try/except block at the end that does the equivalent of:

   try:
 from _accelerated import * # Allow accelerated overrides
   except ImportError:
 pass # Use pure Python versions

This allows each implementation to make its own decisions about
exactly which parts to accelerate without needing to change the pure
Python version. In CPython itself, different *builds* may vary based
on which components are available during the build process.

There are utility functions provided in test.support that allow us to
make sure that these modules are tested both with and without their
accelerated components.

The new unittest package in 2.7 and 3.2 also uses it in the module
__init__ to present the old "flat" namespace despite become a package
under the hood.


Look again. :-)

Benjamin did the refactoring into a package and he obviously dislikes 
"import *" as much as me. If he had used "import *" I would have changed 
it anyway, but he didn't.


We also define a __all__ to make the exported names explicit.

All the best,

Michael


Star imports are certainly open to abuse, but there are legitimate use
cases when you want to lie about where particular APIs live in the
module heirarchy. Those use cases generally involve being imported by
one *specific* other module, such that anyone else importing the
module directly *at all* is already doing the wrong thing.

Cheers,
Nick.




--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Raymond Hettinger]

On Nov 10, 2010, at 5:47 AM, Michael Foord wrote:

> 
> So it is obvious that we don't have a clearly stated policy for what defines 
> the public API of standard library modules.
> 
> How about making this explicit (either pep 8 or our developer docs):

I believe the point of Guido's email was that it is a situation dependent 
judgment call and not readily boiled down to a set of rules for PEP 8.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Hrvoje Niksic]

On 11/10/2010 05:12 AM, Stephen J. Turnbull wrote:

But these identifiers will appear at the module level, not global, no?
Otherwise this technique couldn't be used.  I don't really understand
what Tres is talking about when he writes "modules that expect to be
imported this way".  The *imported* module shouldn't care, no?


I think he's referring to the choice of identifiers, and the usage 
examples given in the documentation and tutorials.  For example, in the 
original PyGTK, all identifiers included "Gtk" in the name, so it made 
sense to write from pygtk import * so you could spell GtkWindow as 
GtkWindow rather than the redundant pygtk.GtkWindow.  In that sense the 
module writer "expected" to be imported this way, although you are right 
that it doesn't the least bit matter for the correct operation of the 
module itself.  For GTK 2 PyGTK switch to "gtk.Window", which 
effectively removes the temptation to import * from the module.


There are other examples of that school, most notably ctypes, but also 
Tkinter and the python2 threading module.  Fortunately it has become 
much less popular in the last ~5 years of Python history.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Floris Bruynooghe]

On 10 November 2010 04:12, Stephen J. Turnbull  wrote:
> Nick Coghlan writes:
>
>  > > Module writers who compound the error by expecting to be imported
>  > > this way, thereby bogarting the global namespace for their own
>  > > purposes, should be fish-slapped. ;)
>  >
>  > Be prepared to fish-slap all of python-dev then - we use precisely
>  > this technique to support optional acceleration modules. The pure
>  > Python versions of pairs like profile/_profile and heapq/_heapq
>  > include a try/except block at the end that does the equivalent of:
>  >
>  >   try:
>  >     from _accelerated import * # Allow accelerated overrides
>  >   except ImportError:
>  >     pass # Use pure Python versions
>
> But these identifiers will appear at the module level, not global, no?
> Otherwise this technique couldn't be used.  I don't really understand
> what Tres is talking about when he writes "modules that expect to be
> imported this way".  The *imported* module shouldn't care, no?  This
> is an issue for the *importing* code to deal with.

I can't think of stdlib examples, but for 3rd party packages I'd say
storm.locals and fabric.api are examples of packages designed with
"from foo import * " in mind.  So this does happen.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Nick Coghlan]

On Wed, Nov 10, 2010 at 10:23 PM, Michael Foord
 wrote:
> On 09/11/2010 22:09, Nick Coghlan wrote:
>> The new unittest package in 2.7 and 3.2 also uses it in the module
>> __init__ to present the old "flat" namespace despite become a package
>> under the hood.
>
> Look again. :-)
>
> Benjamin did the refactoring into a package and he obviously dislikes
> "import *" as much as me. If he had used "import *" I would have changed it
> anyway, but he didn't.
>
> We also define a __all__ to make the exported names explicit.

Fair cop :)

(and in that particular case, the maintenance burden in being explicit
is minimal, since new top-level names in unittest are going to be
significantly more rare than new methods on existing unittest classes)

Even some of the acceleration modules (such as _hashlib) use
approaches that are more explicit than using "import *". The point at
least stands for the cases where the pure Python version is largely
agnostic as to exactly which names the acceleration module overrides.
It's a very, very niche use case though, so the default position of
"if you use a star import anywhere other than at the interactive
prompt, you're most like wrong to do so" is still a reasonable stance
to take :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Stephen J. Turnbull]

Nick Coghlan writes:

 > > Module writers who compound the error by expecting to be imported
 > > this way, thereby bogarting the global namespace for their own
 > > purposes, should be fish-slapped. ;)
 > 
 > Be prepared to fish-slap all of python-dev then - we use precisely
 > this technique to support optional acceleration modules. The pure
 > Python versions of pairs like profile/_profile and heapq/_heapq
 > include a try/except block at the end that does the equivalent of:
 > 
 >   try:
 > from _accelerated import * # Allow accelerated overrides
 >   except ImportError:
 > pass # Use pure Python versions

But these identifiers will appear at the module level, not global, no?
Otherwise this technique couldn't be used.  I don't really understand
what Tres is talking about when he writes "modules that expect to be
imported this way".  The *imported* module shouldn't care, no?  This
is an issue for the *importing* code to deal with.





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 11/09/2010 03:48 PM, Toshio Kuratomi wrote:

> I think there's a valid case for bogarting the namespace in this instance,
> but let me know if there's a better way to do it::
> 
> # Method to use system libraries if available, otherwise use a bundled copy,
> # aka: make both system packagers and developers happy::
> 
> 
> Relevant directories and files for this module::
> 
> + foo/
> +- __init__.py
> ++ compat/
>  +- __init__.py
>  ++ bar/
>   +- __init__.py
>   +- _bar.py
> 
> foo/compat/bar/_bar.py is a bundled module.
> 
> foo/compat/bar/__init__.py has:
> 
> try:
> from bar import *
> from bar import __all__
> except ImportError::
> from foo.compat.bar._bar import *
> from foo.compat.bar._bar import __all__

I guess the usual caveats apply for dopplegangers / proxies. ;)


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzZxzAACgkQ+gerLs4ltQ5UsgCfcaxeFruJCDGnxBA0ma8Pjggg
lW8AoMBx2FYg+PSA/Zbq94UbiPhKGnjO
=/8QU
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Nick Coghlan]

On Wed, Nov 10, 2010 at 4:49 AM, Tres Seaver  wrote:
> Outside an interactive prompt, anyone using "from foo import *" has set
> themselves and their users up to lose anyway.
>
> That syntax is the single worst misfeature in all of Python.  It impairs
> readability and discoverability for *no* benefit beyond one-time typing
> convenience.  Module writers who compound the error by expecting to be
> imported this way, thereby bogarting the global namespace for their own
> purposes, should be fish-slapped. ;)

Be prepared to fish-slap all of python-dev then - we use precisely
this technique to support optional acceleration modules. The pure
Python versions of pairs like profile/_profile and heapq/_heapq
include a try/except block at the end that does the equivalent of:

  try:
from _accelerated import * # Allow accelerated overrides
  except ImportError:
pass # Use pure Python versions

This allows each implementation to make its own decisions about
exactly which parts to accelerate without needing to change the pure
Python version. In CPython itself, different *builds* may vary based
on which components are available during the build process.

There are utility functions provided in test.support that allow us to
make sure that these modules are tested both with and without their
accelerated components.

The new unittest package in 2.7 and 3.2 also uses it in the module
__init__ to present the old "flat" namespace despite become a package
under the hood.

Star imports are certainly open to abuse, but there are legitimate use
cases when you want to lie about where particular APIs live in the
module heirarchy. Those use cases generally involve being imported by
one *specific* other module, such that anyone else importing the
module directly *at all* is already doing the wrong thing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Toshio Kuratomi]

On Tue, Nov 09, 2010 at 01:49:01PM -0500, Tres Seaver wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 11/08/2010 06:26 PM, Bobby Impollonia wrote:
> 
> > This does hurt because anyone who was relying on "import *" to get a
> > name which is now omitted from __all__ is going to upgrade and find
> > their program failing with NameErrors. This is a backwards compatible
> > change and shouldn't happen without a deprecation warning first.
> 
> Outside an interactive prompt, anyone using "from foo import *" has set
> themselves and their users up to lose anyway.
> 
> That syntax is the single worst misfeature in all of Python.  It impairs
> readability and discoverability for *no* benefit beyond one-time typing
> convenience.  Module writers who compound the error by expecting to be
> imported this way, thereby bogarting the global namespace for their own
> purposes, should be fish-slapped. ;)
> 
I think there's a valid case for bogarting the namespace in this instance,
but let me know if there's a better way to do it::

# Method to use system libraries if available, otherwise use a bundled copy,
# aka: make both system packagers and developers happy::


Relevant directories and files for this module::

+ foo/
+- __init__.py
++ compat/
 +- __init__.py
 ++ bar/
  +- __init__.py
  +- _bar.py

foo/compat/bar/_bar.py is a bundled module.

foo/compat/bar/__init__.py has:

try:
from bar import *
from bar import __all__
except ImportError::
from foo.compat.bar._bar import *
from foo.compat.bar._bar import __all__

-Toshio


pgp2MughtFdu4.pgp
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Glyph Lefkowitz]

On Nov 8, 2010, at 4:50 PM, Guido van Rossum wrote:
> On Mon, Nov 8, 2010 at 3:55 PM, Glyph Lefkowitz  
> wrote:
>> This seems like a pretty clear case of "practicality beats purity".  Not 
>> only has nobody complained about deprecatedModuleAttribute, but there are 
>> tons of things which show up in sys.modules that aren't modules in the sense 
>> of 'instances of ModuleType'.  The Twisted reactor, for example, is an 
>> instance, and we've been doing *that* for about 10 years with no complaints.
> 
> But the Twisted universe is only a subset of the Python universe. The
> Python stdlib needs to move more carefully.

While this is true, I think the Twisted universe generally represents a 
particularly conservative, compatibility-conscious area within the Python 
universe (multiverse?).  I know of several Twisted users who regularly upgrade 
to the most recent version of Twisted without incident, but can't move from 
Python 2.4->2.5 because of compatibility issues.

That's not to say that there are no areas within the larger Python ecosystem 
that I'm unaware of where putting non-module-objects into sys.modules would 
cause issues.  But if it were a practice that were at all common, I suspect 
that we would have bumped into it by now.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 11/08/2010 06:26 PM, Bobby Impollonia wrote:

> This does hurt because anyone who was relying on "import *" to get a
> name which is now omitted from __all__ is going to upgrade and find
> their program failing with NameErrors. This is a backwards compatible
> change and shouldn't happen without a deprecation warning first.

Outside an interactive prompt, anyone using "from foo import *" has set
themselves and their users up to lose anyway.

That syntax is the single worst misfeature in all of Python.  It impairs
readability and discoverability for *no* benefit beyond one-time typing
convenience.  Module writers who compound the error by expecting to be
imported this way, thereby bogarting the global namespace for their own
purposes, should be fish-slapped. ;)


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzZl50ACgkQ+gerLs4ltQ5PYQCfUF2l8BjYvaZSu7ATT8/PxweH
jqMAoIWD/D5KIfLp/JOdPVuWJsH/kdc/
=/349
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[exarkun]

On 11:53 am, solip...@pitrou.net wrote:

On Tue, 09 Nov 2010 02:03:23 -
exar...@twistedmatrix.com wrote:


I wonder if there are any actual technical arguments to be made 
against

something like `deprecatedModuleAttribute`?


For example, does it work well with import hacks such as Mercurial's
demandimport?


I haven't tried before, but a quick experiment suggests that the two 
happily co-exist (aside from demandimport getting the blame instead of 
the true offending code, but that's really a problem with the warnings 
module):


 >>> import mercurial.demandimport as di
 >>> di.enable()
 >>> import twisted.python.threadpool as tp
 >>> tp.ThreadSafeList
 /usr/lib/pymodules/python2.6/mercurial/demandimport.py:76: 
DeprecationWarning: twisted.python.threadpool.ThreadSafeList was 
deprecated in Twisted 10.1.0: This was an internal implementation detail 
of support for Jython 2.1, which is now obsolete.

 return getattr(self._module, attr)
 
 >>>

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Antoine Pitrou]

On Tue, 09 Nov 2010 02:03:23 -
exar...@twistedmatrix.com wrote:
> 
> I wonder if there are any actual technical arguments to be made against 
> something like `deprecatedModuleAttribute`?

For example, does it work well with import hacks such as Mercurial's
demandimport?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Eli Bendersky]

On Tue, Nov 9, 2010 at 04:07, R. David Murray  wrote:

> On Mon, 08 Nov 2010 18:10:17 -0600, Ron Adam  wrote:
> > def _private_api():
> >  #
> >  # Isn't it a good practice to use comments here?
> >  #
> >  ...
>
> IMO, no.
>
>
FWIW, I agree completely. Docstrings are a part of Python I don't see a
reason to leave out for "non-public" code. They're convenient in the
beginning of functions and we all are used to seeing them there. IDE's use
them to display helpful "tooltips" on functions, and so on.


Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Nick Coghlan]

On Tue, Nov 9, 2010 at 1:28 PM, Alexander Belopolsky
 wrote:
> On Mon, Nov 8, 2010 at 2:58 PM, Brett Cannon  wrote:
> ..
>> But that doesn't mean we can't go through, fix up our names, and
>> deprecate the old public names; that's fair game in my book.

Indeed. I've now recommended Ron do exactly that for the pydoc patch.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Ron Adam]

On 11/08/2010 07:18 PM, Brett Cannon wrote:

On Mon, Nov 8, 2010 at 16:10, Ron Adam  wrote:



def _private_api():
#
# Isn't it a good practice to use comments here?
#
...


That is ugly. I already hate doing that for unittest, I'm not about to
champion that for anything else.


Ugly?  I suppose it's a matter of what you are used to.



It would also lead to essentially requiring a docstrings for
everything that is public whether someone wants to bother to writing a
docstring or not. I don't think we should be suggesting that a
docstring be required either.


I can see where that would be overly strict in an application or script 
made with python.


But it seems odd to me, to have undocumented api's in a programming 
language.  If it's being replaced with something else, the doc string can 
say that.  A null string is also a valid doc string if you just need a 
place holder until someone gets to it.






Brett, I'm sure you can up with a better alternative.   ;-)


But I don't want to have to do that in the stdlib by remembering what
modules I should or should not import. This is just as much about
developer burden on core devs as it is making sure we don't yank the
rug out from underneath users.


Yes, I agree.  But how to best do that?



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Alexander Belopolsky]

On Mon, Nov 8, 2010 at 2:58 PM, Brett Cannon  wrote:
..
> But that doesn't mean we can't go through, fix up our names, and
> deprecate the old public names; that's fair game in my book.
>

+1

See http://bugs.python.org/issue10371
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[R. David Murray]

On Mon, 08 Nov 2010 18:10:17 -0600, Ron Adam  wrote:
> def _private_api():
>  #
>  # Isn't it a good practice to use comments here?
>  #
>  ...

IMO, no.

--
R. David Murray  www.bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[exarkun]

On 12:50 am, gu...@python.org wrote:
On Mon, Nov 8, 2010 at 3:55 PM, Glyph Lefkowitz 
 wrote:
This seems like a pretty clear case of "practicality beats purity". 
Not only has nobody complained about deprecatedModuleAttribute, but 
there are tons of things which show up in sys.modules that aren't 
modules in the sense of 'instances of ModuleType'.  The Twisted 
reactor, for example, is an instance, and we've been doing *that* for 
about 10 years with no complaints.


But the Twisted universe is only a subset of the Python universe. The
Python stdlib needs to move more carefully.


I think that Twisted developers are pretty careful to consider the 
consequences of changes they make to Twisted.  We have an explicit, 
documented backwards compatibility policy, for example.  We also have 
mandatory code review for all changes, with a documented set of 
guidelines outlining the minimum things a reviewer should be 
considering.


I wonder if there are any actual technical arguments to be made against 
something like `deprecatedModuleAttribute`?


Also, it turns out that ModuleType can be subclassed these days.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Toshio Kuratomi]

On Tue, Nov 09, 2010 at 11:46:59AM +1100, Ben Finney wrote:
> Ron Adam  writes:
> 
> > def _publicly_documented_private_api():
> > """  Not sure why you would want to do this
> >  instead of using comments.
> > """
> > ...
> 
> Because the docstring is available at the interpreter via ‘help()’, and
> because it's automatically available to ‘doctest’, and most of the other
> good reasons for docstrings.
> 
> > The _publicly_documented_private_api() is a problem because people
> > *will* use it even though it has a leading underscore. Especially
> > those who are new to python.
> 
> That isn't an argument against docstrings, since the problem you
> describe isn't dependent on the presence or absence of docstrings.
> 
Just wanted to expand a bit here:  as a general practice, you may be
involved in a project where the _private_api() is not intended by people
outside of the project but is intended to be used in multiple places within
the project.  If you have different people working on those different areas,
it can be very useful for them to be able to use help(_private_api) on the
other functions from within the interpreter shell.

-Toshio


pgpG39YJbm42M.pgp
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Guido van Rossum]

On Mon, Nov 8, 2010 at 4:46 PM, Ben Finney  wrote:
> Ron Adam  writes:
>
>> def _publicly_documented_private_api():
>>     """  Not sure why you would want to do this
>>          instead of using comments.
>>     """
>>     ...
>
> Because the docstring is available at the interpreter via ‘help()’, and
> because it's automatically available to ‘doctest’, and most of the other
> good reasons for docstrings.
>
>> The _publicly_documented_private_api() is a problem because people
>> *will* use it even though it has a leading underscore. Especially
>> those who are new to python.
>
> That isn't an argument against docstrings, since the problem you
> describe isn't dependent on the presence or absence of docstrings.

+1

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Brett Cannon]

On Mon, Nov 8, 2010 at 16:10, Ron Adam  wrote:
>
>
> On 11/08/2010 04:01 PM, Brett Cannon wrote:
>
>>> My understanding is that anything with an actual docstring is part of the
>>> public API.  Any thing with a leading underscore is private.
>>
>> That's a bad rule. Why shouldn't I be able to document something that
>> is not meant for the public so that fellow developers know what the
>> heck should be going on in the code?
>
> You can use comments instead of a docstring.
>
> Here are the possible cases concerned with the subject.  I'm using functions
> here for these examples, but this also applies to other objects.
>
>
> def public_api():
>    """ Should always have a nice docstring. """
>    ...
>
>
> def _private_api():
>    #
>    # Isn't it a good practice to use comments here?
>    #
>    ...

That is ugly. I already hate doing that for unittest, I'm not about to
champion that for anything else.

It would also lead to essentially requiring a docstrings for
everything that is public whether someone wants to bother to writing a
docstring or not. I don't think we should be suggesting that a
docstring be required either.

>
>
> def _publicly_documented_private_api():
>    """  Not sure why you would want to do this
>         instead of using comments.
>    """
>    ...
>
>
> def undocumented_public_api():
>    ...
>
>
> def _undocumented_private_api():
>    ...
>
>
> Out of these, the two that are problematic are the
> _publicly_documented_private_api() and the undocumented_public_api().
>
> The _publicly_documented_private_api() is a problem because people *will*
> use it even though it has a leading underscore.  Especially those who are
> new to python.
>
> The undocumented_public_api() wouldn't be a problem if all private api's
> used leading  underscore, but for older modules, it isn't always clear what
> the intention was.  Was it undocumented because the programmer simply
> forgot, or was it intended to be a private api?
>
>
>
>>> It may also be useful to clarify that importing some "utility" modules is
>>> not recommended because they may be changed more often and may not follow
>>> the standard process.  Would something like the following work, but still
>>> allow for importing if the exception is caught with a try except?
>>>
>>> if __name__ == "__main__":
>>>    main()
>>> else:
>>>    raise ImportWarning("This is utility module and may be changed.")
>>
>> Sure it would work, but that doesn't make it pleasant to use. It
>> already breaks how warnings are typically handled by raising it
>> instead of calling warnings.warn(). Plus I'm now supposed to
>> try/except certain imports? That's messy. At that point we are coding
>> in visibility rules instead of following convention and that doesn't
>> sit well with me.
>
> No, you're not suppose to try/except imports.  That's the point.
>
> You can do that, only if you really want to abuse the intended purpose of a
> module that isn't meant to be imported in the first place.  If someone wants
> to do that, it isn't a problem.  They are well aware of the risks if they do
> it.  (This is just one option and probably one that isn't thought out very
> well.)
>
> Brett, I'm sure you can up with a better alternative.   ;-)

But I don't want to have to do that in the stdlib by remembering what
modules I should or should not import. This is just as much about
developer burden on core devs as it is making sure we don't yank the
rug out from underneath users.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Guido van Rossum]

On Mon, Nov 8, 2010 at 3:55 PM, Glyph Lefkowitz  wrote:
> This seems like a pretty clear case of "practicality beats purity".  Not only 
> has nobody complained about deprecatedModuleAttribute, but there are tons of 
> things which show up in sys.modules that aren't modules in the sense of 
> 'instances of ModuleType'.  The Twisted reactor, for example, is an instance, 
> and we've been doing *that* for about 10 years with no complaints.

But the Twisted universe is only a subset of the Python universe. The
Python stdlib needs to move more carefully.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Ben Finney]

Ron Adam  writes:

> def _publicly_documented_private_api():
> """  Not sure why you would want to do this
>  instead of using comments.
> """
> ...

Because the docstring is available at the interpreter via ‘help()’, and
because it's automatically available to ‘doctest’, and most of the other
good reasons for docstrings.

> The _publicly_documented_private_api() is a problem because people
> *will* use it even though it has a leading underscore. Especially
> those who are new to python.

That isn't an argument against docstrings, since the problem you
describe isn't dependent on the presence or absence of docstrings.

-- 
 \ “I wish there was a knob on the TV to turn up the intelligence. |
  `\  There's a knob called ‘brightness’ but it doesn't work.” |
_o__) —Eugene P. Gallagher |
Ben Finney

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Ben Finney]

Bobby Impollonia  writes:

> On Mon, Nov 8, 2010 at 2:07 PM, Raymond Hettinger
>  wrote:
> > To start with, it doesn't hurt for a maintainer to add an __all__
> > entry and to only document the parts of the API we think need to be
> > exposed.  That way, we can at least declare the parts that are
> > intended to be public on a go-forward basis.
>
> This does hurt because anyone who was relying on "import *" to get a
> name which is now omitted from __all__ is going to upgrade and find
> their program failing with NameErrors. This is a backwards compatible
> change and shouldn't happen without a deprecation warning first.

It also introduces a (perhaps small, but clearly non-zero) maintenance
burden: the name of an object must be added, changed, and removed not
only where it is defined, but also in the ‘__all__’ entry.

This burden is avoided when using the spelling of the name itself as the
indicator for exposure in the API.

-- 
 \ “In any great organization it is far, far safer to be wrong |
  `\  with the majority than to be right alone.” —John Kenneth |
_o__)Galbraith, 1989-07-28 |
Ben Finney

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Ron Adam]

On 11/08/2010 04:01 PM, Brett Cannon wrote:


My understanding is that anything with an actual docstring is part of the
public API.  Any thing with a leading underscore is private.


That's a bad rule. Why shouldn't I be able to document something that
is not meant for the public so that fellow developers know what the
heck should be going on in the code?


You can use comments instead of a docstring.

Here are the possible cases concerned with the subject.  I'm using 
functions here for these examples, but this also applies to other objects.



def public_api():
""" Should always have a nice docstring. """
...


def _private_api():
#
# Isn't it a good practice to use comments here?
#
...


def _publicly_documented_private_api():
"""  Not sure why you would want to do this
 instead of using comments.
"""
...


def undocumented_public_api():
...


def _undocumented_private_api():
...


Out of these, the two that are problematic are the 
_publicly_documented_private_api() and the undocumented_public_api().


The _publicly_documented_private_api() is a problem because people *will* 
use it even though it has a leading underscore.  Especially those who are 
new to python.


The undocumented_public_api() wouldn't be a problem if all private api's 
used leading  underscore, but for older modules, it isn't always clear what 
the intention was.  Was it undocumented because the programmer simply 
forgot, or was it intended to be a private api?





It may also be useful to clarify that importing some "utility" modules is
not recommended because they may be changed more often and may not follow
the standard process.  Would something like the following work, but still
allow for importing if the exception is caught with a try except?

if __name__ == "__main__":
main()
else:
raise ImportWarning("This is utility module and may be changed.")


Sure it would work, but that doesn't make it pleasant to use. It
already breaks how warnings are typically handled by raising it
instead of calling warnings.warn(). Plus I'm now supposed to
try/except certain imports? That's messy. At that point we are coding
in visibility rules instead of following convention and that doesn't
sit well with me.


No, you're not suppose to try/except imports.  That's the point.

You can do that, only if you really want to abuse the intended purpose of a 
module that isn't meant to be imported in the first place.  If someone 
wants to do that, it isn't a problem.  They are well aware of the risks if 
they do it.  (This is just one option and probably one that isn't thought 
out very well.)


Brett, I'm sure you can up with a better alternative.   ;-)

Cheers,
  Ron





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Glyph Lefkowitz]

On Nov 8, 2010, at 2:35 PM, exar...@twistedmatrix.com wrote:

> On 09:57 pm, br...@python.org wrote:
>> On Mon, Nov 8, 2010 at 13:45,   wrote:
>>> On 09:25 pm, br...@python.org wrote:
 
 On Mon, Nov 8, 2010 at 13:03,   wrote:
> 
> On 07:58 pm, br...@python.org wrote:
>>> 
>>> I don't think a strict don't remove without deprecation policy is
>>> workable.  For example, is trace.rx_blank constant part of the trace
>>> module API that needs to be preserved indefinitely?  I don't even know
>>> if it is possible to add a deprecation warning to it, but
>>> CoverageResults._blank_re would certainly be a better place for it.
>> 
>> The deprecation policy obviously cannot apply to module-level
>> attributes.
> 
> I'm not sure why this is.  Can you elaborate?
 
 There is no way to directly trigger a DeprecationWarning for an
 attribute. We can still document it, but there is just no way to
 programmatically enforce it.
>>> 
>>> What about `deprecatedModuleAttribute`
>>> ()
>>> or zope.deprecation
>>> () which inspired it?
>> 
>> Just checked the code and it looks like it substitutes the module for
>> some proxy object? To begin that break subclass checks. After that I
>> don't know the ramifications without really digging into the
>> ModuleType code.
> 
> That could be fixed if ModuleType allowed subclassing. :)
> 
> For what it's worth, no one has complained about problems caused by 
> `deprecatedModuleAttribute`, but we've only been using it for about two and a 
> half years.

This seems like a pretty clear case of "practicality beats purity".  Not only 
has nobody complained about deprecatedModuleAttribute, but there are tons of 
things which show up in sys.modules that aren't modules in the sense of 
'instances of ModuleType'.  The Twisted reactor, for example, is an instance, 
and we've been doing *that* for about 10 years with no complaints.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Guido van Rossum]

On Mon, Nov 8, 2010 at 3:26 PM, Bobby Impollonia  wrote:
> On Mon, Nov 8, 2010 at 2:07 PM, Raymond Hettinger
>  wrote:
>>
>> On Nov 8, 2010, at 11:58 AM, Brett Cannon wrote:
>>
>>> I think we need to, as a group, decide how to handle undocumented APIs
>>> that don't have a leading underscore: they get treated just the same
>>> as the documented APIs, or are they private regardless and thus we can
>>> change them at our whim?
>>
>> To start with, it doesn't hurt for a maintainer to add an __all__ entry and 
>> to only document the parts of the API we think need to be exposed.  That 
>> way, we can at least declare the parts that are intended to be public on a 
>> go-forward basis.
>
> This does hurt because anyone who was relying on "import *" to get a
> name which is now omitted from __all__ is going to upgrade and find
> their program failing with NameErrors. This is a backwards compatible
> change and shouldn't happen without a deprecation warning first.

Given that import * is generally frowned upon you can't make a blanket
statement like this without referring to the specifics of the name
being considered for removal. In fact, for any proposed change the
risk and reward need to be weighed properly. If the risk is "someone's
code could break if they used some undocumented API" it is useful to
estimate the probability that this would happen and that somebody
would care (rather than just fixing their code and moving on). Many
factors go into such an estimate. Just one example would be if we knew
of usage of the offending name in code that could reasonably be
assumed to be widely copied or distributed -- in such cases we should
move very carefully indeed no matter how "officially undocumented"
something is.

I don't want to go into the specifics of the trace module (even if I
wrote it, it's too long ago to remember, nor can I recall using it)
but I do want to warn about the dangers of applying simplifying rules
mindlessly.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Bobby Impollonia]

On Mon, Nov 8, 2010 at 2:07 PM, Raymond Hettinger
 wrote:
>
> On Nov 8, 2010, at 11:58 AM, Brett Cannon wrote:
>
>> I think we need to, as a group, decide how to handle undocumented APIs
>> that don't have a leading underscore: they get treated just the same
>> as the documented APIs, or are they private regardless and thus we can
>> change them at our whim?
>
> To start with, it doesn't hurt for a maintainer to add an __all__ entry and 
> to only document the parts of the API we think need to be exposed.  That way, 
> we can at least declare the parts that are intended to be public on a 
> go-forward basis.

This does hurt because anyone who was relying on "import *" to get a
name which is now omitted from __all__ is going to upgrade and find
their program failing with NameErrors. This is a backwards compatible
change and shouldn't happen without a deprecation warning first.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Terry Reedy]

On 11/8/2010 2:58 PM, Brett Cannon wrote:


I think we need to, as a group, decide how to handle undocumented APIs
that don't have a leading underscore: they get treated just the same
as the documented APIs, or are they private regardless and thus we can
change them at our whim?


How about in between: deprecate as if private, but do so much more 
freely that we would for public stuff. I think this is what you actually 
propose. We might deprecate faster too.



The main reason I have said that non-underscore names should be
properly deprecated (assuming they are not contained in an
underscored-named module) is that dir() and help() do not distinguish.
If you are perusing a module from the interpreter prompt you have no
way to know whether something is public or private if it lacks an
underscore. Is it reasonable to assume that any API found through
dir() or help() must be checked with the official docs before you can
consider using it, even if you have no explicit need to read the
official docs?

I (unfortunately) say no, which is why I have argued that
non-underscored names need to be properly deprecated. This obviously
places a nasty burden on us, though, so I don't like taking this


Completely naive question: Is there anything that could be automated to 
reduce the burden?



position. Unless we can make it clearly known through help() or
something that the official docs must be checked to know what can and
cannot be reliably used I don't think it is reasonable to force users
to not be able to rely on help() (we should probably change help() to
print a big disclaimer for anything with a leading underscore,
though).

But that doesn't mean we can't go through, fix up our names, and
deprecate the old public names; that's fair game in my book.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Terry Reedy]

On 11/8/2010 4:36 PM, Ron Adam wrote:


My understanding is that anything with an actual docstring is part of
the public API. Any thing with a leading underscore is private.


When the trace module was written, the rule seems to have been more 
like: docs (but no docstrings) for public API, docstrings (but no doc 
mention) for private stuff. Eli and I fixed the first part.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[exarkun]

On 09:57 pm, br...@python.org wrote:

On Mon, Nov 8, 2010 at 13:45,   wrote:

On 09:25 pm, br...@python.org wrote:


On Mon, Nov 8, 2010 at 13:03, � wrote:


On 07:58 pm, br...@python.org wrote:


I don't think a strict don't remove without deprecation policy is
workable.  For example, is trace.rx_blank constant part of the 
trace
module API that needs to be preserved indefinitely?  I don't even 
know

if it is possible to add a deprecation warning to it, but
CoverageResults._blank_re would certainly be a better place for 
it.


The deprecation policy obviously cannot apply to module-level
attributes.


I'm not sure why this is. �Can you elaborate?


There is no way to directly trigger a DeprecationWarning for an
attribute. We can still document it, but there is just no way to
programmatically enforce it.


What about `deprecatedModuleAttribute`
()
or zope.deprecation
() which 
inspired it?


Just checked the code and it looks like it substitutes the module for
some proxy object? To begin that break subclass checks. After that I
don't know the ramifications without really digging into the
ModuleType code.


That could be fixed if ModuleType allowed subclassing. :)

For what it's worth, no one has complained about problems caused by 
`deprecatedModuleAttribute`, but we've only been using it for about two 
and a half years.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Steven D'Aprano]

Ron Adam wrote:

My understanding is that anything with an actual docstring is part of 
the public API.


I frequently add docstrings to _private functions. Just because it is 
private doesn't mean I don't want documentation for it, and it is very 
handy for running doctests.


Yes, I test my internal functions *wink*

The convention I use is:

* If __all__ exists, anything in that is public.
* Anything not listed in __all__ but without a leading underscore is 
public, but not part of the module's API; e.g. utility functions, 
imported modules, globals (but hopefully not too many of the last). That 
means I don't expect you to use it, but you can if you want.
* Anything with a _private name is internal use only. That includes 
modules. Any attribute of a private object is also private.


If a class is flagged as private, _MyClass, you wouldn't expect that 
_MyClass.attribute were public just because the attribute name wasn't 
also flagged with an underscore. So why treat _module.name as public?




+1 on the help disclaimer for objects with leading underscores.


I don't know that it will be that useful, but I don't think it will help 
that much. +0.


Currently help() does not see comments when they are used in place of a 
docstring.  I think it would be easy to have help notate things with no 
docstrings as "Warning: Undocumented . Use at your own risk."


I wouldn't like that. I don't think that "no docstring" = "undocumented" 
-- the documentation might exist somewhere else.


Besides, I don't think that help() should start misidentifying public 
objects as private if you run it under python -OO.



It may also be useful to clarify that importing some "utility" modules 
is not recommended because they may be changed more often and may not 
follow the standard process.  Would something like the following work, 
but still allow for importing if the exception is caught with a try except?


if __name__ == "__main__":
main()
else:
raise ImportWarning("This is utility module and may be changed.")


There's no way for the imported module to know what module is importing 
it, is there? Because the API I'd much prefer is:


safe_modules = [a, b, c, d]  # List of modules allowed to import me.
if calling_module not in safe_modules:
warning.warn("private module, are you sure you want to do this?")



--
Steven

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Raymond Hettinger]

On Nov 8, 2010, at 11:58 AM, Brett Cannon wrote:

> I think we need to, as a group, decide how to handle undocumented APIs
> that don't have a leading underscore: they get treated just the same
> as the documented APIs, or are they private regardless and thus we can
> change them at our whim?

To start with, it doesn't hurt for a maintainer to add an __all__ entry and to 
only document the parts of the API we think need to be exposed.  That way, we 
can at least declare the parts that are intended to be public on a go-forward 
basis.

For the most part, the non-underscored parts of the API shouldn't be changed 
"at our whim".  Some sense needs to be applied to the decision.  Google's code 
search is great for showing how people actually have used a module in real 
world code.  If that shows that people are accessing and/or changing an 
attribute, it probably needs to remain exposed.   In the absence of a code 
search, good guesses can be made about what someone might reasonably and 
usefully be accessing (i.e. glob0 isn't likely).   The goal is to improve the 
standard library while minimizing breakage, and that will involve trade-offs 
depending on what is being changed.

IIRC, we've been trying to get away from deprecations because they're so 
disruptive.  For example, when the pprint rewrite is finally ready, if there is 
an incompatible API change, I expect that a new clean class will be offered, 
but that the old will be left in-place so that tons of existing code won't 
break).  Likewise, with the unittest clean-ups, I'm expecting that Michael will 
introduce aliases when fixing-up mis-named methods, rather than break code that 
uses the existing names.

my-two-cents,


Raymond




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Brett Cannon]

On Mon, Nov 8, 2010 at 13:36, Ron Adam  wrote:
>
>
> On 11/08/2010 01:58 PM, Brett Cannon wrote:
>>
>> On Mon, Nov 8, 2010 at 09:20, Alexander Belopolsky
>>   wrote:
>>>
>>> Was: [issue2001] Pydoc interactive browsing enhancement
>>>
>>> On Sun, Nov 7, 2010 at 9:17 AM, Nick Coghlan
>>>  wrote:
>>> ..

 I'd actually started typing out the command to commit this before it
 finally clicked that the patch changes public
 APIs of the pydoc module in incompatible ways. Sure, they aren't
 documented, but the fact they aren't protected
 by an underscore means I'm not comfortable with the idea of removing
 them or radically change their functionality
 without going through a deprecation period first.

>>>
>>> I have a similar issue with the trace module and would appreciate some
>>> guidance on this as well.  The trace module documented API includes
>>> just the Trace class, but the module defines several helper functions
>>> and classes  that do not start with a leading underscore and are not
>>> excluded from * imports by __all__.  (There is no trace.__all__.)
>>
>> I think we need to, as a group, decide how to handle undocumented APIs
>> that don't have a leading underscore: they get treated just the same
>> as the documented APIs, or are they private regardless and thus we can
>> change them at our whim?
>
> My understanding is that anything with an actual docstring is part of the
> public API.  Any thing with a leading underscore is private.

That's a bad rule. Why shouldn't I be able to document something that
is not meant for the public so that fellow developers know what the
heck should be going on in the code?

>
> And to a lesser extent, objects with out docstrings, but have comments
> instead or nothing, may change, so don't depend on them.  Thankfully most
> things do have docstrings.
>
>
>>> I freely admit that I have more questions than answers, so I would
>>> like to hear from a wider audience.
>>
>> The main reason I have said that non-underscore names should be
>> properly deprecated (assuming they are not contained in an
>> underscored-named module) is that dir() and help() do not distinguish.
>> If you are perusing a module from the interpreter prompt you have no
>> way to know whether something is public or private if it lacks an
>> underscore. Is it reasonable to assume that any API found through
>> dir() or help() must be checked with the official docs before you can
>> consider using it, even if you have no explicit need to read the
>> official docs?
>>
>> I (unfortunately) say no, which is why I have argued that
>> non-underscored names need to be properly deprecated. This obviously
>> places a nasty burden on us, though, so I don't like taking this
>> position. Unless we can make it clearly known through help() or
>> something that the official docs must be checked to know what can and
>> cannot be reliably used I don't think it is reasonable to force users
>> to not be able to rely on help() (we should probably change help() to
>> print a big disclaimer for anything with a leading underscore,
>> though).
>
> +1 on the help disclaimer for objects with leading underscores.
>
> Currently help() does not see comments when they are used in place of a
> docstring.  I think it would be easy to have help notate things with no
> docstrings as "Warning: Undocumented . Use at your own risk."
>
> At first, it would probably have a nice side effect of getting any public
> API's documented with doc strings. (if they aren't already.)
>
>
>> But that doesn't mean we can't go through, fix up our names, and
>> deprecate the old public names; that's fair game in my book.
>
> I agree.
>
>
> It may also be useful to clarify that importing some "utility" modules is
> not recommended because they may be changed more often and may not follow
> the standard process.  Would something like the following work, but still
> allow for importing if the exception is caught with a try except?
>
> if __name__ == "__main__":
>    main()
> else:
>    raise ImportWarning("This is utility module and may be changed.")

Sure it would work, but that doesn't make it pleasant to use. It
already breaks how warnings are typically handled by raising it
instead of calling warnings.warn(). Plus I'm now supposed to
try/except certain imports? That's messy. At that point we are coding
in visibility rules instead of following convention and that doesn't
sit well with me.

-Brett

>
> Cheers,
>  Ron
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Brett Cannon]

On Mon, Nov 8, 2010 at 13:45,   wrote:
> On 09:25 pm, br...@python.org wrote:
>>
>> On Mon, Nov 8, 2010 at 13:03,   wrote:
>>>
>>> On 07:58 pm, br...@python.org wrote:
>
> I don't think a strict don't remove without deprecation policy is
> workable.  For example, is trace.rx_blank constant part of the trace
> module API that needs to be preserved indefinitely?  I don't even know
> if it is possible to add a deprecation warning to it, but
> CoverageResults._blank_re would certainly be a better place for it.

 The deprecation policy obviously cannot apply to module-level
 attributes.
>>>
>>> I'm not sure why this is.  Can you elaborate?
>>
>> There is no way to directly trigger a DeprecationWarning for an
>> attribute. We can still document it, but there is just no way to
>> programmatically enforce it.
>
> What about `deprecatedModuleAttribute`
> ()
> or zope.deprecation
> () which inspired it?

Just checked the code and it looks like it substitutes the module for
some proxy object? To begin that break subclass checks. After that I
don't know the ramifications without really digging into the
ModuleType code.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Ron Adam]

On 11/08/2010 01:58 PM, Brett Cannon wrote:

On Mon, Nov 8, 2010 at 09:20, Alexander Belopolsky
  wrote:

Was: [issue2001] Pydoc interactive browsing enhancement

On Sun, Nov 7, 2010 at 9:17 AM, Nick Coghlan  wrote:
..


I'd actually started typing out the command to commit this before it finally 
clicked that the patch changes public
APIs of the pydoc module in incompatible ways. Sure, they aren't documented, 
but the fact they aren't protected
by an underscore means I'm not comfortable with the idea of removing them or 
radically change their functionality
without going through a deprecation period first.



I have a similar issue with the trace module and would appreciate some
guidance on this as well.  The trace module documented API includes
just the Trace class, but the module defines several helper functions
and classes  that do not start with a leading underscore and are not
excluded from * imports by __all__.  (There is no trace.__all__.)


I think we need to, as a group, decide how to handle undocumented APIs
that don't have a leading underscore: they get treated just the same
as the documented APIs, or are they private regardless and thus we can
change them at our whim?


My understanding is that anything with an actual docstring is part of the 
public API.  Any thing with a leading underscore is private.


And to a lesser extent, objects with out docstrings, but have comments 
instead or nothing, may change, so don't depend on them.  Thankfully most 
things do have docstrings.




I freely admit that I have more questions than answers, so I would
like to hear from a wider audience.


The main reason I have said that non-underscore names should be
properly deprecated (assuming they are not contained in an
underscored-named module) is that dir() and help() do not distinguish.
If you are perusing a module from the interpreter prompt you have no
way to know whether something is public or private if it lacks an
underscore. Is it reasonable to assume that any API found through
dir() or help() must be checked with the official docs before you can
consider using it, even if you have no explicit need to read the
official docs?

I (unfortunately) say no, which is why I have argued that
non-underscored names need to be properly deprecated. This obviously
places a nasty burden on us, though, so I don't like taking this
position. Unless we can make it clearly known through help() or
something that the official docs must be checked to know what can and
cannot be reliably used I don't think it is reasonable to force users
to not be able to rely on help() (we should probably change help() to
print a big disclaimer for anything with a leading underscore,
though).


+1 on the help disclaimer for objects with leading underscores.

Currently help() does not see comments when they are used in place of a 
docstring.  I think it would be easy to have help notate things with no 
docstrings as "Warning: Undocumented . Use at your own risk."


At first, it would probably have a nice side effect of getting any public 
API's documented with doc strings. (if they aren't already.)




But that doesn't mean we can't go through, fix up our names, and
deprecate the old public names; that's fair game in my book.


I agree.


It may also be useful to clarify that importing some "utility" modules is 
not recommended because they may be changed more often and may not follow 
the standard process.  Would something like the following work, but still 
allow for importing if the exception is caught with a try except?


if __name__ == "__main__":
main()
else:
raise ImportWarning("This is utility module and may be changed.")

Cheers,
  Ron
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[exarkun]

On 09:25 pm, br...@python.org wrote:

On Mon, Nov 8, 2010 at 13:03,   wrote:

On 07:58 pm, br...@python.org wrote:


I don't think a strict don't remove without deprecation policy is
workable. �For example, is trace.rx_blank constant part of the trace
module API that needs to be preserved indefinitely?  I don't even 
know

if it is possible to add a deprecation warning to it, but
CoverageResults._blank_re would certainly be a better place for it.


The deprecation policy obviously cannot apply to module-level 
attributes.


I'm not sure why this is. �Can you elaborate?


There is no way to directly trigger a DeprecationWarning for an
attribute. We can still document it, but there is just no way to
programmatically enforce it.


What about `deprecatedModuleAttribute` 
() 
or zope.deprecation 
() which inspired 
it?


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Brett Cannon]

On Mon, Nov 8, 2010 at 13:03,   wrote:
> On 07:58 pm, br...@python.org wrote:
>>>
>>> I don't think a strict don't remove without deprecation policy is
>>> workable.  For example, is trace.rx_blank constant part of the trace
>>> module API that needs to be preserved indefinitely?  I don't even know
>>> if it is possible to add a deprecation warning to it, but
>>> CoverageResults._blank_re would certainly be a better place for it.
>>
>> The deprecation policy obviously cannot apply to module-level attributes.
>
> I'm not sure why this is.  Can you elaborate?

There is no way to directly trigger a DeprecationWarning for an
attribute. We can still document it, but there is just no way to
programmatically enforce it.

-Brett

>>
>> The main reason I have said that non-underscore names should be
>> properly deprecated (assuming they are not contained in an
>> underscored-named module) is that dir() and help() do not distinguish.
>> If you are perusing a module from the interpreter prompt you have no
>> way to know whether something is public or private if it lacks an
>> underscore. Is it reasonable to assume that any API found through
>> dir() or help() must be checked with the official docs before you can
>> consider using it, even if you have no explicit need to read the
>> official docs?
>>
>> I (unfortunately) say no, which is why I have argued that
>> non-underscored names need to be properly deprecated. This obviously
>> places a nasty burden on us, though, so I don't like taking this
>> position. Unless we can make it clearly known through help() or
>> something that the official docs must be checked to know what can and
>> cannot be reliably used I don't think it is reasonable to force users
>> to not be able to rely on help() (we should probably change help() to
>> print a big disclaimer for anything with a leading underscore,
>> though).
>>
>> But that doesn't mean we can't go through, fix up our names, and
>> deprecate the old public names; that's fair game in my book.
>
> +1
>
> Jean-Paul
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[exarkun]

On 07:58 pm, br...@python.org wrote:

I don't think a strict don't remove without deprecation policy is
workable. �For example, is trace.rx_blank constant part of the trace
module API that needs to be preserved indefinitely? �I don't even know
if it is possible to add a deprecation warning to it, but
CoverageResults._blank_re would certainly be a better place for it.


The deprecation policy obviously cannot apply to module-level 
attributes.


I'm not sure why this is.  Can you elaborate?


The main reason I have said that non-underscore names should be
properly deprecated (assuming they are not contained in an
underscored-named module) is that dir() and help() do not distinguish.
If you are perusing a module from the interpreter prompt you have no
way to know whether something is public or private if it lacks an
underscore. Is it reasonable to assume that any API found through
dir() or help() must be checked with the official docs before you can
consider using it, even if you have no explicit need to read the
official docs?

I (unfortunately) say no, which is why I have argued that
non-underscored names need to be properly deprecated. This obviously
places a nasty burden on us, though, so I don't like taking this
position. Unless we can make it clearly known through help() or
something that the official docs must be checked to know what can and
cannot be reliably used I don't think it is reasonable to force users
to not be able to rely on help() (we should probably change help() to
print a big disclaimer for anything with a leading underscore,
though).

But that doesn't mean we can't go through, fix up our names, and
deprecate the old public names; that's fair game in my book.


+1

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Brett Cannon]

On Mon, Nov 8, 2010 at 09:20, Alexander Belopolsky
 wrote:
> Was: [issue2001] Pydoc interactive browsing enhancement
>
> On Sun, Nov 7, 2010 at 9:17 AM, Nick Coghlan  wrote:
> ..
>>
>> I'd actually started typing out the command to commit this before it finally 
>> clicked that the patch changes public
>> APIs of the pydoc module in incompatible ways. Sure, they aren't documented, 
>> but the fact they aren't protected
>> by an underscore means I'm not comfortable with the idea of removing them or 
>> radically change their functionality
>> without going through a deprecation period first.
>>
>
> I have a similar issue with the trace module and would appreciate some
> guidance on this as well.  The trace module documented API includes
> just the Trace class, but the module defines several helper functions
> and classes  that do not start with a leading underscore and are not
> excluded from * imports by __all__.  (There is no trace.__all__.)

I think we need to, as a group, decide how to handle undocumented APIs
that don't have a leading underscore: they get treated just the same
as the documented APIs, or are they private regardless and thus we can
change them at our whim?

>
> I don't think a strict don't remove without deprecation policy is
> workable.  For example, is trace.rx_blank constant part of the trace
> module API that needs to be preserved indefinitely?  I don't even know
> if it is possible to add a deprecation warning to it, but
> CoverageResults._blank_re would certainly be a better place for it.

The deprecation policy obviously cannot apply to module-level attributes.

>
> The functions I have specific need to modify (See
> http://bugs.python.org/issue10342) are trace.find_strings(), and
> find_executable_linenos().  The functions take module's file name, but
> I need to make them to take the module object in order to be able to
> deal with modules that have custom loaders.
>
> The trace.find_strings() function is clearly internal.  It's name does
> not even reflect what it does (finding docstring locations), so it was
> never intended for use outside of the trace module.  However, google
> code search reveals that people do use it and other functions in their
> code.
>
> This suggests that trace.find_strings() should probably be preserved
> or properly deprecated.  If this is the case, should we fix bugs in
> it?  Note that it currently has a bug because it ignores the coding
> cookie when opening python source file.  Should this be fixed?
>
> I freely admit that I have more questions than answers, so I would
> like to hear from a wider audience.

The main reason I have said that non-underscore names should be
properly deprecated (assuming they are not contained in an
underscored-named module) is that dir() and help() do not distinguish.
If you are perusing a module from the interpreter prompt you have no
way to know whether something is public or private if it lacks an
underscore. Is it reasonable to assume that any API found through
dir() or help() must be checked with the official docs before you can
consider using it, even if you have no explicit need to read the
official docs?

I (unfortunately) say no, which is why I have argued that
non-underscored names need to be properly deprecated. This obviously
places a nasty burden on us, though, so I don't like taking this
position. Unless we can make it clearly known through help() or
something that the official docs must be checked to know what can and
cannot be reliably used I don't think it is reasonable to force users
to not be able to rely on help() (we should probably change help() to
print a big disclaimer for anything with a leading underscore,
though).

But that doesn't mean we can't go through, fix up our names, and
deprecate the old public names; that's fair game in my book.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Michael Foord]

On Mon, Nov 8, 2010 at 1:39 PM, Michael Foord  wrote:
..

So you have a bug in the module that can only be fixed in a function you
want to deprecate?


No, I have a bug in a function that I want to deprecate.  You said I
don't need to fix it if I add a deprecation warning.  However, as far
as I know, deprecation warnings are not backported  to maintenance
branches while bug fixes are.  So the specific question is: there is a
bug in trace.find_strings() - should it be fixed in 3.1-maint?

My opinion would be:

* No we don't backport the deprecation warning
* No we don't need to fix the bug

Others may disagree. (Logic being that we won't fix the bug in 3.2, if 
we fixed it in 2.7 then we would have to fix it in 3.2. Therefore we 
shouldn't fix in 2.7.)


Michael



--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Alexander Belopolsky]

On Mon, Nov 8, 2010 at 1:39 PM, Michael Foord  wrote:
..
> So you have a bug in the module that can only be fixed in a function you
> want to deprecate?
>
No, I have a bug in a function that I want to deprecate.  You said I
don't need to fix it if I add a deprecation warning.  However, as far
as I know, deprecation warnings are not backported  to maintenance
branches while bug fixes are.  So the specific question is: there is a
bug in trace.find_strings() - should it be fixed in 3.1-maint?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Michael Foord]

On Mon, Nov 8, 2010 at 12:35 PM, Michael Foord
  wrote:
..

If you deprecate it then you don't *have* to fix bugs in it. If we know it
is used then we can't remove it without deprecation.


What about the maintenance branch?
So you have a bug in the module that can only be fixed in a function you 
want to deprecate?


It depends what approach you are taking in 3.2. If you are creating a 
new private function, in which you will fix the bug, but keeping an 
alias around to the old name so that you can deprecate it - then merely 
fixing the bug in the maintenance branch should be fine.


(If you're deprecating the function because it is unneeded then you 
don't need to fix bugs in the maintenance branch either - I guess no-one 
would complain if you did though.)


Michael

--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Alexander Belopolsky]

On Mon, Nov 8, 2010 at 12:35 PM, Michael Foord
 wrote:
..
> If you deprecate it then you don't *have* to fix bugs in it. If we know it
> is used then we can't remove it without deprecation.
>

What about the maintenance branch?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Terry Reedy]

On 11/8/2010 12:20 PM, Alexander Belopolsky wrote:

Was: [issue2001] Pydoc interactive browsing enhancement

On Sun, Nov 7, 2010 at 9:17 AM, Nick Coghlan  wrote:
..


I'd actually started typing out the command to commit this before it finally 
clicked that the patch changes public
APIs of the pydoc module in incompatible ways. Sure, they aren't documented, 
but the fact they aren't protected
by an underscore means I'm not comfortable with the idea of removing them or 
radically change their functionality
without going through a deprecation period first.



I have a similar issue with the trace module and would appreciate some
guidance on this as well.  The trace module documented API includes
just the Trace class, but the module defines several helper functions
and classes  that do not start with a leading underscore and are not
excluded from * imports by __all__.  (There is no trace.__all__.)


The trace module *appears* to be an ancient module written at a time 
(fictional or actual) when there was no '_' and '__all__' convention and 
only a loose 'public' == documented convention. The undocumented 
public-looking private stuff is a huge mess that Eli and I intentionally 
passed over in our July/August patch documenting (and fixing) the public 
stuff. I hope we included everything that should be public.


In order to warn about constants getting renamed or moved, is it 
possible to issue an off-by-default warning on module import, something like
"Trace is an ancient module with public names for many undocumented 
private constants and functions. Use of these is deprecated. See lib doc 
for more."


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Michael Foord]

Was: [issue2001] Pydoc interactive browsing enhancement

[snip...]
This suggests that trace.find_strings() should probably be preserved
or properly deprecated.  If this is the case, should we fix bugs in
it?  Note that it currently has a bug because it ignores the coding
cookie when opening python source file.  Should this be fixed?

I freely admit that I have more questions than answers, so I would
like to hear from a wider audience.


If you deprecate it then you don't *have* to fix bugs in it. If we 
know it is used then we can't remove it without deprecation.


If the function is no longer needed but we want to exclude it from the 
public API, you could create a new function in the module, with a 
leading underscore name, fix the bugs in that and deprecate the old name.




Sorry, this meant to say "if the function is *still needed* (internally 
to the module) but we want to exclude it from the API"...


This would be a good approach to clarifying the public API of standard 
library modules. At least that way we could work towards a consistent 
policy.


All the best,

Michael

Alternatively you could make the old name an alias for the new one 
with a deprecation warning applied. That way the old name does get the 
bugfixes but is still deprecated.


Michael


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk






--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Breaking undocumented API

[Michael Foord]

Was: [issue2001] Pydoc interactive browsing enhancement

[snip...]
This suggests that trace.find_strings() should probably be preserved
or properly deprecated.  If this is the case, should we fix bugs in
it?  Note that it currently has a bug because it ignores the coding
cookie when opening python source file.  Should this be fixed?

I freely admit that I have more questions than answers, so I would
like to hear from a wider audience.


If you deprecate it then you don't *have* to fix bugs in it. If we know 
it is used then we can't remove it without deprecation.


If the function is no longer needed but we want to exclude it from the 
public API, you could create a new function in the module, with a 
leading underscore name, fix the bugs in that and deprecate the old name.


Alternatively you could make the old name an alias for the new one with 
a deprecation warning applied. That way the old name does get the 
bugfixes but is still deprecated.


Michael


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--

http://www.voidspace.org.uk/

READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com