Re: [Python-Dev] Location of tests for packages

[Georg Brandl]

Am 25.01.2011 02:19, schrieb Raymond Hettinger:
> 
> On Jan 24, 2011, at 3:40 PM, Michael Foord wrote:
>> It isn't just unittest, it seems that all *test packages* are in their 
>> respective package and not Lib/test except for the json module where Raymond 
>> already moved the tests:
>> 
>>distutils/tests
>>email/test
>>ctypes/test
>>importlib/test
>>lib2to3/tests
>>sqlite3/test
>>tkinter/test
>> 
>> So I'm a little confused as to why the focus on the *unittest* test suite.
> 
> 
> There's not a focus on unittest.  Importlib should also move under Lib/test
> and when email is ready, it too should fully join the organization of
> the overall project (Doc, Lib, Lib/test, Modules, Objects, Tools).

I'm +0 on moving all tests under Lib/test -- I think the respective
maintainers of the libraries in question should have the final word,
because...

> ISTM, ctypes and disutils could almost be viewed as separate projects.
> We could ship Python without ctypes for example and we've got a policy
> against implementing the rest of library using ctypes.  The same goes
> for tkinter (it is not uncommon to have builds with it). And sqlite3 is 
> close to being completely third-party maintained.

this weakens the argument of having a consistent organization of test
modules: if one or two are allowed to have the test suite intra-package,
it doesn't matter so much any more for others.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Georg Brandl]

Am 25.01.2011 00:14, schrieb "Martin v. Löwis":
>>> This isn't a critical issue (nothing is broken) but we're a week
>>> from another release candidate, so the new Py3.2 package
>>> organization (unittest was flat in Py3.1 and its test were under
>>> Lib/test) is about to become a de-facto decision that will be hard
>>> to undo.
>> 
>> Well can we stop being melodramatic? Tests are not part of the API
>> and so they are free to move whenever we want. No need to hold a
>> release candidate for that.

Yes, let's postpone this for after the final release.

> Of course there is. Any addition or removal of files at this point has
> the chance of breaking the release process, which may fail to pick up
> files, or break in trying to pick up files that it expected to be there.
> This has happened *many* times during the alpha and beta releases of
> 3.2, so it's not at all a theoretical problem.
> 
> After the next release candidate, I'd prefer to see no changes
> whatsoever to the tree (but it's Georg's decision, of course).

I agree with both of you.  Ideally there shouldn't be any but cosmetic
changes after rc2, otherwise I'd be inclined to add an rc3 to the
release schedule.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Fred Drake]

On Mon, Jan 24, 2011 at 4:59 PM, Tres Seaver  wrote:
> It might matter if we want to enable third-party package installation
> into a namespace also used by the stdlib:  ISTR that the 'xml' package
> had such installs at one point.

Almost, but not quite.

The xml package at one point allowed itself to be overridden by
another package (_xmlplus specifically), however that was define.
Experience proved that this was a mistake.

"Namespace packages", as originally defined by setuptools and applied
for the hurry, zc, and zope packages (and many others), are a very
different thing than what was done for the xml/_xmlplus package, and
have proven significantly more useful and usable.

While I heartily approve of "namespace packages" of that sort, I see
no reason to support installing into the same package namespace as the
standard library.  The primary disadvantage I see is that it would be
too easy to foster confusion over what's in the standard library among
newcomers.


  -Fred

--
Fred L. Drake, Jr.    
"A storm broke loose in my mind."  --Albert Einstein
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Toshio Kuratomi]

On Thu, Jan 20, 2011 at 03:27:08PM -0500, Glyph Lefkowitz wrote:
> 
> On Jan 20, 2011, at 11:46 AM, Guido van Rossum wrote:
> Same here. *Most* code will never be shared, or will only be shared
> between users in the same community. When it goes wrong it's also a
> learning opportunity. :-)
> 
> 
> Despite my usual proclivity for being contrarian, I find myself in agreement
> here.  Linux users with locales that don't specify UTF-8 frankly _should_ have
> to deal with all kinds of nastiness until they can transcode their 
> filesystems.
>  MacOS and Windows both have a "right" answer here and your third-party tools
> shouldn't create mojibake in your filenames.
> 
However, if this is the consensus, it makes a lot more sense to pick utf-8
as *the* encoding for python module filenames on Linux.

Why UTF-8:

* UTF-8 can cover the whole range of unicode whereas most (all?) other
  locale friendly encodings cannot.
* UTF-8 is becoming a standard for Linux distributions whether or not Linux
  users are adopting it.
* Third party tools are gaining support for UTF-8 even when they aren't
  gaining support for generic encodings (If I read the spec on zip
  correctly, this is actually what's happening there).

Why not locale:
* Relying on locale is simply not portable.  If nothing prevents people from
  distributing a unicode filename then they will go ahead and do so.  If
  the result works (say, because it's utf-8 and 80% of the Linux userbase is
  using utf-8) then it will get packaged and distributed and people won't
  know that it's a problem until someone with a non-utf-8 locale decids to
  use it.
* Mixing of modules from different locales won't work.  Suppose that the
  system python installs the previous module.  The local site has other
  modules that it has installed using a different filename encoding.
  The users at the site will find that either one or hte other of the two
  modules won't work.
* Because of the portability problems you have no choice but to tell people
  not to distribute python modules with non-ASCII names.  This makes the use
  of unicode names second class indefintely (until the kernel devs decide
  that they're wrong to not enforce a filesystem encoding or Linux becomes
  irrelevant as a platform).
* If you can pick a set of encodings that are valid (utf-8 for Linux and
  MacOS, wide unicode for windows [I get the feeling from other parts of the
  conversation that Windows won't be so lucky, though]) tools to convert
  python names become easier to write.  If you restrict it far enough, you
  could even write tools/importers that automatically do the detection.

PS: Sorry for not replying immediately, the team I'm on is dealing with an
issue at my work and I'm also preparing for a conference later this week.

-Toshio


pgpq1C0qGW77C.pgp
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Raymond Hettinger]

On Jan 24, 2011, at 3:40 PM, Michael Foord wrote:
> It isn't just unittest, it seems that all *test packages* are in their 
> respective package and not Lib/test except for the json module where Raymond 
> already moved the tests:
> 
>distutils/tests
>email/test
>ctypes/test
>importlib/test
>lib2to3/tests
>sqlite3/test
>tkinter/test
> 
> So I'm a little confused as to why the focus on the *unittest* test suite.


There's not a focus on unittest.  Importlib should also move under Lib/test
and when email is ready, it too should fully join the organization of
the overall project (Doc, Lib, Lib/test, Modules, Objects, Tools).

ISTM, ctypes and disutils could almost be viewed as separate projects.
We could ship Python without ctypes for example and we've got a policy
against implementing the rest of library using ctypes.  The same goes
for tkinter (it is not uncommon to have builds with it). And sqlite3 is 
close to being completely third-party maintained.

In contrast, the unittest module and importlib belong with the core distro.

So, I'm thinking that there were some precedents in cases where there
was a really good reason for separating the project (we don't even include
tkinter docs in our doc build), but that we should maintain a strong
preference for keeping the overall project organization intact.

ElementTree was fully folded into the project.  I think we should
follow that precedent and avoid balkanizing the python source
into many little project subtrees (worlds within a world).


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 393: Flexible String Representation

[David Malcolm]

On Mon, 2011-01-24 at 21:17 +0100, "Martin v. Löwis" wrote:

... snip ...

> I'd like to propose PEP 393, which takes a different approach,
> addressing both problems simultaneously: by getting a flexible
> representation (one that can be either 1, 2, or 4 bytes), we can
> support the full range of Unicode on all systems, but still use
> only one byte per character for strings that are pure ASCII (which
> will be the majority of strings for the majority of users).

There was some discussion about this at PyCon 2010, where we referred to
it casually as "Pay-as-you-go unicode"

... snip ...

> - str: shortest-form representation of the unicode string; the lower
>   two bits of the pointer indicate the specific form:
>   01 => 1 byte (Latin-1); 11 => 2 byte (UCS-2); 11 => 4 byte (UCS-4);
Repetition of "11"; I'm guessing that the 2byte/UCS-2 should read "10",
so that they give the width of the char representation.

>   00 => null pointer

Naturally this assumes that all pointers are at least 4-byte aligned (so
that they can be masked off).  I assume that this is sane on every
platform that Python supports, but should it be spelled out explicitly
somewhere in the PEP?

> 
>   The string is null-terminated (in its respective representation).
> - hash, state: same as in Python 3.2
> - utf8_length, utf8: UTF-8 representation (null-terminated)
If this is to share its buffer with the "str" representation for the
Latin-1 case, then I take it this ptr will typically be (str & ~4) ?
i.e. only "str" has the low-order-bit type info.

> - wstr_length, wstr: representation in platform's wchar_t
>   (null-terminated). If wchar_t is 16-bit, this form may use surrogate
>   pairs (in which cast wstr_length differs form length).
> 
> All three representations are optional, although the str form is
> considered the canonical representation which can be absent only
> while the string is being created.

Spelling out the meaning of "optional":
  does this mean that the relevant ptr is NULL; if so, if utf8 is null,
is utf8_length undefined, or is it some dummy value?  (i.e. is the
pointer the first thing to check before we know if utf8_length is
meaningful?); similar consideration for the wstr representation.


> The Py_UNICODE type is still supported but deprecated. It is always
> defined as a typedef for wchar_t, so the wstr representation can double
> as Py_UNICODE representation.
> 
> The str and utf8 pointers point to the same memory if the string uses
> only ASCII characters (using only Latin-1 is not sufficient). The str
...though the ptrs are non-equal for this case, as noted above, as "str"
has an 0x1 typecode.

> and wstr pointers point to the same memory if the string happens to
> fit exactly to the wchar_t type of the platform (i.e. uses some
> BMP-not-Latin-1 characters if sizeof(wchar_t) is 2, and uses some
> non-BMP characters if sizeof(wchar_t) is 4).
> 
> If the string is created directly with the canonical representation
> (see below), this representation doesn't take a separate memory block,
> but is allocated right after the PyUnicodeObject struct.

Is the idea to do pointer arithmentic when deleting the PyUnicodeObject
to determine if the ptr is in that location, and not delete it if it is,
or is there some other way of determining whether the pointers need
deallocating?  If the former, is this embedding an assumption that the
underlying allocator couldn't have allocated a buffer directly adjacent
to the PyUnicodeObject.  I know that GNU libc's malloc/free
implementation has gaps of two machine words between each allocation;
off the top of my head I'm not sure if the optimized Object/obmalloc.c
allocator enforces such gaps.

... snip ...

Extra section:

GDB Debugging Hooks
---
Tools/gdb/libpython.py contains debugging hooks that embed knowledge
about the internals of CPython's data types, include PyUnicodeObject
instances.  It will need to be slightly updated to track the change.

(I can do that change if need be; it shouldn't be too hard).



Hope this is helpful
Dave

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Michael Foord]

On 24/01/2011 23:14, "Martin v. Löwis" wrote:

This isn't a critical issue (nothing is broken) but we're a week
from another release candidate, so the new Py3.2 package
organization (unittest was flat in Py3.1 and its test were under
Lib/test) is about to become a de-facto decision that will be hard
to undo.

Well can we stop being melodramatic? Tests are not part of the API
and so they are free to move whenever we want. No need to hold a
release candidate for that.

Of course there is. Any addition or removal of files at this point has
the chance of breaking the release process, which may fail to pick up
files, or break in trying to pick up files that it expected to be there.
This has happened *many* times during the alpha and beta releases of
3.2, so it's not at all a theoretical problem.

After the next release candidate, I'd prefer to see no changes
whatsoever to the tree (but it's Georg's decision, of course).


What Antoine meant is that we could make the change for 3.2.1 and don't 
need to delay 3.2.


Michael


Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Michael Foord]

On 24/01/2011 19:46, Raymond Hettinger wrote:

Right now, the tests for the unittest package are under the package directory 
instead of Lib/test where we have most of the other tests.

There are some other packages that do the same thing, each for their own reason.

I think we should develop a strong preference for tests going under Lib/test 
unless there is a very compelling reason.  We already have a similar preference 
for all Docs going under Doc/ and that has not proved to be an issue with any 
package maintainer.

* The Windows distro has an install option to exclude Lib/test.  The currrent 
situation with unittest works against it.
* The commingling of tests with the regular code is making it more difficult to 
grep code while excluding tests.
* Having packages create their little worlds within world is making it more 
difficult to find things.
* For regrtest to work, there still needs to be some file in Lib/test that 
dispatches to the alternate test directory.

This isn't a critical issue (nothing is broken) but we're a week from another 
release candidate, so the new Py3.2 package organization (unittest was flat in 
Py3.1 and its test were under Lib/test) is about to become a de-facto decision 
that will be hard to undo.


The tests are already under unittest in 2.7 so that change isn't "new". 
Moving the tests now makes it harder to maintain them (patches to 3.2 
won't apply to 2.7). This is discussed in issue 10572.


http://bugs.python.org/issue10572

It isn't just unittest, it seems that all *test packages* are in their 
respective package and not Lib/test except for the json module where 
Raymond already moved the tests:


distutils/tests
email/test
ctypes/test
importlib/test
lib2to3/tests
sqlite3/test
tkinter/test

So I'm a little confused as to why the focus on the *unittest* test suite.

Brett has expressed a willingness to move the importlib tests under 
Lib/test and R David Murray would *like* to move the email tests there 
(but hasn't). Barry is -0 and so am I. It generally makes a few things 
slightly harder for me but not much. If we make a general policy 
decision to move all package tests out of their packages and into 
Lib/test (and actually do it) then fine, but I'm not overjoyed with a 
unilateral decision that unittest is special in this regard... :-)


All the best,

Michael

I recommend moving it under Lib/test before everything is set in stone.


Raymond


P.S.  I've discussed this with Michael and his preference is against going back 
to the Py3.1 style where the tests were under Lib/test.  He thinks the current 
tree makes it easier to sync with Py2.7 and the unittest2 third-party module.  
Also, he likes grepping the regular source and tests all at once.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Antoine Pitrou]

Le mardi 25 janvier 2011 à 00:14 +0100, "Martin v. Löwis" a écrit :
> >> This isn't a critical issue (nothing is broken) but we're a week
> >> from another release candidate, so the new Py3.2 package
> >> organization (unittest was flat in Py3.1 and its test were under
> >> Lib/test) is about to become a de-facto decision that will be hard
> >> to undo.
> > 
> > Well can we stop being melodramatic? Tests are not part of the API
> > and so they are free to move whenever we want. No need to hold a
> > release candidate for that.
> 
> Of course there is. Any addition or removal of files at this point has
> the chance of breaking the release process, which may fail to pick up
> files, or break in trying to pick up files that it expected to be there.
> This has happened *many* times during the alpha and beta releases of
> 3.2, so it's not at all a theoretical problem.

My point was that these changes can take place after 3.2 (both final and
rc).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 393: Flexible String Representation

[Antoine Pitrou]

Le mardi 25 janvier 2011 à 00:07 +0100, "Martin v. Löwis" a écrit :
> >> I'd like to propose PEP 393, which takes a different approach,
> >> addressing both problems simultaneously: by getting a flexible
> >> representation (one that can be either 1, 2, or 4 bytes), we can
> >> support the full range of Unicode on all systems, but still use
> >> only one byte per character for strings that are pure ASCII (which
> >> will be the majority of strings for the majority of users).
> > 
> > For this kind of experiment, I think a concrete attempt at implementing
> > (together with performance/memory savings numbers) would be much more
> > useful than an abstract proposal.
> 
> I partially agree. An implementation is certainly needed, but there is
> nothing wrong (IMO) with designing the change before implementing it.
> Also, several people have offered to help with the implementation, so
> we need to agree on a specification first (which is actually cheaper
> than starting with the implementation only to find out that people
> misunderstood each other).

I'm not sure it's really cheaper. When implementing you will probably
find out that it makes more sense to change the meaning of some fields,
add or remove some, etc. You will also want to try various tweaks since
the whole point is to lighten the footprint of unicode strings in common
workloads.

So, the only criticism I have, intuitively, is that the unicode
structure seems to become a bit too large. For example, I'm not sure you
need a generic (pointer, size) pair in addition to the
representation-specific ones.

Incidentally, to slightly reduce the overhead the unicode objects,
there's this proposal: http://bugs.python.org/issue1943

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Martin v. Löwis]

>> This isn't a critical issue (nothing is broken) but we're a week
>> from another release candidate, so the new Py3.2 package
>> organization (unittest was flat in Py3.1 and its test were under
>> Lib/test) is about to become a de-facto decision that will be hard
>> to undo.
> 
> Well can we stop being melodramatic? Tests are not part of the API
> and so they are free to move whenever we want. No need to hold a
> release candidate for that.

Of course there is. Any addition or removal of files at this point has
the chance of breaking the release process, which may fail to pick up
files, or break in trying to pick up files that it expected to be there.
This has happened *many* times during the alpha and beta releases of
3.2, so it's not at all a theoretical problem.

After the next release candidate, I'd prefer to see no changes
whatsoever to the tree (but it's Georg's decision, of course).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Brett Cannon]

On Mon, Jan 24, 2011 at 11:18, Georg Brandl  wrote:
> Am 24.01.2011 20:04, schrieb Raymond Hettinger:
>> Looking at http://docs.python.org/dev/library/html.html#module-html it would
>> appear that we've created a new module with a single trivial function.
>>
>> In reality, there was already a python package, html, that served to group
>> two loosely related modules, html.parser and html.entities.
>>
>> ISTM, that if we're going to use python packages as "namespace containers"
>> for categorizing modules, then the top level __init__ namespace should be
>> left empty.
>>
>> Before the placement of html.escape() becomes set in stone, I think we should
>> consider putting it somewhere else.
>
> To be honest, I don't see the issue.  I don't see stdlib packages as
> "namespace containers", but rather as a nice way of structuring functionality.
> And remember that flat is better than nested -- why should escape() be put
> away into a new submodule?

Importlib also acts as a precedent with importlib.import_module(). I
honestly don't feel the need to treat packages as a namespace
explicitly (but then again I also disagree with the argument that
__init__.py needs to be left empty).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 393: Flexible String Representation

[Martin v. Löwis]

>> I'd like to propose PEP 393, which takes a different approach,
>> addressing both problems simultaneously: by getting a flexible
>> representation (one that can be either 1, 2, or 4 bytes), we can
>> support the full range of Unicode on all systems, but still use
>> only one byte per character for strings that are pure ASCII (which
>> will be the majority of strings for the majority of users).
> 
> For this kind of experiment, I think a concrete attempt at implementing
> (together with performance/memory savings numbers) would be much more
> useful than an abstract proposal.

I partially agree. An implementation is certainly needed, but there is
nothing wrong (IMO) with designing the change before implementing it.
Also, several people have offered to help with the implementation, so
we need to agree on a specification first (which is actually cheaper
than starting with the implementation only to find out that people
misunderstood each other).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Martin v. Löwis]

> If that pattern is a goal, having all versions of the namespace's
> __init__.py empty of anything but the __path__-munging majyk /
> boilerplate is required to make such installs work regardless of the
> order of PYTHONPATH.

With PEP 382, having extensible packages won't contradict to having
a non-trivial __init__.py, and no __path__-munging will be necessary.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Terry Reedy]

On 1/24/2011 2:18 PM, Georg Brandl wrote:

Am 24.01.2011 20:04, schrieb Raymond Hettinger:

Looking at http://docs.python.org/dev/library/html.html#module-html
it would appear that we've created a new module with a single
trivial function.

In reality, there was already a python package, html, that served
to group two loosely related modules, html.parser and
html.entities.

ISTM, that if we're going to use python packages as "namespace
containers" for categorizing modules, then the top level __init__
namespace should be left empty.

Before the placement of html.escape() becomes set in stone, I think
we should consider putting it somewhere else.


To be honest, I don't see the issue.  I don't see stdlib packages as
"namespace containers", but rather as a nice way of structuring
functionality. And remember that flat is better than nested -- why
should escape() be put away into a new submodule?

At least you'll need to let us know where you would rather put that
function.


I would put in html.entities, which is also sparse, as it seems to me 
vaguely related.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Benjamin Peterson]

2011/1/24 Éric Araujo :
>> Right now, the tests for the unittest package are under the
>> package directory instead of Lib/test where we have most of the
>> other tests.
>>
>> There are some other packages that do the same thing, each for
>> their own reason.
>
> The corresponding bug report is #10572 (opened by Michael Foord).
>
> R. David Murray was +1 for moving email tests, Barry deferred to him,
> Brett was +0 for importlib, and I was ±0 for distutils.  Maintainers of
> ctypes, json, lib2to3 and sqlite3 haven’t yet expressed themselves.

I prefer lib2to3 tests to stay in lib2to3/.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 393: Flexible String Representation

[Antoine Pitrou]

On Mon, 24 Jan 2011 21:17:34 +0100
"Martin v. Löwis"  wrote:
> I have been thinking about Unicode representation for some time now.
> This was triggered, on the one hand, by discussions with Glyph Lefkowitz
> (who complained that his server app consumes too much memory), and Carl
> Friedrich Bolz (who profiled Python applications to determine that
> Unicode strings are among the top consumers of memory in Python).
> On the other hand, this was triggered by the discussion on supporting
> surrogates in the library better.
> 
> I'd like to propose PEP 393, which takes a different approach,
> addressing both problems simultaneously: by getting a flexible
> representation (one that can be either 1, 2, or 4 bytes), we can
> support the full range of Unicode on all systems, but still use
> only one byte per character for strings that are pure ASCII (which
> will be the majority of strings for the majority of users).

For this kind of experiment, I think a concrete attempt at implementing
(together with performance/memory savings numbers) would be much more
useful than an abstract proposal. It is hard to judge the concrete
effects of the changes you are proposing, even though they might (or
not) make sense in theory. For example, you are adding a lot of
constant overhead to every unicode object, even very small ones, which
might be detrimental. Also, accessing the unicode object's payload
can become quite a bit more cumbersome. Only implementing can tell how
much this is workable in practice.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Antoine Pitrou]

On Mon, 24 Jan 2011 11:46:45 -0800
Raymond Hettinger  wrote:
> 
> This isn't a critical issue (nothing is broken) but we're a week from another 
> release candidate, so the new Py3.2 package organization (unittest was flat 
> in Py3.1 and its test were under Lib/test) is about to become a de-facto 
> decision that will be hard to undo.

Well can we stop being melodramatic? Tests are not part of the API and
so they are free to move whenever we want. No need to hold a release
candidate for that.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 01/24/2011 03:14 PM, Fred Drake wrote:
> On Mon, Jan 24, 2011 at 2:04 PM, Raymond Hettinger
>  wrote:
>> ISTM, that if we're going to use python packages as "namespace containers" 
>> for
>> categorizing modules, then the top level __init__ namespace should be left 
>> empty.
> 
> This is only an issue if the separate components are distributed
> separately; for the standard library, we're not using it as a
> namespace package in the same sense that is done with (for example)
> the "zope" package.

It might matter if we want to enable third-party package installation
into a namespace also used by the stdlib:  ISTR that the 'xml' package
had such installs at one point.

If that pattern is a goal, having all versions of the namespace's
__init__.py empty of anything but the __path__-munging majyk /
boilerplate is required to make such installs work regardless of the
order of PYTHONPATH.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk099jAACgkQ+gerLs4ltQ7e4gCfbYJE8d8bNrX19zrzC4xvfA9Y
KkQAnA7niExvMqXtUBD/XwzZZ9EzHcBm
=/Q/Y
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Barry Warsaw]

On Jan 24, 2011, at 11:46 AM, Raymond Hettinger wrote:

>P.S.  I've discussed this with Michael and his preference is against going
>back to the Py3.1 style where the tests were under Lib/test.  He thinks the
>current tree makes it easier to sync with Py2.7 and the unittest2 third-party
>module.  Also, he likes grepping the regular source and tests all at once.

Which seem like compelling reasons to keep things the way they are for
unittest, in addition to the fact that we're already in RC for 3.2, so you
would need RM approval to make such a change this late in the process.

I agree that it's not ideal, but for certain packages that are also
distributed separately, it can be much easier to keep the tests with the code,
and I'm inclined to defer to the primary maintainer's preference.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Martin v. Löwis]

Am 24.01.2011 16:39, schrieb Victor Stinner:
> Le lundi 24 janvier 2011 11:35:22, Stephen J. Turnbull a écrit :
>> ... VFAT-formatted file systems and Shift JIS file names ...
> 
> I missed something: VFAT stores filenames as unicode (whereas FAT only 
> supports byte filenames). Well, VFAT stores filenames twice: as a 8+3 byte 
> strings and as a 255 unicode (UTF-16-LE) string (UTF-16-LE).

Stephen may not have meant VFAT. Instead, he might have meant FAT32,
or, more likely, exFAT. VFAT is patented by Microsoft, so vendors of
devices using flash memory cards often don't support VFAT.

In any case, file names are encoded in the OEM code page even on VFAT.

> On which OS do you access this VFAT file system? On Windows, you have two 
> APIs: bytes (*A) and wide character (*W). If you use the wide character, 
> there 
> is explicit encoding at all.

Right ("no explicit encoding"). However, this is actually where things
can go wrong: Windows needs to guess the file system, and will guess it
uses the OEM code page. If the device writing the file system uses a
different OEM code age than the Windows installation reading it, you
get moji-bake.

This will actually happen with the *A APIs as well: they do *not* give
you the file name from disk. Instead, Windows converts the OEM
characters on disk to Unicode, and then the Unicode characters to the
ANSI code page.

> Linux has two mount options to control unicode on 
> a VFAT filesystem: "codepage" for the byte filenames (use Shift JIS here) and 
> "iocharset" for the unicode filenames (I don't understand this option). 
> Anyway, both systems support unicode filenames.

Linux doesn't support "unicode file names". Instead, it can support
UTF-8. As Oleg explains: you need one encoding for the bytes on disk
(to know what they mean, when converted to Unicode), and one encoding
to then convert the "abstract" unicode to bytes again to present to
the application. This is similar to how *A works on Windows.

The iocharset is needed even if the file system is known to use UTF-16
(say, NTFS, VFAT, or Joliet).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Location of tests for packages

[Fred Drake]

On Mon, Jan 24, 2011 at 2:46 PM, Raymond Hettinger
 wrote:
> P.S.  I've discussed this with Michael and his preference is against
> going back to the Py3.1 style where the tests were under Lib/test.  He
> thinks the current tree makes it easier to sync with Py2.7 and the
> unittest2 third-party module.  Also, he likes grepping the regular source
> and tests all at once.

I'm with Michael on this.

-1 on pushing all the tests into Lib/test/.


  -Fred

--
Fred L. Drake, Jr.    
"A storm broke loose in my mind."  --Albert Einstein
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 393: Flexible String Representation

[Martin v. Löwis]

I have been thinking about Unicode representation for some time now.
This was triggered, on the one hand, by discussions with Glyph Lefkowitz
(who complained that his server app consumes too much memory), and Carl
Friedrich Bolz (who profiled Python applications to determine that
Unicode strings are among the top consumers of memory in Python).
On the other hand, this was triggered by the discussion on supporting
surrogates in the library better.

I'd like to propose PEP 393, which takes a different approach,
addressing both problems simultaneously: by getting a flexible
representation (one that can be either 1, 2, or 4 bytes), we can
support the full range of Unicode on all systems, but still use
only one byte per character for strings that are pure ASCII (which
will be the majority of strings for the majority of users).

You'll find the PEP at

http://www.python.org/dev/peps/pep-0393/

For convenience, I include it below.

Regards,
Martin

PEP: 393
Title: Flexible String Representation
Version: $Revision: 88168 $
Last-Modified: $Date: 2011-01-24 21:14:21 +0100 (Mo, 24. Jan 2011) $
Author: Martin v. Löwis 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 24-Jan-2010
Python-Version: 3.3
Post-History:

Abstract


The Unicode string type is changed to support multiple internal
representations, depending on the character with the largest Unicode
ordinal (1, 2, or 4 bytes). This will allow a space-efficient
representation in common cases, but give access to full UCS-4 on all
systems. For compatibility with existing APIs, several representations
may exist in parallel; over time, this compatibility should be phased
out.

Rationale
=

There are two classes of complaints about the current implementation
of the unicode type: on systems only supporting UTF-16, users complain
that non-BMP characters are not properly supported. On systems using
UCS-4 internally (and also sometimes on systems using UCS-2), there is
a complaint that Unicode strings take up too much memory - especially
compared to Python 2.x, where the same code would often use ASCII
strings (i.e. ASCII-encoded byte strings). With the proposed approach,
ASCII-only Unicode strings will again use only one byte per character;
while still allowing efficient indexing of strings containing non-BMP
characters (as strings containing them will use 4 bytes per
character).

One problem with the approach is support for existing applications
(e.g. extension modules). For compatibility, redundant representations
may be computed. Applications are encouraged to phase out reliance on
a specific internal representation if possible. As interaction with
other libraries will often require some sort of internal
representation, the specification choses UTF-8 as the recommended way
of exposing strings to C code.

For many strings (e.g. ASCII), multiple representations may actually
share memory (e.g. the shortest form may be shared with the UTF-8 form
if all characters are ASCII). With such sharing, the overhead of
compatibility representations is reduced.

Specification
=

The Unicode object structure is changed to this definition::

  typedef struct {
PyObject_HEAD
Py_ssize_t length;
void *str;
Py_hash_t hash;
int state;
Py_ssize_t utf8_length;
void *utf8;
Py_ssize_t wstr_length;
void *wstr;
  } PyUnicodeObject;

These fields have the following interpretations:

- length: number of code points in the string (result of sq_length)
- str: shortest-form representation of the unicode string; the lower
  two bits of the pointer indicate the specific form:
  01 => 1 byte (Latin-1); 11 => 2 byte (UCS-2); 11 => 4 byte (UCS-4);
  00 => null pointer

  The string is null-terminated (in its respective representation).
- hash, state: same as in Python 3.2
- utf8_length, utf8: UTF-8 representation (null-terminated)
- wstr_length, wstr: representation in platform's wchar_t
  (null-terminated). If wchar_t is 16-bit, this form may use surrogate
  pairs (in which cast wstr_length differs form length).

All three representations are optional, although the str form is
considered the canonical representation which can be absent only
while the string is being created.

The Py_UNICODE type is still supported but deprecated. It is always
defined as a typedef for wchar_t, so the wstr representation can double
as Py_UNICODE representation.

The str and utf8 pointers point to the same memory if the string uses
only ASCII characters (using only Latin-1 is not sufficient). The str
and wstr pointers point to the same memory if the string happens to
fit exactly to the wchar_t type of the platform (i.e. uses some
BMP-not-Latin-1 characters if sizeof(wchar_t) is 2, and uses some
non-BMP characters if sizeof(wchar_t) is 4).

If the string is created directly with the canonical representation
(see below), this representation doesn't take a separate memory block,
but is allocated right after the PyUnicodeObject struct.

String C

Re: [Python-Dev] Location of tests for packages

[Éric Araujo]

> Right now, the tests for the unittest package are under the
> package directory instead of Lib/test where we have most of the
> other tests.
> 
> There are some other packages that do the same thing, each for
> their own reason.

The corresponding bug report is #10572 (opened by Michael Foord).

R. David Murray was +1 for moving email tests, Barry deferred to him,
Brett was +0 for importlib, and I was ±0 for distutils.  Maintainers of
ctypes, json, lib2to3 and sqlite3 haven’t yet expressed themselves.

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Fred Drake]

On Mon, Jan 24, 2011 at 2:04 PM, Raymond Hettinger
 wrote:
> ISTM, that if we're going to use python packages as "namespace containers" for
> categorizing modules, then the top level __init__ namespace should be left 
> empty.

This is only an issue if the separate components are distributed
separately; for the standard library, we're not using it as a
namespace package in the same sense that is done with (for example)
the "zope" package.


  -Fred

--
Fred L. Drake, Jr.    
"A storm broke loose in my mind."  --Albert Einstein
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's new 2.x in 3.x docs.

[Georg Brandl]

Am 23.01.2011 02:48, schrieb Nick Coghlan:
> On Sun, Jan 23, 2011 at 7:23 AM, Raymond Hettinger
>  wrote:
>> On Jan 22, 2011, at 11:04 AM, Terry Reedy wrote:
>>
>>> The 3.x docs mostly started fresh with 3.0. The major exception is the 
>>> What's new section, which goes back to 2.0. The 2.x stuff comprises about 
>>> 650KB in the repository and whatever that translates into in the 
>>> distribution.. I cannot imagine that anyone who only has 3.x and no 2.x 
>>> version would have any interest in the 2.x history. And of course, the 
>>> complete 2.x history will always be available with the latest 2.7.z. And 
>>> the cover page for 3.x could even say so and include a link. So why not 
>>> remove it from the 3.2 release (and have two separate pages for the online 
>>> version)?
>>
>> I think there is value in the older whatsnew docs.  The provide a readable 
>> introduction to various features and nicely augment the plain docs which can 
>> be a little dry.
>>
>> +1 for keeping the links as-is.  Removing them takes away a resource and 
>> gains nothing.
> 
> They're also a useful resource when developing compatibility guides
> for projects that target older versions (including ones that support
> py3k via 2to3).
> 
> With the latest 3.x release always being at the top, I agree with
> Raymond that retaining the history is a better option.

Agreed.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Location of tests for packages

[Raymond Hettinger]

Right now, the tests for the unittest package are under the package directory 
instead of Lib/test where we have most of the other tests.

There are some other packages that do the same thing, each for their own reason.

I think we should develop a strong preference for tests going under Lib/test 
unless there is a very compelling reason.  We already have a similar preference 
for all Docs going under Doc/ and that has not proved to be an issue with any 
package maintainer.

* The Windows distro has an install option to exclude Lib/test.  The currrent 
situation with unittest works against it.
* The commingling of tests with the regular code is making it more difficult to 
grep code while excluding tests.
* Having packages create their little worlds within world is making it more 
difficult to find things. 
* For regrtest to work, there still needs to be some file in Lib/test that 
dispatches to the alternate test directory.

This isn't a critical issue (nothing is broken) but we're a week from another 
release candidate, so the new Py3.2 package organization (unittest was flat in 
Py3.1 and its test were under Lib/test) is about to become a de-facto decision 
that will be hard to undo.

I recommend moving it under Lib/test before everything is set in stone.


Raymond


P.S.  I've discussed this with Michael and his preference is against going back 
to the Py3.1 style where the tests were under Lib/test.  He thinks the current 
tree makes it easier to sync with Py2.7 and the unittest2 third-party module.  
Also, he likes grepping the regular source and tests all at once.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Georg Brandl]

Am 24.01.2011 20:04, schrieb Raymond Hettinger:
> Looking at http://docs.python.org/dev/library/html.html#module-html it would
> appear that we've created a new module with a single trivial function.
> 
> In reality, there was already a python package, html, that served to group
> two loosely related modules, html.parser and html.entities.
> 
> ISTM, that if we're going to use python packages as "namespace containers"
> for categorizing modules, then the top level __init__ namespace should be
> left empty.
> 
> Before the placement of html.escape() becomes set in stone, I think we should
> consider putting it somewhere else.

To be honest, I don't see the issue.  I don't see stdlib packages as
"namespace containers", but rather as a nice way of structuring functionality.
And remember that flat is better than nested -- why should escape() be put
away into a new submodule?

At least you'll need to let us know where you would rather put that function.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Keeping __init__.py empty for Python packages used for module grouping.

[Raymond Hettinger]

Looking at http://docs.python.org/dev/library/html.html#module-html it would 
appear that we've created a new module with a single trivial function.

In reality, there was already a python package, html, that served to group two 
loosely related modules, html.parser and html.entities.

ISTM, that if we're going to use python packages as "namespace containers" for 
categorizing modules, then the top level __init__ namespace should be left 
empty.

Before the placement of html.escape() becomes set in stone, I think we should 
consider putting it somewhere else.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Beta version of the new devguide

[Brett Cannon]

On Mon, Jan 24, 2011 at 07:46, Antoine Pitrou  wrote:
> On Sat, 22 Jan 2011 17:08:00 -0800
> Brett Cannon  wrote:
>>
>> Two, what should the final URL be? Georg picked the current one and I
>> am happy with it.
>
> Ditto for me.
>
>> Three, where should it be linked from? docs.python.org homepage?

Either there and/or the www.python.org homepage.

>> Four, what to do with www.python.org/dev/? Redirect for all the pages?
>
> Right, this whole area (wpo/dev) looks obsolete to me. The devguide
> allows us to easily edit and improve development-related docs, which is
> great! It should be accessible easily from the main site. Perhaps
> "core development" should be renamed "contributing" and redirect to the
> devguide. Also, the submenu displayed below "core development" can be
> trimmed dramatically.

There actually shouldn't be anything at python.org/dev that is useful
which has not been rewritten or linked to from the devguide. So that
whole page can be heavily gutted to the point of probably being
nothing more than a link to the devguide and a link to the PEP 0. But
I will hold off on the gutting until the devguide is "released";
probably end of the week.

>
> (then there's the question of whether the devguide should be
> exhaustive; should it contain reference-like material about
> all aspects of core development?)

That's where the balancing act comes in. If we get too exhaustive then
we have to constantly update the docs anytime we make a change. If we
leave too loose then someone is going to come along and potentially
waste some time on something because they didn't realize what they
should have been doing.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Beta version of the new devguide

[Brett Cannon]

On Mon, Jan 24, 2011 at 07:33, Nick Coghlan  wrote:
> On Mon, Jan 24, 2011 at 9:29 PM, Antoine Pitrou  wrote:
>> On Mon, 24 Jan 2011 20:33:07 +1000
>> Nick Coghlan  wrote:
>>> On Mon, Jan 24, 2011 at 6:22 AM, Brett Cannon  wrote:
>>> >> In "Getting Set Up" it describes how to build a pydebug build. Is that
>>> >> really necessary for those who plan only to contribute by working on
>>> >> pure Python code?
>>> >>
>>> >
>>> > Yes, there is actually a laundry list of reasons even people only
>>> > working on the stdlib should use a pydebug build.
>>>
>>> And one big reason why I don't unless I have a specific need to check
>>> something with it - it makes the already quite long running time for
>>> the full test suite take even longer :)
>>
>> Please try the -j option to regrtest.
>
> While I must admit I'm still not in the habit of running tests in
> parallel, that's a substantial speed improvement regardless of build
> type, so a non-debug build is still noticeably faster.
>
> release (with -j4): 2 min 25 sec (3 min wall clock time)
> pydebug (with -j4): 4 min 43 sec (10 min wall clock time)
>

If you thinks that's slow, trying running it under coverage single-threaded. =)

> Given that I typically *don't* need the extra info from a debug build
> to analyse problems and a full configure and rebuild cycle takes less
> time than a single pydebug test run, I'll happily stick with the much
> faster test execution that comes from using a release build.
>

I'm not going to drag on arguing this point, but there is more to
pydebug builds than some debug info when working in the C code. For
instance, pure Python code can still trigger problems indirectly in C
code which gets picked up by a pydebug. You also have ResourceWarnings
now which are almost exclusively triggered by pure Python code.

My point is there is more to a pydebug build than just direct
debugging support for C code. But if running the test suite w/o a
debug build is what it takes to get people to run the test suite I
will take that over not running it at all.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (no subject)

[Brett Cannon]

Bug reports should be filed at bugs.python.org

On Mon, Jan 24, 2011 at 08:39, Stefan Spoettl  wrote:
> Using:
> Python 2.7.0+ (r27:82500, Sep 15 2010, 18:14:55)
> [GCC 4.4.5] on linux2
> (Ubuntu 10.10)
> Method to reproduce error:
> 1. Defining a module which is later imported by another:
> -
> class SomeThing:
>     def __init__(self):
>         self.variable = 'Where is my bytecode?'
>     def deliver(self):
>         return self.variable
>
> if __name__ == '__main__':
>     obj = SomeThing()
>     print obj.deliver()
> -
> 2. Run this module:
> Output of the Python Shell: Where is my bytecode?
>                                                    >>>
> 3. Defining the importing module:
> -
> class UseSomeThing:
>     def __init__(self, something):
>         self.anything = something
>     def giveanything(self):
>         return self.anything
>
> if __name__ == '__main__':
>     anything = UseSomeThing(SomeThing.SomeThing().deliver()).giveanything()
>     print anything
> -
> 4. Run this module:
> Output of the Python Shell: Where is my bytecode
>                                                     >>>
> (One can find SomeThing.pyc on the disc.)
> 5. Changing the imported module:
> -
> class SomeThing:
>     def __init__(self):
>         self.variable = 'What the hell is this? It could not be Python!'
>     def deliver(self):
>         return self.variable
>
> if __name__ == '__main__':
>     obj = SomeThing()
>     print obj.deliver()
> -
> 6. Run the changed module:
> Output of the Python Shell: What the hell is this? It could not be Python!
>                                                    >>>
> 7. Run the importing module again:
> Output of the Python Shell: Where is my bytecode?
>                                                    >>>
> 8. Deleting the bytecode of the imported module makes no effect!
> Remark: I think that I have observed yesterday late night a similar effect
> on Windows XP
> with Python 2.7.1 and Python 3.1.3. But when I have tried it out today in
> the morning the
> error hasn't appeared. So it may be that the Python interpreter isn't
> working correctly only
> on Ubuntu 10.10.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (no subject)

[Oleg Broytman]

On Mon, Jan 24, 2011 at 04:39:54PM +, Stefan Spoettl wrote:
> So it may be that the Python interpreter isn't working correctly onlyon 
> Ubuntu 10.10

   Than you should report the problem to the Ubuntu developers, right?
And it would be nice if you investigate deeper and send a proper mail -
with a subject, with a properly formatted text, not html.

http://www.catb.org/~esr/faqs/smart-questions.html

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Oleg Broytman]

On Mon, Jan 24, 2011 at 04:39:39PM +0100, Victor Stinner wrote:
> I missed something: VFAT stores filenames as unicode (whereas FAT only 
> supports byte filenames). Well, VFAT stores filenames twice: as a 8+3 byte 
> strings and as a 255 unicode (UTF-16-LE) string (UTF-16-LE).
> 
> On which OS do you access this VFAT file system? On Windows, you have two 
> APIs: bytes (*A) and wide character (*W). If you use the wide character, 
> there 
> is explicit encoding at all. Linux has two mount options to control unicode 
> on 
> a VFAT filesystem: "codepage" for the byte filenames (use Shift JIS here) and 
> "iocharset" for the unicode filenames (I don't understand this option). 

   AFAIU, `codepage` is "remote charset" while `iocharset` is "local
charset". I.e., to mount windows-1251 filesystem to my linux with koi8-r
locale I use codepage=cp866,iocharset=koi8-r (cp866 is OEM encoding for
cp1251 ANSI).

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] tahoe-lafs

[Earney, Billy C.]

I want to make it clear that I am in no way associated with the tahoe-lafs 
project.  I do not want my email to make that project look bad.  That was not 
my intention.

Billy Earney
ear...@umsystem.edu
Programmer/Analyst-Expert
[cid:image001.gif@01C0.03DD8B00]  MySQL Certified DBA

Office of Social and Economic Data Analysis (OSEDA)
University of Missouri
Phone: 573-882-7396
Fax: 573-884-4635

From: Nick Coghlan [mailto:ncogh...@gmail.com]
Sent: Monday, January 24, 2011 9:35 AM
To: Earney, Billy C.
Cc: Python-Dev@python.org
Subject: Re: [Python-Dev] tahoe-lafs

On Mon, Jan 24, 2011 at 11:56 PM, Earney, Billy C. 
mailto:ear...@umsystem.edu>> wrote:
Greetings!

I know that this list is for python development questions/comments,

People that post questions innocently unaware of the nature of this list have 
an excuse.

You don't.

This is not a good way to encourage people to think well of you or your project.

Regards,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   
Brisbane, Australia
<>___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Victor Stinner]

Le lundi 24 janvier 2011 16:39:39, Victor Stinner a écrit :
> Le lundi 24 janvier 2011 11:35:22, Stephen J. Turnbull a écrit :
> > ... VFAT-formatted file systems and Shift JIS file names ...
> 
> I missed something: VFAT stores filenames as unicode (whereas FAT only
> supports byte filenames). Well, VFAT stores filenames twice: as a 8+3 byte
> strings and as a 255 unicode (UTF-16-LE) string (UTF-16-LE).
> 
> On which OS do you access this VFAT file system? On Windows, you have two
> APIs: bytes (*A) and wide character (*W). If you use the wide character,
> there is explicit encoding at all.

Oops, there is *not* explicit encoding a all.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] (no subject)

[Stefan Spoettl]

Using:Python 2.7.0+ (r27:82500, Sep 15 2010, 18:14:55) [GCC 4.4.5] on 
linux2(Ubuntu 10.10)
Method to reproduce error:
1. Defining a module which is later imported by another:
-
class SomeThing:
def __init__(self):self.variable = 'Where is my bytecode?'
def deliver(self):return self.variable

if __name__ == '__main__':obj = SomeThing()print obj.deliver()
-
2. Run this module:Output of the Python Shell: Where is my bytecode?
   >>>
3. Defining the importing module:
-
class UseSomeThing:
def __init__(self, something):self.anything = something
def giveanything(self):return self.anything

if __name__ == '__main__':anything = 
UseSomeThing(SomeThing.SomeThing().deliver()).giveanything()print anything
-
4. Run this module:Output of the Python Shell: Where is my bytecode 
   >>>(One can find SomeThing.pyc on the 
disc.)
5. Changing the imported module:
-class 
SomeThing:def __init__(self):self.variable = 'What the hell is 
this? It could not be Python!'def deliver(self):return 
self.variableif __name__ == '__main__':obj = SomeThing()print 
obj.deliver()-
6. Run the changed module:Output of the Python Shell: What the hell is this? It 
could not be Python!   >>>
7. Run the importing module again:Output of the Python Shell: Where is my 
bytecode?   >>>8. Deleting the 
bytecode of the imported module makes no effect!
Remark: I think that I have observed yesterday late night a similar effect on 
Windows XPwith Python 2.7.1 and Python 3.1.3. But when I have tried it out 
today in the morning theerror hasn't appeared. So it may be that the Python 
interpreter isn't working correctly onlyon Ubuntu 10.10.
 ___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Victor Stinner]

Le lundi 24 janvier 2011 11:35:22, Stephen J. Turnbull a écrit :
> ... VFAT-formatted file systems and Shift JIS file names ...

I missed something: VFAT stores filenames as unicode (whereas FAT only 
supports byte filenames). Well, VFAT stores filenames twice: as a 8+3 byte 
strings and as a 255 unicode (UTF-16-LE) string (UTF-16-LE).

On which OS do you access this VFAT file system? On Windows, you have two 
APIs: bytes (*A) and wide character (*W). If you use the wide character, there 
is explicit encoding at all. Linux has two mount options to control unicode on 
a VFAT filesystem: "codepage" for the byte filenames (use Shift JIS here) and 
"iocharset" for the unicode filenames (I don't understand this option). 
Anyway, both systems support unicode filenames.

I suppose that Shift JIS is used to encode the filename in the 8+3 byte string 
form.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Beta version of the new devguide

[Antoine Pitrou]

On Sat, 22 Jan 2011 17:08:00 -0800
Brett Cannon  wrote:
> 
> Two, what should the final URL be? Georg picked the current one and I
> am happy with it.

Ditto for me.

> Three, where should it be linked from? docs.python.org homepage?
> Four, what to do with www.python.org/dev/? Redirect for all the pages?

Right, this whole area (wpo/dev) looks obsolete to me. The devguide
allows us to easily edit and improve development-related docs, which is
great! It should be accessible easily from the main site. Perhaps
"core development" should be renamed "contributing" and redirect to the
devguide. Also, the submenu displayed below "core development" can be
trimmed dramatically.

(then there's the question of whether the devguide should be
exhaustive; should it contain reference-like material about
all aspects of core development?)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] tahoe-lafs

[Nick Coghlan]

On Mon, Jan 24, 2011 at 11:56 PM, Earney, Billy C. wrote:

> Greetings!
>
>
>
> I know that this list is for python development questions/comments,
>

People that post questions innocently unaware of the nature of this list
have an excuse.

You don't.

This is not a good way to encourage people to think well of you or your
project.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Beta version of the new devguide

[Nick Coghlan]

On Mon, Jan 24, 2011 at 9:29 PM, Antoine Pitrou  wrote:
> On Mon, 24 Jan 2011 20:33:07 +1000
> Nick Coghlan  wrote:
>> On Mon, Jan 24, 2011 at 6:22 AM, Brett Cannon  wrote:
>> >> In "Getting Set Up" it describes how to build a pydebug build. Is that
>> >> really necessary for those who plan only to contribute by working on
>> >> pure Python code?
>> >>
>> >
>> > Yes, there is actually a laundry list of reasons even people only
>> > working on the stdlib should use a pydebug build.
>>
>> And one big reason why I don't unless I have a specific need to check
>> something with it - it makes the already quite long running time for
>> the full test suite take even longer :)
>
> Please try the -j option to regrtest.

While I must admit I'm still not in the habit of running tests in
parallel, that's a substantial speed improvement regardless of build
type, so a non-debug build is still noticeably faster.

release (with -j4): 2 min 25 sec (3 min wall clock time)
pydebug (with -j4): 4 min 43 sec (10 min wall clock time)

Given that I typically *don't* need the extra info from a debug build
to analyse problems and a full configure and rebuild cycle takes less
time than a single pydebug test run, I'll happily stick with the much
faster test execution that comes from using a release build.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] tahoe-lafs

[Antoine Pitrou]

On Mon, 24 Jan 2011 07:56:56 -0600
"Earney, Billy C."  wrote:
> Greetings!
> 
> I know that this list is for python development questions/comments, but I 
> wanted to bring up the tahoe-lafs project [...]

You should really post such messages to comp.lang.python.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] tahoe-lafs

[Earney, Billy C.]

Greetings!

I know that this list is for python development questions/comments, but I 
wanted to bring up the tahoe-lafs project if people are interested in a project 
developed in python that allows for secure distributed storage.  For more 
information see http://tahoe-lafs.org

For those of you interested in joining a tahoe-lafs storage grid, I'm a member 
of a newly created storage grid called volunteer-grid2, and we are currently 
looking for new members.  The requirements to be a member can be viewed at 
http://bigpig.org/twiki/bin/view/Main/AboutVolunteerGrid2

Billy Earney
ear...@umsystem.edu
Programmer/Analyst-Expert
[cid:image001.gif@01CBBB9B.FB70CDB0]  MySQL Certified DBA

Office of Social and Economic Data Analysis (OSEDA)
University of Missouri
Phone: 573-882-7396
Fax: 573-884-4635

<>___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Beta version of the new devguide

[Antoine Pitrou]

On Mon, 24 Jan 2011 20:33:07 +1000
Nick Coghlan  wrote:
> On Mon, Jan 24, 2011 at 6:22 AM, Brett Cannon  wrote:
> >> In "Getting Set Up" it describes how to build a pydebug build. Is that
> >> really necessary for those who plan only to contribute by working on
> >> pure Python code?
> >>
> >
> > Yes, there is actually a laundry list of reasons even people only
> > working on the stdlib should use a pydebug build.
> 
> And one big reason why I don't unless I have a specific need to check
> something with it - it makes the already quite long running time for
> the full test suite take even longer :)

Please try the -j option to regrtest.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Beta version of the new devguide

[Mark Summerfield]

Hi Brett,

On Sat, 22 Jan 2011 17:08:00 -0800
Brett Cannon  wrote:
> http://docs.python.org/devguide/

Personally, I found the first paragraph of "Contributing" a bit
off-putting.

How about replacing:

People who wish to contribute to Python must read the following
documents in the order provided. You can stop where you feel
comfortable and begin contributing immediately without reading and
understanding these documents all at once, but please do not skip
around within the documentation as everything is written assuming
preceding documentation has been read.

With something like:

The Python core development team always welcomes new contributors,
so we are very glad of your interest! Please read the following
documents---in the order shown---to ensure that you understand how
Python's development process works. This will ensure that your
contributions are considered purely on their merit and don't get
rejected due to missing or incorrectly performing a step in the
process.

In "Getting Set Up" it describes how to build a pydebug build. Is that
really necessary for those who plan only to contribute by working on
pure Python code?

I had a quick skim over the rest and got the feeling that no clear
distinction is made between C and Python work. Personally, I feel that
more of a distinction should be made since not everyone will be
confident or interested in C. (And maybe more distinction should be made
between working on CPython and the standard library?)

Overall I think this document is *extremely welcome* and I am very glad
you have done it. I'm sure that once it starts to get known it will help
add to the pool of people contributing to Python as well as helping to
keep the processes clear:-)

-- 
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
C++, Python, Qt, PyQt - training and consultancy
"Advanced Qt Programming" - ISBN 0321635906
http://www.qtrac.eu/aqpbook.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Nick Coghlan]

On Mon, Jan 24, 2011 at 8:35 PM, Stephen J. Turnbull  wrote:
> First of all, these aren't just phones; these are all kinds of gadgets
> (the example I gave was a camera).  They're not as smart as an Android
> or iPhone-like device, and I don't know what OS they use.

We're getting a little far afield from the original question though -
once it was pointed out that non-ASCII module names already work on
some systems but not others, it became fairly clear that Victor's
patch is about fixing an existing feature to be more robust rather
than adding something new.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Stephen J. Turnbull]

"Martin v. Löwis" writes:

 > It's one thing how the file systems are formatted, but another thing
 > how they are presented to APIs. For example, the phones using Windows CE
 > would have to convert the file names to Unicode in the OS kernel.
 > 
 > So: for these phones - do you know how they present file names to the
 > application?

First of all, these aren't just phones; these are all kinds of gadgets
(the example I gave was a camera).  They're not as smart as an Android
or iPhone-like device, and I don't know what OS they use.

As for "presentation to the application", as I said, my older phones
presented themselves as "removable memory devices" (specifically on
the USB port), with VFAT-formatted file systems and Shift JIS file
names.  In that case you can surely have the kinds of problems
described, even if the app is not running on the device itself.  I
don't know if this is still true of more modern devices, but I was a
little shocked that is was true at all, even 5 or 6 years ago.

That may be one reason why the phone I have now doesn't provide a USB
interface at all.  That kind of interface is not only unnecessary with
Bluetooth, but Bluetooth uses more robust protocols.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Beta version of the new devguide

[Nick Coghlan]

On Mon, Jan 24, 2011 at 6:22 AM, Brett Cannon  wrote:
>> In "Getting Set Up" it describes how to build a pydebug build. Is that
>> really necessary for those who plan only to contribute by working on
>> pure Python code?
>>
>
> Yes, there is actually a laundry list of reasons even people only
> working on the stdlib should use a pydebug build.

And one big reason why I don't unless I have a specific need to check
something with it - it makes the already quite long running time for
the full test suite take even longer :)

I figure it's beneficial to have people running a mixture of debug and
release builds anyway - it helps catch things that work in one mode
and not the other.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Martin v. Löwis]

>  > Really? I would have thought that cell phones have long been the
>  > platforms most supportive of Unicode.
> 
> I would think so too, except in Japan.
> 
> However, my previous phones exposed file systems with names encoded in
> Shift JIS to USB and IR browsers, though.  (My current one uses
> Bluetooth, and I don't know how to "get at" the filesystem itself.)  A
> lot of these devices also tend to present themselves as VFAT-formatted
> drives (a la a USB memory stick), and Shift JIS is very commonly used
> on those for reasons I don't really understand.

It's one thing how the file systems are formatted, but another thing
how they are presented to APIs. For example, the phones using Windows CE
would have to convert the file names to Unicode in the OS kernel.

So: for these phones - do you know how they present file names to the
application?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Import and unicode: part two

[Stephen J. Turnbull]

Guido van Rossum writes:

 > Really? I would have thought that cell phones have long been the
 > platforms most supportive of Unicode.

I would think so too, except in Japan.

However, my previous phones exposed file systems with names encoded in
Shift JIS to USB and IR browsers, though.  (My current one uses
Bluetooth, and I don't know how to "get at" the filesystem itself.)  A
lot of these devices also tend to present themselves as VFAT-formatted
drives (a la a USB memory stick), and Shift JIS is very commonly used
on those for reasons I don't really understand.

In any case, AIUI here the problem is like the problem of refactoring
a "make"-based system.  There are identifiers which are "spelled" one
way inside of files which need to match the "spelling" of names of
external filesystem objects.  If you transport such a set of files to
a POSIX system (which AFAIK most servers still are), then it's quite
possible that the file names will get translated to the locale's
encoding while the identifiers will not.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com