Re: [Python-Dev] PEP-8 wart... it recommends short names because of DOS

[eryksun]

On 10/21/15, Serhiy Storchaka  wrote:
> On 21.10.15 04:25, Gregory P. Smith wrote:
>> https://www.python.org/dev/peps/pep-0008/#names-to-avoid
>>
>> /"Since module names are mapped to file names, and some file systems are
>> case insensitive and truncate long names, it is important that module
>> names be chosen to be fairly short -- this won't be a problem on Unix,
>> but it may be a problem when the code is transported to older Mac or
>> Windows versions, or DOS."/
>>
>> There haven't been computers with less than 80 character file or path
>> name element length limits in wide use in decades... ;)
>
> We should also avoid special file names like con.py or lpt1.py.

Other file names to avoid on Windows are conin$.py, conout$.py,
aux.py, prn.py, nul.py, lpt[1-9].py, and com[1-9].py.

Using these device names in a file name requires the fully qualified
wide-character path, prefixed by \\?\. Incidentally this prefix also
allows paths that have up to 32768 characters, if there's concern that
long module names in packages might exceed the Windows 260-character
limit.

Here's an example of what would actually be opened for con.py, etc, at
least on my current Windows 10 machine:

devs = ('aux prn com1 com9 lpt1 lpt9 '
'nul con conin$ conout$'.split())

for dev in devs:
ntpath = to_nt(r'C:\%s.py' % dev)
print(ntpath.ljust(11), '=>' ,query_link(ntpath))

output:

\??\aux => \DosDevices\COM1
\??\prn => \DosDevices\LPT1
\??\com1=> object name not found
\??\com9=> object name not found
\??\lpt1=> \Device\Parallel0
\??\lpt9=> object name not found
\??\nul => \Device\Null
\??\con => \Device\ConDrv\Console
\??\conin$  => \Device\ConDrv\CurrentIn
\??\conout$ => \Device\ConDrv\CurrentOut

The \\?\ prefix avoids DOS name translation. The only change made by
the system is to replace \\?\ with \?? in the path:

for dev in devs:
print(to_nt(r'\\?\C:\%s.py' % dev))

output:

\??\C:\aux.py
\??\C:\prn.py
\??\C:\com1.py
\??\C:\com9.py
\??\C:\lpt1.py
\??\C:\lpt9.py
\??\C:\nul.py
\??\C:\con.py
\??\C:\conin$.py
\??\C:\conout$.py

On this machine, \??\C: is a link to \Device\HarddiskVolume2.

(to_nt and query_link call the native API functions
RtlDosPathNameToNtPathName_U, NtOpenSymbolicLinkObject, and
NtQuerySymbolicLinkObject. Note that Microsoft doesn't support calling
the native NT API from applications in user mode.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Joe Jevnik]

Sorry about introducing this. Where can I subscribe to these automated
emails. Also, how do I go about running this locally? On default I tried
running
`./python -m test -l test_capi` did not print anything about leaks. I think
that
using `object.__new__` as a decorator here is the same as subclassing
object,
overriding __new__ and then making a call to `super().__new__` so I would
imagine this bug could appear in less "clever" situations. I would love to
help
fix this issue; Benjamin, you mentioned that you think that maybe all
heaptypes
should have gc, do you have a suggestion on where I can look in the code to
try
to make this change?

On Wed, Oct 21, 2015 at 11:53 AM, Random832  wrote:

> Raymond Hettinger  writes:
> > Thanks for hunting this down.  I had seen the automated reference leak
> > posts but didn't suspect that a pure python class would have caused
> > the leak.
> >
> > I'm re-opening
> > https://mail.python.org/pipermail/python-dev/2015-October/141993.html
> > and will take a look at it this weekend.  If I don't see an obvious
> > fix, I'll revert Joe's patch until a correct patch is supplied and
> > reviewed.
>
> If a pure python class can cause a reference leak, doesn't that mean it
> is only a symptom rather than the real cause? Or is it that the use of
> @object.__new__ is considered "too clever" to be worth fixing?
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/joe%40quantopian.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Random832]

Raymond Hettinger  writes:
> Thanks for hunting this down.  I had seen the automated reference leak
> posts but didn't suspect that a pure python class would have caused
> the leak.
>
> I'm re-opening
> https://mail.python.org/pipermail/python-dev/2015-October/141993.html
> and will take a look at it this weekend.  If I don't see an obvious
> fix, I'll revert Joe's patch until a correct patch is supplied and
> reviewed.

If a pure python class can cause a reference leak, doesn't that mean it
is only a symptom rather than the real cause? Or is it that the use of
@object.__new__ is considered "too clever" to be worth fixing?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Eric Snow]

On Wed, Oct 21, 2015 at 9:32 AM, Raymond Hettinger
 wrote:
> I'm re-opening 
> https://mail.python.org/pipermail/python-dev/2015-October/141993.html

Presumably you meant http://bugs.python.org/issue24379. :)

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Antoine Pitrou]

Le 21/10/2015 17:32, Raymond Hettinger a écrit :
> 
> Thanks for hunting this down.  I had seen the automated reference leak posts
> but didn't suspect that a pure python class would have caused the leak.  

Yes, it's a bit baffling at first.

> I'm re-opening 
> https://mail.python.org/pipermail/python-dev/2015-October/141993.html 
> and will take a look at it this weekend.  If I don't see an obvious fix, I'll 
> revert Joe's patch
> until a correct patch is supplied and reviewed.

Well, the simple workaround is to remove the __slots__ field.  I don't
think it matters here, especially for a singleton.

Also, Benjamin's suggestion to make all heap type instances GC-enabled
sounds reasonable to me.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Raymond Hettinger]

> On Oct 20, 2015, at 3:57 PM, Antoine Pitrou  wrote:
> 
> 
>> These leaks have been here a while. Anyone know the cause?
>> 
>> On Tue, 20 Oct 2015 at 01:47  wrote:
>> 
>>> results for d7e490db8d54 on branch "default"
>>> 
>>> 
>>> test_capi leaked [5411, 5411, 5411] references, sum=16233
>>> test_capi leaked [1421, 1423, 1423] memory blocks, sum=4267
>>> test_functools leaked [0, 2, 2] memory blocks, sum=4
>>> test_threading leaked [10820, 10820, 10820] references, sum=32460
>>> test_threading leaked [2842, 2844, 2844] memory blocks, sum=8530
> 
> Bisection shows they were probably introduced by:
> 
> changeset:   97413:dccc4e63aef5
> user:Raymond Hettinger 
> date:Sun Aug 16 19:43:34 2015 -0700
> files:   Doc/library/operator.rst Doc/whatsnew/3.6.rst
> Lib/operator.py Lib/test/test_operator.py
> description:
> Issue #24379: Add operator.subscript() as a convenience for building slices.
> 
> 
> If you comment out `@object.__new__` on line 411 in operator.py, or if
> you remove the __slots__ assignment (which is a bit worrying), the leak
> seems suppressed.
> 

Thanks for hunting this down.  I had seen the automated reference leak posts
but didn't suspect that a pure python class would have caused the leak.  

I'm re-opening 
https://mail.python.org/pipermail/python-dev/2015-October/141993.html 
and will take a look at it this weekend.  If I don't see an obvious fix, I'll 
revert Joe's patch
until a correct patch is supplied and reviewed.


Raymond

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Brett Cannon]

On Wed, 21 Oct 2015 at 09:33 Joe Jevnik  wrote:

> Sorry about introducing this. Where can I subscribe to these automated
> emails.
>

The emails are sent to the python-checkins mailing list.


> Also, how do I go about running this locally?
>

If you look at the bottom of the email that reports the leaks you will
notice it tells you how the tests were run, e.g.:

 ['./python', '-m', 'test.regrtest', '-uall', '-R',
'3:3:/home/psf-users/antoine/refleaks/reflogyrNnBL', '--timeout', '7200']

-Brett




> On default I tried running
> `./python -m test -l test_capi` did not print anything about leaks. I
> think that
> using `object.__new__` as a decorator here is the same as subclassing
> object,
> overriding __new__ and then making a call to `super().__new__` so I would
> imagine this bug could appear in less "clever" situations. I would love to
> help
> fix this issue; Benjamin, you mentioned that you think that maybe all
> heaptypes
> should have gc, do you have a suggestion on where I can look in the code
> to try
> to make this change?
>
> On Wed, Oct 21, 2015 at 11:53 AM, Random832 
> wrote:
>
>> Raymond Hettinger  writes:
>> > Thanks for hunting this down.  I had seen the automated reference leak
>> > posts but didn't suspect that a pure python class would have caused
>> > the leak.
>> >
>> > I'm re-opening
>> > https://mail.python.org/pipermail/python-dev/2015-October/141993.html
>> > and will take a look at it this weekend.  If I don't see an obvious
>> > fix, I'll revert Joe's patch until a correct patch is supplied and
>> > reviewed.
>>
>> If a pure python class can cause a reference leak, doesn't that mean it
>> is only a symptom rather than the real cause? Or is it that the use of
>> @object.__new__ is considered "too clever" to be worth fixing?
>>
>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/joe%40quantopian.com
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Ethan Furman]

On 10/21/2015 08:53 AM, Random832 wrote:


If a pure python class can cause a reference leak, doesn't that mean it
is only a symptom rather than the real cause? Or is it that the use of
@object.__new__ is considered "too clever" to be worth fixing?


Where can I find out more about using `object.__new__` as a decorator?

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (d7e490db8d54): sum=61494

[Steven D'Aprano]

On Wed, Oct 21, 2015 at 10:10:56AM -0700, Ethan Furman wrote:
> On 10/21/2015 08:53 AM, Random832 wrote:
> 
> >If a pure python class can cause a reference leak, doesn't that mean it
> >is only a symptom rather than the real cause? Or is it that the use of
> >@object.__new__ is considered "too clever" to be worth fixing?
> 
> Where can I find out more about using `object.__new__` as a decorator?

How about the interactive interpreter?

py> @object.__new__
... class X:
... pass
...
py> X
<__main__.X object at 0xb7b4dacc>


Consider the pre-decorator-syntax way of writing that:

class X:
pass

X = object.__new__(X)

That's a way of setting X = X(), except that it only works for X a class 
(can't decorate a function this way), and it avoids calling the __init__ 
method.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 484 -- proposal to allow @overload in non-stub files

[Guido van Rossum]

PEP 484 (Type Hinting) currently disallows @overload outside stub files.
The hope was that a PEP for multi-dispatch would emerge, but that's been
slow coming.

Meanwhile, over in https://github.com/ambv/typehinting/issues/72 a proposal
has emerged to allow @overload in regular modules, as long as it is
followed by a non-overloaded definition that serves as the
default/fallback. A motivating example is __getitem__, which is often
overloaded for item access and slicing.

In stubs, you can use:

class Foo(Generic[T]):
@overload
def __getitem__(self, i: int) -> T: ...
@overload
def __getitem__(self, s: slice) -> Foo[T]: ...

(Note that the '...' are part of the actual code -- an ellipsis is how you
represent the body of all functions in stub files.)

However, in source files the best you can do is:

class Foo(Generic[T]):
def __getitem__(self, i: Union[int, slice]) -> Union[T, List[T]]: ...

which will require unacceptable casts at every call site. You can work
around it by having a stub file but that's cumbersome if this is the only
reason to have one.

The proposal is to allow this to be written as follows in implementation
(non-stub) modules:

class Foo(Generic[T]):
@overload
def __getitem__(self, i: int) -> T: ...
@overload
def __getitem__(self, s: slice) -> Foo[T]: ...
def __getitem__(self, x):


The actual implementation must be last, so at run time it will override the
definition. It has to use isinstance() to distinguish the cases. A type
checker would have to recognize this as a special case (so as not to
complain about the non-overloaded version). Jukka thinks it would be about
a day's work to implement in mypy; the work in typing.py would be a few
minutes.

Thoughts?

--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 484 -- proposal to allow @overload in non-stub files

[Random832]

Guido van Rossum  writes:
> The proposal is to allow this to be written as follows in
> implementation (non-stub) modules:
>
> class Foo(Generic[T]):
> @overload
> def __getitem__(self, i: int) -> T: ...
> @overload
> def __getitem__(self, s: slice) -> Foo[T]: ...
> def __getitem__(self, x):
> 
>
> The actual implementation must be last, so at run time it will
> override the definition.

How about this to allow overloads to have actual implementations?

@overloaded
def __getitem__(self, x):


@overloaded returns a function which will check the types against the
overloads (or anyway any overloads that have actual implementations),
call them returning the result if applicable, otherwise call the
original function.

Some magic with help() would improve usability, too - it could print all
the overloads and their docstrings.  Maybe even @overload('__getitem__')
def __get_int(self, i: int), to make it so order doesn't matter.

That just leaves the question of how's this all gonna work with
subclasses.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 484 -- proposal to allow @overload in non-stub files

[Guido van Rossum]

Well the whole point is not to have to figure out how to implement that
right now.

On Wed, Oct 21, 2015 at 6:45 PM, Random832  wrote:

> Guido van Rossum  writes:
> > The proposal is to allow this to be written as follows in
> > implementation (non-stub) modules:
> >
> > class Foo(Generic[T]):
> > @overload
> > def __getitem__(self, i: int) -> T: ...
> > @overload
> > def __getitem__(self, s: slice) -> Foo[T]: ...
> > def __getitem__(self, x):
> > 
> >
> > The actual implementation must be last, so at run time it will
> > override the definition.
>
> How about this to allow overloads to have actual implementations?
>
> @overloaded
> def __getitem__(self, x):
> 
>
> @overloaded returns a function which will check the types against the
> overloads (or anyway any overloads that have actual implementations),
> call them returning the result if applicable, otherwise call the
> original function.
>
> Some magic with help() would improve usability, too - it could print all
> the overloads and their docstrings.  Maybe even @overload('__getitem__')
> def __get_int(self, i: int), to make it so order doesn't matter.
>
> That just leaves the question of how's this all gonna work with
> subclasses.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should PEP 498 specify if rf'...' is valid?

[Ryan Gonzalez]

It mentions fr'...' as a formatted raw string but doesn't say anything about 
rf'...'. Right now, in implementing PEP 498 support in Howl 
(https://github.com/howl-editor/howl/pull/118 and 
https://github.com/howl-editor/howl/commit/1e577da89efc1c1de780634b531f64346cf586d6#diff-851d9b84896270cc7e3bbea3014007a5R86),
 I assumed both were valid. Should the PEP be more specific?

BTW, at the rate language-python is going, GitHub will get syntax highlighting 
for f-strings in 2050. :D
-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP-8 wart... it recommends short names because of DOS

[Serhiy Storchaka]

On 21.10.15 04:25, Gregory P. Smith wrote:

https://www.python.org/dev/peps/pep-0008/#names-to-avoid

/"Since module names are mapped to file names, and some file systems are
case insensitive and truncate long names, it is important that module
names be chosen to be fairly short -- this won't be a problem on Unix,
but it may be a problem when the code is transported to older Mac or
Windows versions, or DOS."/

There haven't been computers with less than 80 character file or path
name element length limits in wide use in decades... ;)


We should also avoid special file names like con.py or lpt1.py.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com