Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Ethan Furman]

On 07/09/2014 09:02 PM, Nick Coghlan wrote:

On 9 Jul 2014 17:14, "Ethan Furman" wrote:


I like the 'onerror' API better primarily because it gives a single
point to deal with the errors. [...]


The "onerror" approach can also deal with readdir failing, which the
 PEP currently glosses over.


Do we want this, though?  I can see an error handler for individual entries, but if one of the *dir commands fails that 
would seem to be fairly catastrophic.



I'm somewhat inclined towards the current approach in the PEP, but I'd like to 
see an explanation of two aspects:

1. How a scandir variant with an 'onerror' option could be implemented given 
the version in the PEP


Here's a stab at it:

def scandir_error(path, info=None, onerror=None):
for entry in scandir(path):
if info == 'type':
try:
entry.is_dir()
except OSError as exc:
if onerror is None:
raise
if not onerror(exc, entry):
continue
elif info == 'lstat':
try:
entry.lstat()
except OSError as exc:
if onerror is None:
raise
if not onerror(exc, entry):
continue
yield entry

Here it is again with an attempt to deal with opendir/readdir/closedir 
exceptions:

def scandir_error(path, info=None, onerror=None):
entries = scandir(path)
try:
entry = next(entries)
except StopIteration:
# pass it through
raise
except Exception as exc:
if onerror is None:
raise
if not onerror(exc, 'what else here?'):
# what do we do on False?
# what do we do on True?
else:
for entry in scandir(path):
if info == 'type':
try:
entry.is_dir()
except OSError as exc:
if onerror is None:
raise
if not onerror(exc, entry):
continue
elif info == 'lstat':
try:
entry.lstat()
except OSError as exc:
if onerror is None:
raise
if not onerror(exc, entry):
continue
yield entry



2. How the existing scandir module handles the 'onerror' parameter to its 
directory walking function


Here's the first third of it from the repo:

def walk(top, topdown=True, onerror=None, followlinks=False):
"""Like os.walk(), but faster, as it uses scandir() internally."""
# Determine which are files and which are directories
dirs = []
nondirs = []
try:
for entry in scandir(top):
if entry.is_dir():
dirs.append(entry)
else:
nondirs.append(entry)
except OSError as error:
if onerror is not None:
onerror(error)
return
...

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3121, 384 Refactoring Issues

[Ethan Furman]

On 07/10/2014 04:57 PM, Alexander Belopolsky wrote:

On Thu, Jul 10, 2014 at 2:59 PM, Mark Lawrence wrote:


I'm just curious as to why there are 54 open issues after both of
these PEPs have been accepted and 384 is listed as finished.  Did
 we hit some unforeseen technical problem which stalled development?


I tried to bring some sanity to that effort by opening a "meta issue":

http://bugs.python.org/issue15787

My enthusiasm, however, vanished after I reviewed the refactoring for the 
datetime module:

http://bugs.python.org/issue15390

My main objections are to following PEP 384 
 (Stable ABI) within stdlib
modules.  I see little benefit for the stdlib (which is shipped fresh with 
every new version of Python) from following
those guidelines.


If we aren't going to implement the changes (and I agree there's little value for the stdlib to do so), let's mark the 
issues as "won't fix" and close them.


And thanks, Mark, for bringing it up.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3121, 384 Refactoring Issues

[Alexander Belopolsky]

On Thu, Jul 10, 2014 at 2:59 PM, Mark Lawrence 
wrote:

> I'm just curious as to why there are 54 open issues after both of these
> PEPs have been accepted and 384 is listed as finished.  Did we hit some
> unforeseen technical problem which stalled development?


I tried to bring some sanity to that effort by opening a "meta issue":

http://bugs.python.org/issue15787

My enthusiasm, however, vanished after I reviewed the refactoring for the
datetime module:

http://bugs.python.org/issue15390

My main objections are to following PEP 384
 (Stable ABI) within stdlib
modules.  I see little benefit for the stdlib (which is shipped fresh with
every new version of Python) from following those guidelines.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3121, 384 Refactoring Issues

[Guido van Rossum]

I don't know the details, but I suspect that was the result of my general
guideline "don't start projects cleaning up lots of stdlib code just to
satisfy some new style rule or just to use a new API" -- which came from
hard-won experience where such a cleanup project introduced some new bugs
that weren't found by review nor by tests. Though that was admittedly a
long time. Still, such a project can really sap reviewer resources for
relatively little benefit.


On Thu, Jul 10, 2014 at 12:59 PM, Brett Cannon  wrote:

> [for those that don't know, 3121 is extension module inti/finalization and
> 384 is the stable ABI]
>
>
> On Thu Jul 10 2014 at 3:47:03 PM, Mark Lawrence 
> wrote:
>
>> I'm just curious as to why there are 54 open issues after both of these
>> PEPs have been accepted and 384 is listed as finished.  Did we hit some
>> unforeseen technical problem which stalled development?
>>
>
> No, the PEPs were fine and were accepted properly. A huge portion of the
> open issues are from Robin Schreiber who as part of GSoC 2012 --
> https://www.google-melange.com/gsoc/project/details/google/gsoc2012/robin_hood/5668600916475904
> -- went through and updated the stdlib to follow the new practices
> introduced in the two PEPs. Not sure if there was some policy decision made
> that updating the code wasn't worth it or people simply didn't get around
> to applying the patches.
>
> -Brett
>
>
>>
>> For these and any other open issues if you need some Windows testing
>> doing please feel free to put me on the nosy list and ask for a test run.
>>
>> --
>> My fellow Pythonistas, ask not what our language can do for you, ask
>> what you can do for our language.
>>
>> Mark Lawrence
>>
>> ---
>> This email is free from viruses and malware because avast! Antivirus
>> protection is active.
>> http://www.avast.com
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> brett%40python.org
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3121, 384 Refactoring Issues

[Brett Cannon]

[for those that don't know, 3121 is extension module inti/finalization and
384 is the stable ABI]

On Thu Jul 10 2014 at 3:47:03 PM, Mark Lawrence 
wrote:

> I'm just curious as to why there are 54 open issues after both of these
> PEPs have been accepted and 384 is listed as finished.  Did we hit some
> unforeseen technical problem which stalled development?
>

No, the PEPs were fine and were accepted properly. A huge portion of the
open issues are from Robin Schreiber who as part of GSoC 2012 --
https://www.google-melange.com/gsoc/project/details/google/gsoc2012/robin_hood/5668600916475904
-- went through and updated the stdlib to follow the new practices
introduced in the two PEPs. Not sure if there was some policy decision made
that updating the code wasn't worth it or people simply didn't get around
to applying the patches.

-Brett


>
> For these and any other open issues if you need some Windows testing
> doing please feel free to put me on the nosy list and ask for a test run.
>
> --
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
>
> Mark Lawrence
>
> ---
> This email is free from viruses and malware because avast! Antivirus
> protection is active.
> http://www.avast.com
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 3121, 384 Refactoring Issues

[Mark Lawrence]

I'm just curious as to why there are 54 open issues after both of these 
PEPs have been accepted and 384 is listed as finished.  Did we hit some 
unforeseen technical problem which stalled development?


For these and any other open issues if you need some Windows testing 
doing please feel free to put me on the nosy list and ask for a test run.


--
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.


Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection 
is active.
http://www.avast.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Ethan Furman]

On 07/10/2014 06:58 AM, Nick Coghlan wrote:


The info we want for scandir is that of the *link itself*. That makes it
easy to implement things like the "followlinks" flag of os.walk. The
 *far end* of the link isn't relevant at this level.


This also mirrors listdir, correct?  scandir is simply* returning something 
smarter than a string.


The docs just need to be clear that DirEntry objects always match lstat(), 
never stat().


Agreed.

--
~Ethan~

* As well as being a less resource-intensive generator.  :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Ben Hoyt]

>> DirEntry methods will remain free (no syscall) for directories and
>> regular files. One extra syscall will be needed only for symlinks,
>> which are more rare than other file types (for example, you wrote "
>> Windows typically makes little use of symlinks").
>
> The info we want for scandir is that of the *link itself*. That makes it
> easy to implement things like the "followlinks" flag of os.walk. The *far
> end* of the link isn't relevant at this level.
>
> The docs just need to be clear that DirEntry objects always match lstat(),
> never stat().

Yeah, I agree with this. It makes the function (and documentation and
implementation) quite a lot simpler to understand.

scandir() is a lowish-level function which deals with the directory
entries themselves, and mirrors both Windows FindNextFile and POSIX
readdir() in that. If the user wants follow-links behaviour, they can
easily call os.stat() themselves. If this is clearly documented that
seems much simpler to me (and it also seems implicit to me in the fact
that you're calling is_dir() on the *entry*).

Otherwise we might as well go down the route of -- the objects
returned are just like pathlib.Path(), but with stat() and lstat()
cached on first use.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Nick Coghlan]

On 10 Jul 2014 03:39, "Victor Stinner"  wrote:
>
> 2014-07-10 9:04 GMT+02:00 Paul Moore :
> > As someone (Tim?) pointed out later in the thread,
> > FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor
> > do the dirent entries on Unix). So whether or not it's "natural", the
> > "free" functionality provided by the OS is that of lstat, not that of
> > stat. Presumably because it's possible to build symlink-following code
> > on top of non-following code, but not the other way around.
>
> DirEntry methods will remain free (no syscall) for directories and
> regular files. One extra syscall will be needed only for symlinks,
> which are more rare than other file types (for example, you wrote "
> Windows typically makes little use of symlinks").

The info we want for scandir is that of the *link itself*. That makes it
easy to implement things like the "followlinks" flag of os.walk. The *far
end* of the link isn't relevant at this level.

The docs just need to be clear that DirEntry objects always match lstat(),
never stat().

Cheers,
Nick.

>
> See my pseudo-code:
> https://mail.python.org/pipermail/python-dev/2014-July/135439.html
>
> On Windows, _lstat and _stat attributes will be filled directly in the
> constructor on Windows for regular files and directories.
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Victor Stinner]

2014-07-10 9:04 GMT+02:00 Paul Moore :
> As someone (Tim?) pointed out later in the thread,
> FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor
> do the dirent entries on Unix). So whether or not it's "natural", the
> "free" functionality provided by the OS is that of lstat, not that of
> stat. Presumably because it's possible to build symlink-following code
> on top of non-following code, but not the other way around.

DirEntry methods will remain free (no syscall) for directories and
regular files. One extra syscall will be needed only for symlinks,
which are more rare than other file types (for example, you wrote "
Windows typically makes little use of symlinks").

See my pseudo-code:
https://mail.python.org/pipermail/python-dev/2014-July/135439.html

On Windows, _lstat and _stat attributes will be filled directly in the
constructor on Windows for regular files and directories.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbot.python.org down again?

[Martin v. Löwis]

Am 08.07.14 16:48, schrieb Guido van Rossum:
> May the true owner of buildbot.python.org 
> stand up!

Well, I think that's me (atleast by my definition of "true owner").
I requested that the machine be set up, and I deployed the software
that is running the service (it was also me who originally introduced
buildbot to the Python project).

On the other hand, I'm not at all "in charge" of that infrastructure
piece. I haven't logged into the machine in many months, and it's
Antoine who currently maintains its configuration. So I don't want to
be pinged when the machine is down.

> (But I do think there may well not be anyone who feels they own it. And
> that's a problem for its long term viability.)

I don't think that's actually the case for "ownership". But then, I also
think that ownership is not a very important concept for pydotorg. Most
owners will likely agree that they lose their right to have a say in it
when they stop maintaining the piece that they own.

> Generally speaking, as an organization we should set up a process for
> managing ownership of *all* infrastructure in a uniform way. I don't
> mean to say that we need to manage all infrastructure uniformly, just
> that we need to have a process for identifying and contacting the
> owner(s) for each piece of infrastructure, as well as collecting other
> information that people besides the owners might need to know. You can
> use a wiki page for that list for all I care, but have a process for
> what belongs there, how/when to update it, and even an owner for the
> wiki page!

Unfortunately, that plan keeps failing. Everybody agrees that such a
list would be useful, so everybody makes their own list. I was
maintaining such a list in the Python wiki for some time, until a
board member decided that a publically-visible inventory is not
appropriate, and it must be a password-protected wiki - where I now
keep forgetting where the wiki is, in the first place, let alone
remembering how to log in.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Tim Delaney]

On 10 July 2014 17:04, Paul Moore  wrote:

> On 10 July 2014 01:23, Victor Stinner  wrote:
> >> As a Windows user with only a superficial understanding of how
> >> symlinks should behave, (...)
> >
> > FYI Windows also supports symbolic links since Windows Vista. The
> > feature is unknown because it is restricted to the administrator
> > account. Try the "mklink" command in a terminal (cmd.exe) ;-)
> > http://en.wikipedia.org/wiki/NTFS_symbolic_link
> >
> > ... To be honest, I never created a symlink on Windows. But since it
> > is supported, you need to know it to write correctly your Windows
> > code.
>
> I know how symlinks *do* behave, and I know how Windows supports them.
> What I meant was that, because Windows typically makes little use of
> symlinks, I have little or no intuition of what feels natural to
> people using an OS where symlinks are common.
>
> As someone (Tim?) pointed out later in the thread,
> FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor
> do the dirent entries on Unix).


It wasn't me (I didn't even see it - lost in the noise).


> So whether or not it's "natural", the
> "free" functionality provided by the OS is that of lstat, not that of
> stat. Presumably because it's possible to build symlink-following code
> on top of non-following code, but not the other way around.
>

For most uses the most natural thing is to follow symlinks (e.g. opening a
symlink in a text editor should open the target). However, I think not
following symlinks by default is better approach for exactly the reason
Paul has noted above.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Paul Moore]

On 10 July 2014 01:23, Victor Stinner  wrote:
>> As a Windows user with only a superficial understanding of how
>> symlinks should behave, (...)
>
> FYI Windows also supports symbolic links since Windows Vista. The
> feature is unknown because it is restricted to the administrator
> account. Try the "mklink" command in a terminal (cmd.exe) ;-)
> http://en.wikipedia.org/wiki/NTFS_symbolic_link
>
> ... To be honest, I never created a symlink on Windows. But since it
> is supported, you need to know it to write correctly your Windows
> code.

I know how symlinks *do* behave, and I know how Windows supports them.
What I meant was that, because Windows typically makes little use of
symlinks, I have little or no intuition of what feels natural to
people using an OS where symlinks are common.

As someone (Tim?) pointed out later in the thread,
FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor
do the dirent entries on Unix). So whether or not it's "natural", the
"free" functionality provided by the OS is that of lstat, not that of
stat. Presumably because it's possible to build symlink-following code
on top of non-following code, but not the other way around.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com