Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> Very much agreed that this isn't necessary for just readdir/FindNext >> errors. We've never had this level of detail before -- if listdir() >> fails half way through (very unlikely) it just bombs with OSError and >> you get no entries at all. >> >> If you really really want this (again very unli

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 11 Jul 2014 12:46, "Ben Hoyt" wrote: > > [replying to python-dev this time] > > >> The "onerror" approach can also deal with readdir failing, which the > >> PEP currently glosses over. > > > > > > Do we want this, though? I can see an error handler for individual entries, > > but if one of th

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[replying to python-dev this time] >> The "onerror" approach can also deal with readdir failing, which the >> PEP currently glosses over. > > > Do we want this, though? I can see an error handler for individual entries, > but if one of the *dir commands fails that would seem to be fairly > catas

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 09:02 PM, Nick Coghlan wrote: On 9 Jul 2014 17:14, "Ethan Furman" wrote: I like the 'onerror' API better primarily because it gives a single point to deal with the errors. [...] The "onerror" approach can also deal with readdir failing, which the PEP currently glosses over. D

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/10/2014 06:58 AM, Nick Coghlan wrote: The info we want for scandir is that of the *link itself*. That makes it easy to implement things like the "followlinks" flag of os.walk. The *far end* of the link isn't relevant at this level. This also mirrors listdir, correct? scandir is simply*

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> DirEntry methods will remain free (no syscall) for directories and >> regular files. One extra syscall will be needed only for symlinks, >> which are more rare than other file types (for example, you wrote " >> Windows typically makes little use of symlinks"). > > The info we want for scandir is

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 10 Jul 2014 03:39, "Victor Stinner" wrote: > > 2014-07-10 9:04 GMT+02:00 Paul Moore : > > As someone (Tim?) pointed out later in the thread, > > FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor > > do the dirent entries on Unix). So whether or not it's "natural", the > > "

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

2014-07-10 9:04 GMT+02:00 Paul Moore : > As someone (Tim?) pointed out later in the thread, > FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor > do the dirent entries on Unix). So whether or not it's "natural", the > "free" functionality provided by the OS is that of lstat, no

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 10 July 2014 17:04, Paul Moore wrote: > On 10 July 2014 01:23, Victor Stinner wrote: > >> As a Windows user with only a superficial understanding of how > >> symlinks should behave, (...) > > > > FYI Windows also supports symbolic links since Windows Vista. The > > feature is unknown because

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 10 July 2014 01:23, Victor Stinner wrote: >> As a Windows user with only a superficial understanding of how >> symlinks should behave, (...) > > FYI Windows also supports symbolic links since Windows Vista. The > feature is unknown because it is restricted to the administrator > account. Try th

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 Jul 2014 17:14, "Ethan Furman" wrote: > > On 07/09/2014 02:42 PM, Ben Hoyt wrote: >>> >>> >>> Okay, so using that [no platform specific] logic we should head over to the os module and remove: >>> >>> >>> ctermid, getenv, getegid... >>> >>> Okay, I'm tired of typing, but that list is not even

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

Ben Hoyt writes: ... > ``scandir()`` yields a ``DirEntry`` object for each file and directory > in ``path``. Just like ``listdir``, the ``'.'`` and ``'..'`` > pseudo-directories are skipped, and the entries are yielded in > system-dependent order. Each ``DirEntry`` object has the following > attri

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 10 July 2014 10:23, Victor Stinner wrote: > 2014-07-09 17:26 GMT+02:00 Paul Moore : > > On 9 July 2014 16:05, Victor Stinner wrote: > >> The PEP says that DirEntry should mimic pathlib.Path, so I think that > >> DirEntry.is_dir() should work as os.path.isir(): if the entry is a > >> symbolic

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

Oh, since I'm proposing to add a new stat() method to DirEntry, we can optimize it. stat() can reuse lstat() result if the file is not a symlink. It simplifies is_dir(). New pseudo-code: --- class DirEntry: def __init__(self, path, name, lstat=None, d_type=None): self.name = name

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 05:15 PM, Victor Stinner wrote: 2014-07-09 17:29 GMT+02:00 Ben Hoyt : Would this not "break" the tree size script being discussed in the other thread, as it would follow links and include linked directories in the "size" of the tree? The get_tree_size() function in the PEP would

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

Ben Hoyt writes: > So here's the ways in which option #2 is now more complicated than option #1: > > 1) it has an additional "info" argument, the values of which have to > be documented ('os', 'type', 'lstat', and what each one means) > 2) it has an additional "onerror" argument, the signature of

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

2014-07-09 17:26 GMT+02:00 Paul Moore : > On 9 July 2014 16:05, Victor Stinner wrote: >> The PEP says that DirEntry should mimic pathlib.Path, so I think that >> DirEntry.is_dir() should work as os.path.isir(): if the entry is a >> symbolic link, you should follow the symlink to get the status of

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

2014-07-09 17:29 GMT+02:00 Ben Hoyt : >> Would this not "break" the tree size script being discussed in the >> other thread, as it would follow links and include linked directories >> in the "size" of the tree? The get_tree_size() function in the PEP would use: "if not entry.is_symlink() and entry

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 04:22 PM, MRAB wrote: On 2014-07-09 23:50, Ethan Furman wrote: Okay, marry the two ideas together: scandir(path, info=None, onerror=None) """ Return a generator that returns one directory entry at a time in a DirEntry object Should that be "that yields one

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 2014-07-09 23:50, Ethan Furman wrote: On 07/09/2014 02:33 PM, Ben Hoyt wrote: On a system which did not supply is_dir automatically I would write that as: for entry in os.scandir(path): if ignore_entry(entry.name): continue if os.path.isdir(entry.full_name):

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 02:33 PM, Ben Hoyt wrote: On a system which did not supply is_dir automatically I would write that as: for entry in os.scandir(path): if ignore_entry(entry.name): continue if os.path.isdir(entry.full_name): # do something interesting Not har

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 02:38 PM, Victor Stinner wrote: 2014-07-09 22:44 GMT+02:00 Ethan Furman: On 07/09/2014 01:24 PM, Victor Stinner wrote: [...] Python must remain portable and you should not write code specific to one specific platform. Okay, so using that logic we should head over to the os mod

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 02:42 PM, Ben Hoyt wrote: Okay, so using that [no platform specific] logic we should head over to the os module and remove: ctermid, getenv, getegid... Okay, I'm tired of typing, but that list is not even half-way through the os page, and those are all methods or attributes that

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

2014-07-09 22:44 GMT+02:00 Ethan Furman : > On 07/09/2014 01:24 PM, Victor Stinner wrote: >> Sorry, I didn't follow the whole discussion. IMO DirEntry must use >> methods and you should not expose nor document which infos are already >> provided by the OS or not. DirEntry should be a best-effort bl

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

I really don't understand why you *want* a worse, much less cross-platform API? > Okay, so using that logic we should head over to the os module and remove: > > ctermid, getenv, getegid... > > Okay, I'm tired of typing, but that list is not even half-way through the os > page, and those are all me

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> On a system which did not supply is_dir automatically I would write that as: > > for entry in os.scandir(path): # info defaults to 'os', which is > basically None in this case > if ignore_entry(entry.name): > continue > if os.path.isdir(entry.full_name): > # do

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 01:24 PM, Victor Stinner wrote: Sorry, I didn't follow the whole discussion. IMO DirEntry must use methods and you should not expose nor document which infos are already provided by the OS or not. DirEntry should be a best-effort black-box object providing an API similar to pathlib

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 01:57 PM, Paul Moore wrote: On 9 July 2014 21:24, Victor Stinner wrote: Example where you may sometimes need is_dir(), but not always --- for entry in os.scandir(path): if ignore_entry(entry.name): # this entry is not interesting, lstat_result is useless here contin

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 July 2014 21:24, Victor Stinner wrote: > Example where you may sometimes need is_dir(), but not always > --- > for entry in os.scandir(path): > if ignore_entry(entry.name): > # this entry is not interesting, lstat_result is useless here > continue > if entry.is_dir(): # fetch r

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

2014-07-09 21:59 GMT+02:00 Ben Hoyt : > Other python-devers, please chime in with your thoughts or votes. Sorry, I didn't follow the whole discussion. IMO DirEntry must use methods and you should not expose nor document which infos are already provided by the OS or not. DirEntry should be a best-e

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> 1) it has an additional "info" argument, the values of which have to >> be documented ('os', 'type', 'lstat', and what each one means) >> 2) it has an additional "onerror" argument, the signature of which and >> fairly complicated return value is non-obvious and has to be >> documented >> 3) it

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 12:03 PM, Ben Hoyt wrote: So here's the ways in which option #2 is now more complicated than option #1: 1) it has an additional "info" argument, the values of which have to be documented ('os', 'type', 'lstat', and what each one means) 2) it has an additional "onerror" argument, t

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

This is just getting way too complex ... further thoughts below. >> This is an interesting idea, but it's just getting more and more >> complex, and I'm guessing that being able to change the attributes of >> DirEntry will make the C implementation more complex. >> >> Also, I'm not sure it's very

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 11:04 AM, Paul Moore wrote: On 9 July 2014 17:35, Ethan Furman wrote: More specifically, if we go with choice 1 (no built-in error handling, no mutable DirEntry), how would I implement choice 2? Would I have to write my own CustomDirEntry object? Having built-in error handling

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 10:10 AM, Paul Moore wrote: On 9 July 2014 14:22, Ben Hoyt wrote: One issue with option #2 that I just realized -- does scandir yield the entry at all if there's a stat error? It can't really, because the caller will except the .lstat attribute to be set (assuming he asked for typ

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 July 2014 17:35, Ethan Furman wrote: > More specifically, if we go with choice 1 (no built-in error handling, no > mutable DirEntry), how would I implement choice 2? Would I have to write my > own CustomDirEntry object? Having built-in error handling is, I think, a key point. That's where #

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 08:35 AM, Ben Hoyt wrote: One issue with option #2 that I just realized -- does scandir yield the entry at all if there's a stat error? It can't really, because the caller will expect the .lstat attribute to be set (assuming he asked for type='lstat') but it won't be. Is effectivel

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 July 2014 14:22, Ben Hoyt wrote: > One issue with option #2 that I just realized -- does scandir yield > the entry at all if there's a stat error? It can't really, because the > caller will except the .lstat attribute to be set (assuming he asked > for type='lstat') but it won't be. Is effect

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> One issue with option #2 that I just realized -- does scandir yield the >> entry at all if there's a stat error? It >> can't really, because the caller will expect the .lstat attribute to be >> set (assuming he asked for type='lstat') but >> >> it won't be. Is effectively removing these entries

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> The PEP says that DirEntry should mimic pathlib.Path, so I think that >> DirEntry.is_dir() should work as os.path.isir(): if the entry is a >> symbolic link, you should follow the symlink to get the status of the >> linked file with os.stat(). > > Would this not "break" the tree size script bein

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 July 2014 16:05, Victor Stinner wrote: > The PEP says that DirEntry should mimic pathlib.Path, so I think that > DirEntry.is_dir() should work as os.path.isir(): if the entry is a > symbolic link, you should follow the symlink to get the status of the > linked file with os.stat(). Would this

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

2014-07-09 15:12 GMT+02:00 Ben Hoyt : >> Ok, so it means that your example grouping files per type, files and >> directories, is also wrong. Or at least, it behaves differently than >> os.walk(). You should put symbolic links to directories in the "dirs" >> list too. >> >> if entry.is_dir(): # is

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 06:41 AM, Ethan Furman wrote: Leave it up to the onerror handler. If it returns None, skip yielding the entry, otherwise yield whatever it returned -- which also means the error handler should be able to set fields on the DirEntry: def log_err(exc, entry): logger.warn

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 8 Jul 2014, at 15:52, Ben Hoyt wrote: Hi folks, After some very good python-dev feedback on my first version of PEP 471, I've updated the PEP to clarify a few things and added various "Rejected ideas" subsections. Here's a link to the new version (I've also copied the full text below): http

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 06:22 AM, Ben Hoyt wrote: One issue with option #2 that I just realized -- does scandir yield the entry at all if there's a stat error? It can't really, because the caller will expect the .lstat attribute to be set (assuming he asked for type='lstat') but it won't be. Is effectiv

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/09/2014 05:48 AM, Ben Hoyt wrote: So how about tweaking option #2 a tiny bit more to this: def scandir(path='.', info=None, onerror=None): ... * if info is None (the default), only the .name and .full_name attributes are present * if info is 'type', scandir ensures the is_dir/is_file/is_

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 July 2014 14:22, Ben Hoyt wrote: >> So maybe the onerror function should also receive the DirEntry object >> - which will only have the name and full_name attributes, but that's >> all that is needed. > > That's an interesting idea -- though enough of a deviation from > os.walk()'s onerror th

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Option 2: > def log_err(exc): > logger.warn("Cannot stat {}".format(exc.filename)) > > def get_tree_size(path): > total = 0 > for entry in os.scandir(path, info='lstat', onerror=log_err): > if entry.is_dir: > total += get_tree_size(entry.full_name) > else:

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 July 2014 13:48, Ben Hoyt wrote: > Okay folks -- please respond: option #1 as per the current PEP 471, or > option #2 with Ethan's multi-level thing tweaks as per the above? I'm probably about 50/50 at the moment. What will swing it for me is likely error handling, so let's try both approach

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Ok, so it means that your example grouping files per type, files and > directories, is also wrong. Or at least, it behaves differently than > os.walk(). You should put symbolic links to directories in the "dirs" > list too. > > if entry.is_dir(): # is_dir() checks os.lstat() > dirs.append(e

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

2014-07-08 22:09 GMT+02:00 Ben Hoyt : >>> I think you're misunderstanding is_dir() and is_file(), as these don't >>> actually call os.stat(). All DirEntry methods either call nothing or >>> os.lstat() to get the stat info on the entry itself (not the >>> destination of the symlink). >> >> >> Oh. Ex

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> In this case because the names are exactly the same as the os versions which > /do/ make a system call. Fair enough. > So if I'm finally understanding the root problem here: > > - listdir returns a list of strings, one for each filename and one for > each directory, and keeps no other O/S

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 9 July 2014 02:08, Ben Hoyt wrote: > Comments and votes, please! +1 on option 1 (current PEP approach) at the moment, but I would like to see how the error handling would look (suppose the function logs files that can't be statted, and assumes a size of 0 for them). The idea of a multi-level e

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/08/2014 06:08 PM, Ben Hoyt wrote: Just like an attribute does not imply a system call, having a method named 'is_dir' /does/ imply a system call, and not having one can be just as misleading. Why does a method imply a system call? os.path.join() and str.lower() don't make system calls. I

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> I did better than that -- I read the whole thing! ;) Thanks. :-) > -1 on the PEP's implementation. > > Just like an attribute does not imply a system call, having a > method named 'is_dir' /does/ imply a system call, and not > having one can be just as misleading. Why does a method imply a sy

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/08/2014 01:22 PM, Ethan Furman wrote: I think caching the attributes for DirEntry is fine, but let's do it as a snapshot of that moment in time, not name now, and attributes in 30 minutes when we finally get to you because we had a lot of processing/files ahead of you (you being a DirEnt

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/08/2014 12:34 PM, Ben Hoyt wrote: Better to just have the attributes be None if they were not fetched. None is better than hasattr anyway, at least in the respect of not having to catch exceptions to function properly. The thing is, is_dir() and lstat() are not attributes (for a good re

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> I think you're misunderstanding is_dir() and is_file(), as these don't >> actually call os.stat(). All DirEntry methods either call nothing or >> os.lstat() to get the stat info on the entry itself (not the >> destination of the symlink). > > > Oh. Extract of your PEP: "is_dir(): like os.path.is

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

Le mardi 8 juillet 2014, Ben Hoyt a écrit : > > > It is not clear to me which methods share the cache. > > > > On UNIX, is_dir() and is_file() call os.stat(); whereas lstat() and > > is_symlink() call os.lstat(). > > > > If os.stat() says that the file is not a symlink, I guess that you can > > u

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Better to just have the attributes be None if they were not fetched. None > is better than hasattr anyway, at least in the respect of not having to > catch exceptions to function properly. The thing is, is_dir() and lstat() are not attributes (for a good reason). Please read the relevant "Rejec

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 07/08/2014 11:05 AM, Ben Hoyt wrote: Only exposing what the OS provides for free will make the API too difficult to use in the common case. But is there a nice way to expand the API that will allow the user who is trying to avoid extra expense know what information is already available? Even

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Only exposing what the OS provides for free will make the API too difficult > to use in the common case. But is there a nice way to expand the API that > will allow the user who is trying to avoid extra expense know what > information is already available? > > Even if the initial version doesn't

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> I remember a pending question on python-dev: > > - Martin von Loewis asked if the scandir generator would have send() > and close() methods as any Python generator. I didn't see a reply on > the mailing (nor in the PEP). Good call. Looks like you're referring to this message: https://mail.python

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

On 7/8/2014 9:52 AM, Ben Hoyt wrote: DirEntry fields being "static" attribute-only objects - In `this July 2014 python-dev message `_, Paul Moore suggested a solution that was

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

Hi, 2014-07-08 15:52 GMT+02:00 Ben Hoyt : > After some very good python-dev feedback on my first version of PEP > 471, I've updated the PEP to clarify a few things and added various > "Rejected ideas" subsections. Here's a link to the new version (I've > also copied the full text below): Thanks,

[Python-Dev] Updates to PEP 471, the os.scandir() proposal

Hi folks, After some very good python-dev feedback on my first version of PEP 471, I've updated the PEP to clarify a few things and added various "Rejected ideas" subsections. Here's a link to the new version (I've also copied the full text below): http://legacy.python.org/dev/peps/pep-0471/ -- n