Even though there is tangible performance improvement from scandir(), it
would be useful to find out if the API fits well.
Got it -- I see where you're coming from now. I'll take a quick look
(hopefully later this week).
-Ben
___
Python-Dev mailing
Hi,
2014-07-20 18:50 GMT+02:00 Antoine Pitrou anto...@python.org:
Have you tried modifying importlib's _bootstrap.py to use scandir() instead
of listdir() + stat()?
IMO the current os.scandir() API does not fit importlib requirements.
importlib usually wants fresh data, whereas DirEntry cache
Victor Stinner wrote:
2014-07-20 18:50 GMT+02:00 Antoine Pitrou anto...@python.org:
Have you tried modifying importlib's _bootstrap.py to use scandir()
instead of listdir() + stat()?
IMO the current os.scandir() API does not fit importlib requirements.
importlib usually wants fresh data,
Thanks for an initial look into this, Victor.
IMO the current os.scandir() API does not fit importlib requirements.
importlib usually wants fresh data, whereas DirEntry cache cannot be
invalidated. It's probably possible to cache some os.stat() result in
importlib, but it looks like it
On 22 Jul 2014 02:46, Steve Dower steve.do...@microsoft.com wrote:
Personally I'd make it a string subclass and put one-shot properties on
it (i.e. call/cache stat() on first access where we don't already know the
answer), which I think is close enough to where it's landed that I'm happy.
(As
2014-07-21 18:48 GMT+02:00 Ben Hoyt benh...@gmail.com:
By the way, DirEntry constructor is not documented in the PEP. Should
we document it? It might be a way to invalidate the cache:
I would prefer not to, just to keep things simple. Similar to creating
os.stat_result() objects ... you can
We should mimic os.stat() and os.stat_result: os.stat_result symbol
exists in the os namespace, but the type constructor is not
documented. No need for extra protection like not adding the type in
the os module, or adding a _ prefix to the name.
Yeah, that works for me.
By the way, it's
Hi,
Thanks Victor, Nick, Ethan, and others for continued discussion on the
scandir PEP 471 (most recent thread starts at
https://mail.python.org/pipermail/python-dev/2014-July/135377.html).
Have you tried modifying importlib's _bootstrap.py to use scandir()
instead of listdir() + stat()?
Have you tried modifying importlib's _bootstrap.py to use scandir() instead
of listdir() + stat()?
No, I haven't -- I'm not familiar with that code. What does
_bootstrap.py do -- does it do a lot of listdir calls and stat-ing of
many files?
-Ben
___
Oh yes. :) The file Antoine is referring to is the implementation of import.
On Sun, Jul 20, 2014, 17:34 Ben Hoyt benh...@gmail.com wrote:
Have you tried modifying importlib's _bootstrap.py to use scandir()
instead
of listdir() + stat()?
No, I haven't -- I'm not familiar with that code.
Le 20/07/2014 17:34, Ben Hoyt a écrit :
Have you tried modifying importlib's _bootstrap.py to use scandir() instead
of listdir() + stat()?
No, I haven't -- I'm not familiar with that code. What does
_bootstrap.py do -- does it do a lot of listdir calls and stat-ing of
many files?
Quite a
Le mardi 15 juillet 2014, Ben Hoyt benh...@gmail.com a écrit :
Victor had one other question:
What happens to name and full_name with followlinks=True?
Do they contain the name in the directory (name of the symlink)
or name of the linked file?
I would say they should contain the name
On 14 Jul 2014 22:50, Ben Hoyt benh...@gmail.com wrote:
In light of that, I propose I update the PEP to basically follow
Victor's model of is_X() and stat() following symlinks by default, and
allowing you to specify follow_symlinks=False if you want something
other than that.
Victor had one
Looks doable. Just make sure the cached entries reflect the
'follow_symlinks' setting -- so a symlink could end up with both an lstat
cached entry and a stat cached entry.
Yes, good point -- basically the functions will use the _stat cache if
follow_symlinks=True, otherwise the _lstat cache.
Sorry, I don't remember who but someone proposed to add the follow_symlinks
parameter in scandir() directly. If the parameter is added to methods,
there is no such issue.
Yeah, I think having the DirEntry methods do different things
depending on how scandir() was called is a really bad idea.
I'd *keep DirEntry.lstat() method* regardless of existence of
.stat(*, follow_symlinks=True) method (despite the slight violation of
DRY principle) for readability. `dir_entry.lstat().st_mode` is more
consice than `dir_entry.stat(follow_symlinks=False).st_mode` and the
meaning of lstat is
On 15 July 2014 13:19, Ben Hoyt benh...@gmail.com wrote:
Hmmm, perhaps. You suggest .full_name implies it's the absolute path,
which isn't true. I don't mind .path, but it kind of sounds like the
Path object associated with this entry. I think full_name is fine
-- it's not abs_name.
On 07/14/2014 11:25 PM, Victor Stinner wrote:
Again: remove any garantee about the cache in the definitions of methods,
instead copy the doc from os.path and os. Add a global remark saying that
most methods don't need any syscall in general, except for symlinks (with
follow_symlinks=True).
I was going to stay out of this one...
On 14Jul2014 10:25, Victor Stinner victor.stin...@gmail.com wrote:
2014-07-14 4:17 GMT+02:00 Nick Coghlan ncogh...@gmail.com:
Or the ever popular symlink to . (or a directory higher in the tree).
. and .. are explicitly ignored by os.listdir() an
2014-07-14 2:33 GMT+02:00 Ben Hoyt benh...@gmail.com:
If we go with Victor's link-following .is_dir() and .is_file(), then
we probably need to add his suggestion of a follow_symlinks=False
parameter (defaults to True). Either that or you have to say
stat.S_ISDIR(entry.lstat().st_mode) instead,
2014-07-14 4:17 GMT+02:00 Nick Coghlan ncogh...@gmail.com:
Or the ever popular symlink to . (or a directory higher in the tree).
. and .. are explicitly ignored by os.listdir() an os.scandir().
I think os.walk() is a good source of inspiration here: call the flag
followlink and default it to
2014-07-14 6:52 GMT+02:00 Ethan Furman et...@stoneleaf.us:
We shoIf you put the option on scandir(), you uld have a flag for that, and
default it to False:
scandir(path, *, followlinks=False, info=None, onerror=None)
What happens to name and full_name with followlinks=True? Do they
contain
First, just to clarify a couple of points.
You forgot one of my argument: we must have exactly the same API than
os.path.is_dir() and pathlib.Path.is_dir(), because it would be very
confusing (source of bugs) to have a different behaviour.
Actually, I specifically included that argument. It's
Nick Coghlan ncogh...@gmail.com writes:
On 13 Jul 2014 20:54, Tim Delaney timothy.c.dela...@gmail.com wrote:
On 14 July 2014 10:33, Ben Hoyt benh...@gmail.com wrote:
If we go with Victor's link-following .is_dir() and .is_file(), then
we probably need to add his suggestion of a
Let's not multiply entities beyond necessity.
There is well-defined *follow_symlinks* parameter
https://docs.python.org/3/library/os.html#follow-symlinks
e.g., os.access, os.chown, os.link, os.stat, os.utime and many other
functions in os module support follow_symlinks parameter, see
On 07/14/2014 07:48 PM, Ben Hoyt wrote:
In any case, here's the modified proposal:
scandir(path='.') - generator of DirEntry objects, which have:
* name: name as per listdir()
* full_name: full path name (not necessarily absolute), equivalent of
os.path.join(path, entry.name)
*
Hi folks,
Thanks Victor, Nick, Ethan, and others for continued discussion on the
scandir PEP 471 (most recent thread starts at
https://mail.python.org/pipermail/python-dev/2014-July/135377.html).
Just an aside ... I was reminded again recently why scandir() matters:
a scandir user emailed me the
On 14 July 2014 10:33, Ben Hoyt benh...@gmail.com wrote:
If we go with Victor's link-following .is_dir() and .is_file(), then
we probably need to add his suggestion of a follow_symlinks=False
parameter (defaults to True). Either that or you have to say
stat.S_ISDIR(entry.lstat().st_mode)
On 13 Jul 2014 20:54, Tim Delaney timothy.c.dela...@gmail.com wrote:
On 14 July 2014 10:33, Ben Hoyt benh...@gmail.com wrote:
If we go with Victor's link-following .is_dir() and .is_file(), then
we probably need to add his suggestion of a follow_symlinks=False
parameter (defaults to True).
On 14 July 2014 12:17, Nick Coghlan ncogh...@gmail.com wrote:
I think os.walk() is a good source of inspiration here: call the flag
followlink and default it to False.
Actually, that's followlinks, and I'd forgotten that os.walk() defaulted
to not follow - definitely behaviour to match IMO :)
On 07/13/2014 05:33 PM, Ben Hoyt wrote:
On the recent python-dev thread, Victor especially made some well
thought out suggestions. It seems to me there's general agreement that
the basic API in PEP 471 is good (with Ethan not a fan at first, but
it seems he's on board after further discussion
31 matches
Mail list logo
