New submission from Thierry Parmentelat <thierry.parmente...@inria.fr>:
I have observed this on a linux box running fedora29
$ python3 --version
Python 3.7.5
$ uname -a
Linux faraday.inria.fr 5.3.11-100.fc29.x86_64 #1 SMP Tue Nov 12 20:41:25 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/fedora-release
Fedora release 29 (Twenty Nine)
============ steps to reproduce:
This assumes that /root is not readable by lambda users
----- as root:
# mkdir /tmp/foo
# cd /tmp/foo
# touch a b d e
# ln -s /root/anywhere c
# ls -l
total 0
-rw-r--r-- 1 root root 0 Nov 22 14:51 a
-rw-r--r-- 1 root root 0 Nov 22 14:51 b
lrwxrwxrwx 1 root root 14 Nov 22 14:53 c -> /root/anywhere
-rw-r--r-- 1 root root 0 Nov 22 14:51 d
-rw-r--r-- 1 root root 0 Nov 22 14:51 e
----- as a lambda user:
we can see all files
$ ls -l /tmp/foo
total 0
-rw-r--r-- 1 root root 0 Nov 22 14:51 a
-rw-r--r-- 1 root root 0 Nov 22 14:51 b
lrwxrwxrwx 1 root root 14 Nov 22 14:53 c -> /root/anywhere
-rw-r--r-- 1 root root 0 Nov 22 14:51 d
-rw-r--r-- 1 root root 0 Nov 22 14:51 e
and with glob.glob() too
In [1]: import glob
In [2]: for filename in glob.glob("/tmp/foo/*"):
...: print(filename)
...:
/tmp/foo/c
/tmp/foo/e
/tmp/foo/d
/tmp/foo/b
/tmp/foo/a
BUT Path.glob() is not working as expected
In [3]: from pathlib import Path
In [4]: for filename in Path("/tmp/foo/").glob("*"):
...: print(filename)
...:
----- If I now I go back as root and remove the problematic file in /tmp/foo
# rm /tmp/foo/c
----- and try again as a lambda user
In [5]: for filename in Path("/tmp/foo/").glob("*"):
...: print(filename)
...:
/tmp/foo/e
/tmp/foo/d
/tmp/foo/b
/tmp/foo/a
============ discussion
in my case in a real application I was getting *some* files - not an empty list
like here.
I ran strace on that real application
it's fairly clear from that output that the odd symlink is causing the scanning
of all files to break instead of continuing (see snip below)
of course the order in which files are read from the disk will impact the
behaviour, that's why I created the symlink last, that might need to be changed
to reproduce successfully in another setup
============ strace extract
<snip>
getdents64(3, /* 189 entries */, 32768) = 8640
getdents64(3, /* 0 entries */, 32768) = 0
close(3) = 0
stat("/var/lib/rhubarbe-images/centos.ndz", {st_mode=S_IFREG|0644,
st_size=1002438656, ...}) = 0
stat("/var/lib/rhubarbe-images/oai-enb.ndz", {st_mode=S_IFREG|0644,
st_size=2840592384, ...}) = 0
<snip>
stat("/var/lib/rhubarbe-images/ubuntu-floodlight.ndz", {st_mode=S_IFREG|0644,
st_size=2559574016, ...}) = 0
stat("/var/lib/rhubarbe-images/ndnsim.ndz", {st_mode=S_IFREG|0644,
st_size=4153409536, ...}) = 0
==> that's the line about the broken symlink in my real app
stat("/var/lib/rhubarbe-images/push-to-preplab.sh", 0x7ffd3ac4a140) = -1 EACCES
(Permission denied)
==> and here it stops scanning files while there are still quite a lot to be
dealt with
write(1, "/var/lib/rhubarbe-images/fedora-"...,
82/var/lib/rhubarbe-images/fedora-31.ndz
/var/lib/rhubarbe-images/fedora-31-ssh.ndz
) = 82
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER,
sa_restorer=0x7fc583705e70}, {sa_handler=0x7fc583936f10, sa_mask=[], \
sa_flags=SA_RESTORER, sa_restorer=0x7fc583705e70}, 8) = 0
sigaltstack(NULL, {ss_sp=0x560a7dac3330, ss_flags=0, ss_size=16384}) = 0
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}, NULL) = 0
exit_group(0) = ?
+++ exited with 0 +++
----------
messages: 357284
nosy: thierry.parmentelat
priority: normal
severity: normal
status: open
title: Path.glob() sometimes misses files that match
type: behavior
versions: Python 3.7
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue38894>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
