[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Donald Stufft]

Changes by Donald Stufft :


--
pull_requests: +851

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Xiang Zhang]

Xiang Zhang added the comment:

It works. Close. :-)

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

test_walk_bad_dir renamed one of subdirs and tearDown failed to clean up 
changed tree. An error happened depending on the order of listed 
subdirectories: ['SUB1', 'SUB2'] -- passed, ['SUB2', 'SUB1'] -- failed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Roundup Robot]

Roundup Robot added the comment:

New changeset 655510dd46fd by Serhiy Storchaka in branch '3.5':
Issue #28353: Make test_os.WalkTests.test_walk_bad_dir stable.
https://hg.python.org/cpython/rev/655510dd46fd

New changeset df28e536f19d by Serhiy Storchaka in branch '3.6':
Issue #28353: Make test_os.WalkTests.test_walk_bad_dir stable.
https://hg.python.org/cpython/rev/df28e536f19d

New changeset aa891d13e1cf by Serhiy Storchaka in branch 'default':
Issue #28353: Make test_os.WalkTests.test_walk_bad_dir stable.
https://hg.python.org/cpython/rev/aa891d13e1cf

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Xiang Zhang]

Xiang Zhang added the comment:

Seems like the dir you add breaks other cases. I change addCleanUp to os.chmod 
and get:

==
FAIL: test_file_like_path (test.test_os.BytesWalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in 
test_file_like_path
self.test_walk_prune(_PathLike(self.walk_path))
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2

==
FAIL: test_walk_bottom_up (test.test_os.BytesWalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in 
test_walk_bottom_up
self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), 
('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 
'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), 
('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', 
['SUB2', 'SUB1'], ['tmp1'])]

==
FAIL: test_walk_prune (test.test_os.BytesWalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2

==
FAIL: test_walk_topdown (test.test_os.BytesWalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in 
test_walk_topdown
self.assertEqual(len(all), 4)
AssertionError: 5 != 4

==
FAIL: test_file_like_path (test.test_os.FwalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in 
test_file_like_path
self.test_walk_prune(_PathLike(self.walk_path))
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2

==
FAIL: test_walk_bottom_up (test.test_os.FwalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in 
test_walk_bottom_up
self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), 
('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 
'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), 
('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', 
['SUB2', 'SUB1'], ['tmp1'])]

==
FAIL: test_walk_prune (test.test_os.FwalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2

==
FAIL: test_walk_topdown (test.test_os.FwalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in 
test_walk_topdown
self.assertEqual(len(all), 4)
AssertionError: 5 != 4

==
FAIL: test_file_like_path (test.test_os.WalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in 
test_file_like_path
self.test_walk_prune(_PathLike(self.walk_path))
  File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2

==
FAIL: test_walk_bottom_up (test.test_os.WalkTests)
--
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in 
test_walk_bottom_up
self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), 
('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Hmm, tests were passed when I ran them before committing. Seems this is 
heisenbug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Xiang Zhang]

Xiang Zhang added the comment:

Serhiy, after your commits, test_os requires root privileges or it'll fail. 
This is not the case before.

[cpython]$ ./python -m test test_os
Run tests sequentially
0:00:00 [1/1] test_os
test test_os crashed -- Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 164, in 
runtest_inner
test_runner()
  File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 163, in 
test_runner
support.run_unittest(tests)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1849, in 
run_unittest
_run_suite(suite)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1824, in 
_run_suite
raise TestFailed(err)
test.support.TestFailed: multiple errors occurred; run in verbose mode for 
details

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 167, in 
runtest_inner
test_time = time.time() - start_time
  File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 278, in 
__exit__
restore(original)
  File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 235, in 
restore_files
support.rmtree(fn)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree
_rmtree(path)
  File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd
onerror(os.open, fullname, sys.exc_info())
  File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd
dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'SUB21'

'test_os' left behind directory '@test_23400_tmp' and it couldn't be removed: 
[Errno 13] Permission denied: 'SUB21'
test_os failed

1 test failed:
test_os

Total duration: 292 ms
Tests result: FAILURE
Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 914, in 
temp_dir
yield path
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in 
temp_cwd
yield cwd_dir
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main
self._main(tests, kwargs)
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 497, in _main
sys.exit(len(self.bad) > 0 or self.interrupted)
SystemExit: True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
  File "/home/angwer/cpython/Lib/runpy.py", line 85, in _run_code
exec(code, run_globals)
  File "/home/angwer/cpython/Lib/test/__main__.py", line 2, in 
main()
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 532, in main
Regrtest().main(tests=tests, **kwargs)
  File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main
self._main(tests, kwargs)
  File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__
self.gen.throw(type, value, traceback)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in 
temp_cwd
yield cwd_dir
  File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__
self.gen.throw(type, value, traceback)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 917, in 
temp_dir
rmtree(path)
  File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree
_rmtree(path)
  File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd
onerror(os.open, fullname, sys.exc_info())
  File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd
dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'SUB21'

--
nosy: +xiang.zhang
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Roundup Robot]

Roundup Robot added the comment:

New changeset 470224ec16b6 by Serhiy Storchaka in branch '3.5':
Issue #28353: Fixed tests of os.fwalk() with broken links.
https://hg.python.org/cpython/rev/470224ec16b6

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Roundup Robot]

Roundup Robot added the comment:

New changeset ec12e16ea6a1 by Serhiy Storchaka in branch '3.6':
Issue #28353: Try to fix tests.
https://hg.python.org/cpython/rev/ec12e16ea6a1

New changeset a0913dbadea6 by Serhiy Storchaka in branch 'default':
Issue #28353: Try to fix tests.
https://hg.python.org/cpython/rev/a0913dbadea6

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Roundup Robot]

Roundup Robot added the comment:

New changeset fe1fea4ded04 by Serhiy Storchaka in branch '3.5':
Issue #28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/fe1fea4ded04

New changeset e99ec3c77a63 by Serhiy Storchaka in branch '3.6':
Issue #28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/e99ec3c77a63

New changeset 33a1a3dd0051 by Serhiy Storchaka in branch 'default':
Issue #28353: os.fwalk() no longer fails on broken links.
https://hg.python.org/cpython/rev/33a1a3dd0051

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is a patch with tests. Needed to test it on Windows.

--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
keywords: +patch
Added file: http://bugs.python.org/file44982/fwalk_oserror.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Samson Lee]

Samson Lee added the comment:

This is definitely a bug because there is no way an unhandled exception like 
this is acceptable behaviour. The behaviour is that the fwalk iteration will 
stop and there is no way to recover / continue. Also fwalk takes an onerror 
callback function that won't be called with this error.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Mariatta Wijaya]

Changes by Mariatta Wijaya :


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yes, there is a difference between os.walk() and os.fwalk() in some cases. I 
don't know whether it was introduced deliberately or by accident. os.fwalk() 
was added in issue13734, FileNotFoundError is used since issue14773.

--
nosy: +hynek, loewis, neologix
type:  -> behavior
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Samson Lee]

Samson Lee added the comment:

Sorry, the target file needs to be in a directory you don't have permission to 
access:

$ mkdir handsoff
$ sudo chown root:root handsoff
$ sudo chmod 700 handsoff
$ ln -s handsoff/anything blah

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I can't reproduce the issue.

$ mkdir testdir
$ touch testdir/handsoff
$ sudo chown root:root testdir/handsoff
$ sudo chmod 700 testdir/handsoff
$ ln -s handsoff testdir/blah
$ python3.5
>>> import os
>>> list(os.walk('testdir'))
[('testdir', [], ['handsoff', 'blah'])]
>>> list(os.fwalk('testdir'))
[('testdir', [], ['handsoff', 'blah'], 3)]

Ubuntu 16.04, Linux 4.4, tested on ext4 and tmpfs filesystems.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Samson Lee]

Samson Lee added the comment:

Similar to http://bugs.python.org/issue25860

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28353] os.fwalk() unhandled exception when error occurs accessing symbolic link target

[Samson Lee]

New submission from Samson Lee:

The bug is os.fwalk() crashes with unhandled exception when there is an error
accessing symbolic link targets.

To reproduce the bug, create a symbolic link that targets a file that you do not
have permission to access:

$ touch handsoff
$ sudo chown root:root handsoff
$ sudo chmod 700 handsoff
$ ln -s handsoff blah

Now, os.fwalk() fails:

>>> for root, dirs, files, fd in os.fwalk('.'):
... print(root, dirs, files)
...
Traceback (most recent call last):
  File "test_fwalk_permission_error.py", line 3, in 
for root, dirs, files, fd in os.fwalk('.'):
  File "/usr/lib64/python3.5/os.py", line 520, in fwalk
yield from _fwalk(topfd, top, topdown, onerror, follow_symlinks)
  File "/usr/lib64/python3.5/os.py", line 537, in _fwalk
if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
PermissionError: [Errno 13] Permission denied: 'blah'


The cause of the problem is in this part of os.py:

for name in names:
try:
# Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
# walk() which reports symlinks to directories as directories.
# We do however check for symlinks before recursing into
# a subdirectory.
if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
dirs.append(name)
else:
nondirs.append(name)
except FileNotFoundError:
try:
# Add dangling symlinks, ignore disappeared files
if st.S_ISLNK(stat(name, dir_fd=topfd, follow_symlinks=False)
.st_mode):
nondirs.append(name)
except FileNotFoundError:
continue

To fix it, simply replace FileNotFoundError with more general OSError.

Cheers

--
components: Library (Lib)
messages: 278016
nosy: Samson Lee
priority: normal
severity: normal
status: open
title: os.fwalk() unhandled exception when error occurs accessing symbolic link 
target
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com