[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

> For convenience, a handler that retries unlink() and rmdir() could be 
> distributed with shutil. For ease of use, it could be enabled by default on 
> Windows.

+1 on that. I bumped into this many times over the years as occasional and 
hardly reproducible test failures when cleaning up test files/dirs. 
The tricky part is how to distinguish a legitimate "directory is not empty" 
error though.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Eryk Sun]

Eryk Sun  added the comment:

Your case probably isn't due to a anti-malware filesystem filter. Explorer 
keeps handles open to directories to get updates via ReadDirectoryChangesExW. 
It opens watched directories with shared delete  access, so deleting the child 
succeeds. But as discussed above, the directory isn't unlinked from the parent 
until Explorer closes its handle. Apparently it's not fast enough on the 
systems you tested.

As a workaround, you can define an onerror handler for use with shutil.rmtree() 
that retries the rmdir() call in a loop for up to a given timeout period, such 
as 10 ms. For convenience, a handler that retries unlink() and rmdir() could be 
distributed with shutil. For ease of use, it could be enabled by default on 
Windows.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Yu Liu]

Yu Liu  added the comment:

These are results on another slower machine. Note these results are attained on 
Windows 10, while the above on Windows 7. Just in case it has some influence.

$ python test.py
num_retries: 6
wait_time: 0.0008691726957804373
$ python test.py
num_retries: 3
wait_time: 0.0007175661796639806
$ python test.py
num_retries: 6
wait_time: 0.0007962191842657514
$ python test.py
num_retries: 3
wait_time: 0.0006970480045504753
$ python test.py
num_retries: 4
wait_time: 0.0009637842810260455
$ python test.py
num_retries: 4
wait_time: 0.001005390580561765
$ python test.py
num_retries: 3
wait_time: 0.000654301806397339
$ python test.py
num_retries: 6
wait_time: 0.0008857012257329832
$ python test.py
num_retries: 4
wait_time: 0.0009227479307990348
$ python test.py
num_retries: 4
wait_time: 0.0008976701612158615

And this is result without `bar` opened in Explorer.

$ python test.py
num_retries: 0
wait_time: 0.00019834235943055225

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Yu Liu]

Yu Liu  added the comment:

This time I run it a couple of consecutive times manually. The result shows 
that the first time it retried 12 times and the wait time was 
0.0004259171330071893. Then it seems to be steady, and the num_retried is 3 or 
4, and the wait_time is about 0.00025.

I also run it without `bar` opened in Explorer. The num_retried is 0 and the 
wait_time is 4.46745114706336e-05.

Because of some reasons, I can't post the exact results here. I will run it on 
another computer later and post the results if you need all of the exact 
results.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Eryk Sun]

Eryk Sun  added the comment:

A sub-millisecond wait is fairly quick, but it depends on the machine speed. I 
should have included a counter. Try the following. It's not reproducing the 
problem if num_retries doesn't get incremented.

import os
import time

ERROR_DIR_NOT_EMPTY = 145

PARENT_PATH = 'foo'
CHILD_PATH = os.path.join(PARENT_PATH, 'bar')

os.rmdir(CHILD_PATH)

num_retries = 0
t0 = time.perf_counter()

while True:
try:
os.rmdir(PARENT_PATH)
break
except OSError as e:
if e.winerror != ERROR_DIR_NOT_EMPTY:
   raise
num_retries += 1

wait_time = time.perf_counter() - t0

print('num_retries:', num_retries)
print('wait_time:', wait_time)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Yu Liu]

Yu Liu  added the comment:

The result is 0.00026412295632975946 on my computer. Does this mean it is 
caused by an anti-malware program?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Eryk Sun]

Eryk Sun  added the comment:

This is not an uncommon problem. If there's one or more existing references to 
a file or empty directory that were opened with shared delete access, then a 
delete operation will succeed, but the file or directory will not be unlinked 
from the parent directory. The filesystem only unlinks a file or directory if 
its "delete disposition" is set when the last reference is closed. 

Removing the parent directory thus requires a loop that retries the delete 
until it succeeds, i.e. until existing references are closed and the directory 
finally becomes empty and thus deletable. If the problem is caused by an 
anti-malware program, it should typically be resolved within a short time. 
Exactly how long to wait in a retry loop before failing the operation should be 
configurable.

Maybe you can conduct a simple experiment to measure the wait time required in 
your case. Run the following with "bar" opened in Explorer. Substitute the real 
path of "foo" in PARENT_PATH.

import os
import time

ERROR_DIR_NOT_EMPTY = 145

PARENT_PATH = 'foo'
CHILD_PATH = os.path.join(PARENT_PATH, 'bar')

os.rmdir(CHILD_PATH)
t0 = time.perf_counter()

while True:
try:
os.rmdir(PARENT_PATH)
wait_time = time.perf_counter() - t0
break
except OSError as e:
if e.winerror != ERROR_DIR_NOT_EMPTY:
   raise

print(wait_time)

--
components: +IO
nosy: +eryksun
stage:  -> test needed
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33240] shutil.rmtree fails when the inner floder is opened in Explorer on Windows

[Yu Liu]

New submission from Yu Liu :

Given the following directory structure on a Windows machine:
- foo
 - bar

a call to `shutil.rmtree("foo")` will fail when the inner folder `bar` is 
opened in an Explorer. The error message indicates the `foo` directory is not 
empty, while after the execution, although it failed, the `foo` directory is 
empty. So the inner folder `bar` was removed successfully, but `foo` was not. 
And the error message is misleading.

It will not fail when `foo` is opened in an Explorer, neither on Linux system.

--
components: Library (Lib), Windows
messages: 315077
nosy: giampaolo.rodola, paul.moore, philius, steve.dower, tarek, tim.golden, 
zach.ware
priority: normal
severity: normal
status: open
title: shutil.rmtree fails when the inner floder is opened in Explorer on 
Windows
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com