Package: internetarchive
Version: 1.8.1-1
Severity: important
I've tried to upload to archive.org and noticed ia crashes on
large folders.
$ ulimit -n
1024
$ ia upload asdf ./folder-with-more-than-1024-files/
[...]
OSError: [Errno 24] Too many open files
[...]
$
The bug is present in src:python-internetarchive, I found a patch that
resolves the issue from 2018 that was never applied. You can find a
patch that cleanly applies to the current debian/sid below. The original
author is github.com/Arkiver2:
--- >8 ---
diff --git a/internetarchive/utils.py b/internetarchive/utils.py
index db8412a..7385857 100644
--- a/internetarchive/utils.py
+++ b/internetarchive/utils.py
@@ -235,14 +235,16 @@ def recursive_file_count(files, item=None,
checksum=False):
is_dir = False
if is_dir:
for x, _ in iter_directory(f):
- lmd5 = get_md5(open(x, 'rb'))
+ with open(x, 'rb') as f_:
+ lmd5 = get_md5(f_)
if lmd5 in md5s:
continue
else:
total_files += 1
else:
try:
- lmd5 = get_md5(open(f, 'rb'))
+ with open(x, 'rb') as f_:
+ lmd5 = get_md5(f_)
except TypeError:
# Support file-like objects.
lmd5 = get_md5(f)
--- 8< ---
