I would bet the update will fix it. Huge manifests go into memory, in bulk, and in lists. The lists stay around and if Python GC is not working properly, the bulk reads do as well. During processing, there were as many as 3 copies of the manifest in memory, one bulk, one the result of re iteration, and the other in the collection list. That kind of misuse can really screw up GC.
As answer to your question, changing to 'ulimit -n 10240' could help in a very cases, but I would try both individually to see which one does the job. If they are still on the 0.6-series, the 'ulimit' fix is the only one that would work. On Sun, Feb 11, 2018 at 12:34 PM, Michael Terry <m...@mterry.name> wrote: > I'd also be curious if this is because of duplicity <0.7.16 or because > of the ulimit line we use for deja-dup-monitor. It's not clear to me if > I should change the ulimit line or if we just need to update duplicity > for those users affected. > > -- > You received this bug notification because you are a bug assignee. > https://bugs.launchpad.net/bugs/1727653 > > Title: > error: can't start new thread > > Status in Duplicity: > In Progress > Status in deja-dup package in Ubuntu: > Triaged > Status in duplicity package in Ubuntu: > In Progress > > Bug description: > $ uname -a > Linux mbpr13b 4.13.0-16-generic #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC > 2017 x86_64 x86_64 x86_64 GNU/Linux > > duplicity gets started daily via deja-dup automatically. > > Every now and then (approx. every third time) I get the following > error: > > ------------------ > > Failed with an unknown error. > > Traceback (most recent call last): > File "/usr/bin/duplicity", line 1546, in <module> > with_tempdir(main) > File "/usr/bin/duplicity", line 1540, in with_tempdir > fn() > File "/usr/bin/duplicity", line 1391, in main > do_backup(action) > File "/usr/bin/duplicity", line 1468, in do_backup > restore(col_stats) > File "/usr/bin/duplicity", line 731, in restore > restore_get_patched_rop_iter(col_stats)): > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 560, in Write_ROPaths > for ropath in rop_iter: > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 523, in integrate_patch_iters > for patch_seq in collated: > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 389, in yield_tuples > setrorps(overflow, elems) > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 378, in setrorps > elems[i] = iter_list[i].next() > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 107, in filter_path_iter > for path in path_iter: > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 121, in difftar2path_iter > tarinfo_list = [tar_iter.next()] > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 339, in next > self.set_tarfile() > File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line > 333, in set_tarfile > self.current_fp = self.fileobj_iter.next() > File "/usr/bin/duplicity", line 768, in get_fileobj_iter > manifest.volume_info_dict[vol_num]) > File "/usr/bin/duplicity", line 811, in restore_get_enc_fileobj > fileobj = tdp.filtered_open_with_delete("rb") > File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line > 119, in filtered_open_with_delete > fh = FileobjHooked(path.DupPath.filtered_open(self, mode)) > File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 778, > in filtered_open > return gpg.GPGFile(False, self, gpg_profile) > File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 202, > in __init__ > 'logger': self.logger_fp}) > File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", > line 374, in run > create_fhs, attach_fhs) > File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", > line 420, in _attach_fork_exec > process.thread.start() > File "/usr/lib/python2.7/threading.py", line 736, in start > _start_new_thread(self.__bootstrap, ()) > error: can't start new thread > > -------------- > > Restarting deja-dup manually fixes the problem and the next few days, > automatic backup will run correctly until (see above). > > ProblemType: Bug > DistroRelease: Ubuntu 17.10 > Package: duplicity 0.7.12-1ubuntu1 > ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4 > Uname: Linux 4.13.0-16-generic x86_64 > ApportVersion: 2.20.7-0ubuntu3 > Architecture: amd64 > CurrentDesktop: ubuntu:GNOME > Date: Thu Oct 26 10:49:45 2017 > InstallationDate: Installed on 2017-10-20 (5 days ago) > InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 > (20171018) > SourcePackage: duplicity > UpgradeStatus: No upgrade log present (probably fresh install) > > To manage notifications about this bug go to: > https://bugs.launchpad.net/duplicity/+bug/1727653/+subscriptions > -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to deja-dup in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread Status in Duplicity: In Progress Status in deja-dup package in Ubuntu: Triaged Status in duplicity package in Ubuntu: In Progress Bug description: $ uname -a Linux mbpr13b 4.13.0-16-generic #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux duplicity gets started daily via deja-dup automatically. Every now and then (approx. every third time) I get the following error: ------------------ Failed with an unknown error. Traceback (most recent call last): File "/usr/bin/duplicity", line 1546, in <module> with_tempdir(main) File "/usr/bin/duplicity", line 1540, in with_tempdir fn() File "/usr/bin/duplicity", line 1391, in main do_backup(action) File "/usr/bin/duplicity", line 1468, in do_backup restore(col_stats) File "/usr/bin/duplicity", line 731, in restore restore_get_patched_rop_iter(col_stats)): File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 560, in Write_ROPaths for ropath in rop_iter: File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 523, in integrate_patch_iters for patch_seq in collated: File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 389, in yield_tuples setrorps(overflow, elems) File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 378, in setrorps elems[i] = iter_list[i].next() File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 107, in filter_path_iter for path in path_iter: File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 121, in difftar2path_iter tarinfo_list = [tar_iter.next()] File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 339, in next self.set_tarfile() File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 333, in set_tarfile self.current_fp = self.fileobj_iter.next() File "/usr/bin/duplicity", line 768, in get_fileobj_iter manifest.volume_info_dict[vol_num]) File "/usr/bin/duplicity", line 811, in restore_get_enc_fileobj fileobj = tdp.filtered_open_with_delete("rb") File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 119, in filtered_open_with_delete fh = FileobjHooked(path.DupPath.filtered_open(self, mode)) File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 778, in filtered_open return gpg.GPGFile(False, self, gpg_profile) File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 202, in __init__ 'logger': self.logger_fp}) File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run create_fhs, attach_fhs) File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 420, in _attach_fork_exec process.thread.start() File "/usr/lib/python2.7/threading.py", line 736, in start _start_new_thread(self.__bootstrap, ()) error: can't start new thread -------------- Restarting deja-dup manually fixes the problem and the next few days, automatic backup will run correctly until (see above). ProblemType: Bug DistroRelease: Ubuntu 17.10 Package: duplicity 0.7.12-1ubuntu1 ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4 Uname: Linux 4.13.0-16-generic x86_64 ApportVersion: 2.20.7-0ubuntu3 Architecture: amd64 CurrentDesktop: ubuntu:GNOME Date: Thu Oct 26 10:49:45 2017 InstallationDate: Installed on 2017-10-20 (5 days ago) InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 (20171018) SourcePackage: duplicity UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/duplicity/+bug/1727653/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp