[Desktop-packages] [Bug 1878560] Re: Restore fails with “too many open files”
** Changed in: duplicity Status: Opinion => Invalid -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1878560 Title: Restore fails with “too many open files” Status in Duplicity: Invalid Status in duplicity package in Ubuntu: New Bug description: I use duplicity with Backblaze B2, have an incremental backup every hour and a full backup every 90 days. I recently tried to restore a file to the state that it had 14 days ago. Eventually it failed saying “too many open files”. Apparently the last full backup was 2020-03-08 15:00:05Z. So that means that _a lot_ of incremental backups needed to be resolved. These are the last lines of output: Get: duplicity-inc.20200414T190005Z.to.20200414T25Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-619jt1ub-252 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T25Z.to.20200414T210005Z.manifest' (7297) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T25Z.to.20200414T210005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-z7996evy-253 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T210005Z.to.20200414T220005Z.manifest' (5983) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T210005Z.to.20200414T220005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-alqkws80-254 Traceback (innermost last): File "/usr/bin/duplicity", line 100, in with_tempdir(main) File "/usr/bin/duplicity", line 86, in with_tempdir fn() File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1538, in main do_backup(action) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1618, in do_backup restore(col_stats) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 723, in restore if not patchdir.Write_ROPaths(globals.local_path, File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 578, in Write_ROPaths for ropath in rop_iter: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 541, in integrate_patch_iters for patch_seq in collated: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 407, in yield_tuples setrorps(overflow, elems) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 396, in setrorps elems[i] = next(iter_list[i]) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 126, in difftar2path_iter tarinfo_list = [next(tar_iter)] File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 355, in __next__ self.set_tarfile() File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 348, in set_tarfile self.current_fp = next(self.fileobj_iter) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 759, in get_fileobj_iter yield restore_get_enc_fileobj(backup_set.backend, File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 804, in restore_get_enc_fileobj fileobj = tdp.filtered_open_with_delete(u"rb") File "/usr/lib64/python3.8/site-packages/duplicity/dup_temp.py", line 125, in filtered_open_with_delete fh = FileobjHooked(path.DupPath.filtered_open(self, mode)) File "/usr/lib64/python3.8/site-packages/duplicity/path.py", line 802, in filtered_open return gpg.GPGFile(False, self, gpg_profile) File "/usr/lib64/python3.8/site-packages/duplicity/gpg.py", line 225, in __init__ p1 = gnupg.run([u'--decrypt'], create_fhs=gnupg_fhs, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 374, in run process = self._attach_fork_exec(gnupg_commands, args, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 403, in _attach_fork_exec pipe = os.pipe() OSError: [Errno 24] Too many open files I would assume that it stumbles over the 1024 file descriptor limit and it just cannot resolve a backup with that many incremental backups. Basically with the current state of the software my incremental backups are worthless because I cannot restore. Is there something that I can do except doing less frequent incremental updates and more full backups? To manage notifications about this bug go to: https://bugs.launchpad.net/duplicity/+bug/1878560/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1878560] Re: Restore fails with “too many open files”
** Changed in: duplicity Status: New => Opinion -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1878560 Title: Restore fails with “too many open files” Status in Duplicity: Opinion Status in duplicity package in Ubuntu: New Bug description: I use duplicity with Backblaze B2, have an incremental backup every hour and a full backup every 90 days. I recently tried to restore a file to the state that it had 14 days ago. Eventually it failed saying “too many open files”. Apparently the last full backup was 2020-03-08 15:00:05Z. So that means that _a lot_ of incremental backups needed to be resolved. These are the last lines of output: Get: duplicity-inc.20200414T190005Z.to.20200414T25Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-619jt1ub-252 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T25Z.to.20200414T210005Z.manifest' (7297) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T25Z.to.20200414T210005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-z7996evy-253 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T210005Z.to.20200414T220005Z.manifest' (5983) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T210005Z.to.20200414T220005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-alqkws80-254 Traceback (innermost last): File "/usr/bin/duplicity", line 100, in with_tempdir(main) File "/usr/bin/duplicity", line 86, in with_tempdir fn() File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1538, in main do_backup(action) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1618, in do_backup restore(col_stats) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 723, in restore if not patchdir.Write_ROPaths(globals.local_path, File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 578, in Write_ROPaths for ropath in rop_iter: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 541, in integrate_patch_iters for patch_seq in collated: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 407, in yield_tuples setrorps(overflow, elems) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 396, in setrorps elems[i] = next(iter_list[i]) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 126, in difftar2path_iter tarinfo_list = [next(tar_iter)] File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 355, in __next__ self.set_tarfile() File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 348, in set_tarfile self.current_fp = next(self.fileobj_iter) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 759, in get_fileobj_iter yield restore_get_enc_fileobj(backup_set.backend, File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 804, in restore_get_enc_fileobj fileobj = tdp.filtered_open_with_delete(u"rb") File "/usr/lib64/python3.8/site-packages/duplicity/dup_temp.py", line 125, in filtered_open_with_delete fh = FileobjHooked(path.DupPath.filtered_open(self, mode)) File "/usr/lib64/python3.8/site-packages/duplicity/path.py", line 802, in filtered_open return gpg.GPGFile(False, self, gpg_profile) File "/usr/lib64/python3.8/site-packages/duplicity/gpg.py", line 225, in __init__ p1 = gnupg.run([u'--decrypt'], create_fhs=gnupg_fhs, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 374, in run process = self._attach_fork_exec(gnupg_commands, args, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 403, in _attach_fork_exec pipe = os.pipe() OSError: [Errno 24] Too many open files I would assume that it stumbles over the 1024 file descriptor limit and it just cannot resolve a backup with that many incremental backups. Basically with the current state of the software my incremental backups are worthless because I cannot restore. Is there something that I can do except doing less frequent incremental updates and more full backups? To manage notifications about this bug go to: https://bugs.launchpad.net/duplicity/+bug/1878560/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1878560] Re: Restore fails with “too many open files”
** Changed in: duplicity (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1878560 Title: Restore fails with “too many open files” Status in Duplicity: New Status in duplicity package in Ubuntu: New Bug description: I use duplicity with Backblaze B2, have an incremental backup every hour and a full backup every 90 days. I recently tried to restore a file to the state that it had 14 days ago. Eventually it failed saying “too many open files”. Apparently the last full backup was 2020-03-08 15:00:05Z. So that means that _a lot_ of incremental backups needed to be resolved. These are the last lines of output: Get: duplicity-inc.20200414T190005Z.to.20200414T25Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-619jt1ub-252 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T25Z.to.20200414T210005Z.manifest' (7297) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T25Z.to.20200414T210005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-z7996evy-253 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T210005Z.to.20200414T220005Z.manifest' (5983) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T210005Z.to.20200414T220005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-alqkws80-254 Traceback (innermost last): File "/usr/bin/duplicity", line 100, in with_tempdir(main) File "/usr/bin/duplicity", line 86, in with_tempdir fn() File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1538, in main do_backup(action) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1618, in do_backup restore(col_stats) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 723, in restore if not patchdir.Write_ROPaths(globals.local_path, File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 578, in Write_ROPaths for ropath in rop_iter: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 541, in integrate_patch_iters for patch_seq in collated: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 407, in yield_tuples setrorps(overflow, elems) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 396, in setrorps elems[i] = next(iter_list[i]) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 126, in difftar2path_iter tarinfo_list = [next(tar_iter)] File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 355, in __next__ self.set_tarfile() File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 348, in set_tarfile self.current_fp = next(self.fileobj_iter) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 759, in get_fileobj_iter yield restore_get_enc_fileobj(backup_set.backend, File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 804, in restore_get_enc_fileobj fileobj = tdp.filtered_open_with_delete(u"rb") File "/usr/lib64/python3.8/site-packages/duplicity/dup_temp.py", line 125, in filtered_open_with_delete fh = FileobjHooked(path.DupPath.filtered_open(self, mode)) File "/usr/lib64/python3.8/site-packages/duplicity/path.py", line 802, in filtered_open return gpg.GPGFile(False, self, gpg_profile) File "/usr/lib64/python3.8/site-packages/duplicity/gpg.py", line 225, in __init__ p1 = gnupg.run([u'--decrypt'], create_fhs=gnupg_fhs, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 374, in run process = self._attach_fork_exec(gnupg_commands, args, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 403, in _attach_fork_exec pipe = os.pipe() OSError: [Errno 24] Too many open files I would assume that it stumbles over the 1024 file descriptor limit and it just cannot resolve a backup with that many incremental backups. Basically with the current state of the software my incremental backups are worthless because I cannot restore. Is there something that I can do except doing less frequent incremental updates and more full backups? To manage notifications about this bug go to: https://bugs.launchpad.net/duplicity/+bug/1878560/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1878560] Re: Restore fails with “too many open files”
This is a known issue with duplicity. You will need to increase your systems number of open files since it's set so low. I'd suggest 16384 or higher if you have a lot of incrementals. I'd also suggest limiting the number of incrementals and doing more full backups. ** Also affects: duplicity (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1878560 Title: Restore fails with “too many open files” Status in Duplicity: New Status in duplicity package in Ubuntu: New Bug description: I use duplicity with Backblaze B2, have an incremental backup every hour and a full backup every 90 days. I recently tried to restore a file to the state that it had 14 days ago. Eventually it failed saying “too many open files”. Apparently the last full backup was 2020-03-08 15:00:05Z. So that means that _a lot_ of incremental backups needed to be resolved. These are the last lines of output: Get: duplicity-inc.20200414T190005Z.to.20200414T25Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-619jt1ub-252 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T25Z.to.20200414T210005Z.manifest' (7297) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T25Z.to.20200414T210005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-z7996evy-253 Verarbeitung des lokalen Manifest b'/home/mu/.cache/duplicity/17532b308f659397a23dae3345234484/duplicity-inc.20200414T210005Z.to.20200414T220005Z.manifest' (5983) 1 Volumen in Manifest gefunden Get: duplicity-inc.20200414T210005Z.to.20200414T220005Z.vol1.difftar.gpg -> temp/duplicity-a7n2sb9u-tempdir/mktemp-alqkws80-254 Traceback (innermost last): File "/usr/bin/duplicity", line 100, in with_tempdir(main) File "/usr/bin/duplicity", line 86, in with_tempdir fn() File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1538, in main do_backup(action) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 1618, in do_backup restore(col_stats) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 723, in restore if not patchdir.Write_ROPaths(globals.local_path, File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 578, in Write_ROPaths for ropath in rop_iter: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 541, in integrate_patch_iters for patch_seq in collated: File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 407, in yield_tuples setrorps(overflow, elems) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 396, in setrorps elems[i] = next(iter_list[i]) File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 126, in difftar2path_iter tarinfo_list = [next(tar_iter)] File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 355, in __next__ self.set_tarfile() File "/usr/lib64/python3.8/site-packages/duplicity/patchdir.py", line 348, in set_tarfile self.current_fp = next(self.fileobj_iter) File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 759, in get_fileobj_iter yield restore_get_enc_fileobj(backup_set.backend, File "/usr/lib64/python3.8/site-packages/duplicity/dup_main.py", line 804, in restore_get_enc_fileobj fileobj = tdp.filtered_open_with_delete(u"rb") File "/usr/lib64/python3.8/site-packages/duplicity/dup_temp.py", line 125, in filtered_open_with_delete fh = FileobjHooked(path.DupPath.filtered_open(self, mode)) File "/usr/lib64/python3.8/site-packages/duplicity/path.py", line 802, in filtered_open return gpg.GPGFile(False, self, gpg_profile) File "/usr/lib64/python3.8/site-packages/duplicity/gpg.py", line 225, in __init__ p1 = gnupg.run([u'--decrypt'], create_fhs=gnupg_fhs, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 374, in run process = self._attach_fork_exec(gnupg_commands, args, File "/usr/lib64/python3.8/site-packages/duplicity/gpginterface.py", line 403, in _attach_fork_exec pipe = os.pipe() OSError: [Errno 24] Too many open files I would assume that it stumbles over the 1024 file descriptor limit and it just cannot resolve a backup with that many incremental backups. Basically with the current state of the software my incremental backups are worthless because I cannot restore. Is there something that I can do except doing less frequent incremental updates and more full backups? To manage notifications about this bug go to: https://bugs.launchpad.net/duplicity/+bug/1878560/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to :
