I would bet the update will fix it.  Huge manifests go into memory, in
bulk, and in lists.  The lists stay around and if Python GC is not working
properly, the bulk reads do as well.  During processing, there were as many
as 3 copies of the manifest in memory, one bulk, one the result of re
iteration, and the other in the collection list.  That kind of misuse can
really screw up GC.

As answer to your question, changing to 'ulimit -n 10240' could help in a
very cases, but I would try both individually to see which one does the job.

If they are still on the 0.6-series, the 'ulimit' fix is the only one that
would work.

On Sun, Feb 11, 2018 at 12:34 PM, Michael Terry <m...@mterry.name>
wrote:

> I'd also be curious if this is because of duplicity <0.7.16 or because
> of the ulimit line we use for deja-dup-monitor. It's not clear to me if
> I should change the ulimit line or if we just need to update duplicity
> for those users affected.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1727653
>
> Title:
>   error: can't start new thread
>
> Status in Duplicity:
>   In Progress
> Status in deja-dup package in Ubuntu:
>   Triaged
> Status in duplicity package in Ubuntu:
>   In Progress
>
> Bug description:
>   $ uname -a
>   Linux mbpr13b 4.13.0-16-generic #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC
> 2017 x86_64 x86_64 x86_64 GNU/Linux
>
>   duplicity gets started daily via deja-dup automatically.
>
>   Every now and then (approx. every third time) I get the following
>   error:
>
>   ------------------
>
>   Failed with an unknown error.
>
>   Traceback (most recent call last):
>     File "/usr/bin/duplicity", line 1546, in <module>
>       with_tempdir(main)
>     File "/usr/bin/duplicity", line 1540, in with_tempdir
>       fn()
>     File "/usr/bin/duplicity", line 1391, in main
>       do_backup(action)
>     File "/usr/bin/duplicity", line 1468, in do_backup
>       restore(col_stats)
>     File "/usr/bin/duplicity", line 731, in restore
>       restore_get_patched_rop_iter(col_stats)):
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 560, in Write_ROPaths
>       for ropath in rop_iter:
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 523, in integrate_patch_iters
>       for patch_seq in collated:
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 389, in yield_tuples
>       setrorps(overflow, elems)
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 378, in setrorps
>       elems[i] = iter_list[i].next()
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 107, in filter_path_iter
>       for path in path_iter:
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 121, in difftar2path_iter
>       tarinfo_list = [tar_iter.next()]
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 339, in next
>       self.set_tarfile()
>     File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line
> 333, in set_tarfile
>       self.current_fp = self.fileobj_iter.next()
>     File "/usr/bin/duplicity", line 768, in get_fileobj_iter
>       manifest.volume_info_dict[vol_num])
>     File "/usr/bin/duplicity", line 811, in restore_get_enc_fileobj
>       fileobj = tdp.filtered_open_with_delete("rb")
>     File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line
> 119, in filtered_open_with_delete
>       fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
>     File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 778,
> in filtered_open
>       return gpg.GPGFile(False, self, gpg_profile)
>     File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 202,
> in __init__
>       'logger': self.logger_fp})
>     File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 374, in run
>       create_fhs, attach_fhs)
>     File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 420, in _attach_fork_exec
>       process.thread.start()
>     File "/usr/lib/python2.7/threading.py", line 736, in start
>       _start_new_thread(self.__bootstrap, ())
>   error: can't start new thread
>
>   --------------
>
>   Restarting deja-dup manually fixes the problem and the next few days,
>   automatic backup will run correctly until (see above).
>
>   ProblemType: Bug
>   DistroRelease: Ubuntu 17.10
>   Package: duplicity 0.7.12-1ubuntu1
>   ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
>   Uname: Linux 4.13.0-16-generic x86_64
>   ApportVersion: 2.20.7-0ubuntu3
>   Architecture: amd64
>   CurrentDesktop: ubuntu:GNOME
>   Date: Thu Oct 26 10:49:45 2017
>   InstallationDate: Installed on 2017-10-20 (5 days ago)
>   InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64
> (20171018)
>   SourcePackage: duplicity
>   UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1727653/+subscriptions
>

-- 
You received this bug notification because you are a member of Desktop
Packages, which is subscribed to deja-dup in Ubuntu.
https://bugs.launchpad.net/bugs/1727653

Title:
  error: can't start new thread

Status in Duplicity:
  In Progress
Status in deja-dup package in Ubuntu:
  Triaged
Status in duplicity package in Ubuntu:
  In Progress

Bug description:
  $ uname -a
  Linux mbpr13b 4.13.0-16-generic #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC 2017 
x86_64 x86_64 x86_64 GNU/Linux

  duplicity gets started daily via deja-dup automatically.

  Every now and then (approx. every third time) I get the following
  error:

  ------------------

  Failed with an unknown error.

  Traceback (most recent call last):
    File "/usr/bin/duplicity", line 1546, in <module>
      with_tempdir(main)
    File "/usr/bin/duplicity", line 1540, in with_tempdir
      fn()
    File "/usr/bin/duplicity", line 1391, in main
      do_backup(action)
    File "/usr/bin/duplicity", line 1468, in do_backup
      restore(col_stats)
    File "/usr/bin/duplicity", line 731, in restore
      restore_get_patched_rop_iter(col_stats)):
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 560, in 
Write_ROPaths
      for ropath in rop_iter:
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 523, in 
integrate_patch_iters
      for patch_seq in collated:
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 389, in 
yield_tuples
      setrorps(overflow, elems)
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 378, in 
setrorps
      elems[i] = iter_list[i].next()
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 107, in 
filter_path_iter
      for path in path_iter:
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 121, in 
difftar2path_iter
      tarinfo_list = [tar_iter.next()]
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 339, in 
next
      self.set_tarfile()
    File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 333, in 
set_tarfile
      self.current_fp = self.fileobj_iter.next()
    File "/usr/bin/duplicity", line 768, in get_fileobj_iter
      manifest.volume_info_dict[vol_num])
    File "/usr/bin/duplicity", line 811, in restore_get_enc_fileobj
      fileobj = tdp.filtered_open_with_delete("rb")
    File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 119, in 
filtered_open_with_delete
      fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
    File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 778, in 
filtered_open
      return gpg.GPGFile(False, self, gpg_profile)
    File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 202, in 
__init__
      'logger': self.logger_fp})
    File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 
374, in run
      create_fhs, attach_fhs)
    File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 
420, in _attach_fork_exec
      process.thread.start()
    File "/usr/lib/python2.7/threading.py", line 736, in start
      _start_new_thread(self.__bootstrap, ())
  error: can't start new thread

  --------------

  Restarting deja-dup manually fixes the problem and the next few days,
  automatic backup will run correctly until (see above).

  ProblemType: Bug
  DistroRelease: Ubuntu 17.10
  Package: duplicity 0.7.12-1ubuntu1
  ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
  Uname: Linux 4.13.0-16-generic x86_64
  ApportVersion: 2.20.7-0ubuntu3
  Architecture: amd64
  CurrentDesktop: ubuntu:GNOME
  Date: Thu Oct 26 10:49:45 2017
  InstallationDate: Installed on 2017-10-20 (5 days ago)
  InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 (20171018)
  SourcePackage: duplicity
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/duplicity/+bug/1727653/+subscriptions

-- 
Mailing list: https://launchpad.net/~desktop-packages
Post to     : desktop-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~desktop-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to