[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

** Changed in: duplicity Status: In Progress => Fix Released -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread Status in

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

** Changed in: duplicity Milestone: 0.7.18 => None ** Changed in: duplicity Assignee: Kenneth Loafman (kenneth-loafman) => (unassigned) -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu.

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

>From the permissions error message it looks like you tried to run that as yourself. Try going into root and running the commands again. ** Changed in: duplicity Milestone: None => 0.7.18 -- You received this bug notification because you are a member of Desktop Packages, which is

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

>Try this then: >sh -c "ulimit -v 100; ulimit -n 8192; exec /usr/lib/x86_64-linux- gnu/deja-dup/deja-dup-monitor" >It adds the ulimit -n call and may do what you need now that the systemd settings are in place. I added this and the backup did not start automatically yesterday so I tried to

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Try running the command as: DEJA_DUP_DEBUG=1 sh -c "ulimit -v 100; ulimit -n 8192; exec /usr/lib/x86_64-linux-gnu /deja-dup/deja-dup-monitor" This will create a large debug log in ~/.cache/deja-dup/. Please add that as an attachment to this bug report. -- You received this bug

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Try this then: sh -c "ulimit -v 100; ulimit -n 8192; exec /usr/lib/x86_64-linux-gnu /deja-dup/deja-dup-monitor" It adds the ulimit -n call and may do what you need now that the systemd settings are in place. -- You received this bug notification because you are a member of Desktop

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

>A few more requests then: >What command are you running? Full command line please. This is running through deja-dup on start-up. The command that it's using is: sh -c "ulimit -v 100; exec /usr/lib/x86_64-linux-gnu/deja-dup/deja-dup-monitor" >How much memory do you have available to

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

A few more requests then: What command are you running? Full command line please. How much memory do you have available to duplicity? Is it being run under Deja-Dup, cron, or command line? -- You received this bug notification because you are a member of Desktop Packages, which is subscribed

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I was able to get the open files changed to 65535 using the above link. Today after I started my computer I got the same error I've been getting about not being able to create a new thread. ~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

See this: https://superuser.com/a/1200818 Let me know if it works. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to deja-dup in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread Status in Duplicity:

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

~$ ulimit -n 8192 bash: ulimit: open files: cannot modify limit: Operation not permitted ~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

It's odd you would get an error on 'ulimit -n 8192'. Do it again and paste in the full error message. As to limits.conf, did you logout and log back in? Do a 'ulimit -a' paste that in as well. -- You received this bug notification because you are a member of Desktop Packages, which is

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

** Changed in: duplicity Status: Incomplete => Confirmed ** Changed in: duplicity Assignee: (unassigned) => Kenneth Loafman (kenneth-loafman) ** Changed in: duplicity Assignee: Kenneth Loafman (kenneth-loafman) => (unassigned) ** Changed in: duplicity Assignee:

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I'm unable to increase the ulimit on my system. I've added the new size in limits.conf but that does not help. Running the command manually gives me a permissions error with files open. At this point the upgrade to 0.7.17 did not help so it appears something else need to be look at. -- You

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Try upping your ulimit on num files from 1024 to 8192 or higher. To do this for duplicity, run like so: $ ulimit -n 8192 $ duplicity ... To set this system wide see: https://www.google.com/search?q=linux+increase+number+of+open+files=linux+increase+num=chrome.0.0j69i57j0l4.12567j0j7=chrome=UTF-8

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I'm also seeing this issue and I just purged Duplicity and then added it back with the PPA and I am still seeing the issue. I am using 0.7.17 now. What information can I provide to help fix this? Ubuntu Version: UbuntuMate 17.10 uname -a Linux 4.13.0-36-generic #40-Ubuntu SMP Fri Feb 16 20:07:48

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I think this is fixed in the latest releases, however, the OPs are not responding and I have no test case to run against. Marking this incomplete until someone can confirm that it does not work on 0.7.17 and later. ** Changed in: duplicity Status: In Progress => Incomplete ** Changed in:

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

** Changed in: duplicity Milestone: 0.7.17 => None -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to deja-dup in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread Status in Duplicity: In

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I would bet the update will fix it. Huge manifests go into memory, in bulk, and in lists. The lists stay around and if Python GC is not working properly, the bulk reads do as well. During processing, there were as many as 3 copies of the manifest in memory, one bulk, one the result of re

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I'd also be curious if this is because of duplicity <0.7.16 or because of the ulimit line we use for deja-dup-monitor. It's not clear to me if I should change the ulimit line or if we just need to update duplicity for those users affected. -- You received this bug notification because you are a

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Please upgrade to the current version of duplicity, 0.7.16. This will assure that any bugs fixed since your release are available and may fix your issue. This applies especially to duplicity versions between 0.7.03 and 0.7.14 inclusive. There was a fix in 0.7.15 that reduced memory usage

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

This seems to be a regression introduced by a fix for bug #1302416. ** Changed in: deja-dup (Ubuntu) Status: Confirmed => Triaged ** Changed in: deja-dup (Ubuntu) Importance: Undecided => Critical ** Tags added: regression-release -- You received this bug notification because you

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Just adding -n 4096 or -s 16384 alone or in combination does not do the trick. Only adding these two PLUS removing -v 100 works. This and the fact that on my mbpr15a the original line works fine confuses me. -- You received this bug notification because you are a member of Desktop

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

It's probably the -n option (number of open files). The default is 1024 and that's way too low. On Sun, Jan 28, 2018 at 4:07 AM, Wolf Rogner wrote: > Sorry for the delay, I wanted to make sure my answer is accurate. > > I have three machines set up with 17.10 afresh. > mbpr13a

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Sorry for the delay, I wanted to make sure my answer is accurate. I have three machines set up with 17.10 afresh. mbpr13a 8GB, 4.13.0-25-generic mbpr15a 16GB, 4.13.0-25-generic mbpr13b 16GB, 4.13.0-32-generic mbpr15a does backups ooB mbpr13a/b have the reported issue The original line in

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

On 26 January 2018 at 11:59, Wolf Rogner wrote: > I cannot confirm that ulimit -s 16384 will heal the issue (suggested in > #30). I have inserted ulimit -s 16384 to no avail. > > Exec=sh -c "ulimit -v 100; ulimit -s 16384; ulimit -n 4096; exec >

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I cannot confirm that ulimit -s 16384 will heal the issue (suggested in #30). I have inserted ulimit -s 16384 to no avail. Exec=sh -c "ulimit -v 100; ulimit -s 16384; ulimit -n 4096; exec /usr/lib/x86_64-linux-gnu/deja-dup/deja-dup-monitor" -> still causes the issue. Trying to check my file

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

That does explain why you can't open more gpg threads. Each instance of gpg will take several file handles and eats more resources. I'm going to leave this open for now. Perhaps duplicity should put out a warning after this error telling people to increase their num-files limits, or something

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

@Kenneth: The -v limit was suggested in comment #19. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to deja-dup in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread Status in Duplicity: In Progress

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Thanks for catching that. Was the system installed with those limits, or had you put them in earlier? I've never seen a distro that tried to limit virtual memory. On Wed, Jan 24, 2018 at 12:40 AM, Václav Haisman wrote: > So, I have been experimenting. I have removed ulimit

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

So, I have been experimenting. I have removed ulimit -v 100...0 out of the command line because I have unlimited limit by default. What I have added though is ulimit -s 16384 which doubles stack limit and I have also ulimit -n 2048 to double file limit. Todays backup passed without the usual

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Whoops! You are correct. I thought you were on 0.7.10 from a previous message. OK, what I need to see is: - A debug run (-v9) with the first 200 lines and the last 200 lines, - OS version, Python version, memory size. On Mon, Jan 22, 2018 at 11:01 AM, Václav Haisman

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

@Kenneth: The call stack is clearly different. Also, it does not explain the hanging threads in Duplicity and processes of GPG waiting on pipe write to return. Also, I have been running the PPA version last few months. -- You received this bug notification because you are a member of Desktop

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I think this is a duplicate of https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1730451 and that you've just run out of memory for a new thread. Please upgrade to the newest version and try again. There are three options: * Release tarball Install -

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Debugging the Python process of Duplicity shows no polling or reading from appropriate number of pipes: * 1Thread 0x7f7f510d0740 (LWP 19797) 0x7f7f50ce9f96 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x559f3d5e48f0) at

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

So I have several (over 5) processes in the previously mentioned state: Duplicity with a dozen or so gpg processes under it, hanging, duplicity is being run as `usr/bin/duplicity restore`. I have attached generated a core dump, downloaded symbols and analysed the core dump with GDB. This is the

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: deja-dup (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to deja-dup in Ubuntu.

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

** Also affects: deja-dup (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread Status

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

@Kenneth #17: I verified that the settings recommended in comment #17 change nothing (several kernel updates later) a) if anything, it should be nofile and not nfile b) /etc/security/limits.conf may be altered but any settings there are ignored c) ulimit is a bash parameter and cannot be applied

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Deja-Dup and duplicity seemed to work with kernel 4.13.0-19. After that (21 and 22) the issue reappeared. Very frustrating. Workaround: Start Deja-Dup manually after it failed. Careful: Deja-dup claims that the backup has been carried out even when the above error occurs. In my case, there is

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Hello, Today I hit the same problem as in bug #1722454, MemoryError in manifest.py. I am on Ubuntu 17.10 with duplicity 0.7.12-1ubuntu1. I also found a bugreport in Debian that seems to be the same thing: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=881021 Here is my backtrace: Traceback

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

This will work when duplicity is run from DejaDup, not when it is run from the console or from cron. The system wide solution above will work for all situations. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu.

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

The backup completed succesfully (while running automatically) after I changed in /etc/xdg/autostart/org.gnome.DejaDup.Monitor.desktop from Exec=sh -c "ulimit -v 100; exec /usr/lib/x86_64-linux-gnu/deja-dup /deja-dup-monitor" to Exec=sh -c "ulimit -v 100; ulimit -n 4096; exec

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

>From /etc/xdg/autostart/org.gnome.DejaDup.Monitor.desktop (and https://git.launchpad.net/deja- dup/tree/data/org.gnome.DejaDup.Monitor.desktop.in): # Try to limit memory -- we have reports of runaway deja-dup-monitor processes # but I can't reproduce it. So until we fix whatever is happening

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

ulimit is settable system wide, so, at the end of /etc/security/limits.conf add the lines: *softnfile 16384 *hardnfile 32768 then reboot the system. That will set system wide limits for all tasks. On Sun, Nov 5, 2017 at 4:29 AM, Wolf Rogner

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

@Kenneth: I increased ulimit to 4096 -> same error Then I wanted to understand how ulimit works: ulimit influences the shell (bash!). I can start duplicity via deja-dup from the console -> works I can start deja-dup from the launcher (GNOME-Shell) and manually trigger backup (from the

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Do you have a passphrase on the backup? I don't see any key use, and it looks like gpg-agent is turned off. On the gpg processes, could you run an strace and see what fd it is hanging on? "strace -p " should do it. You can get a debug trace by doing: DEJA_DUP_DEBUG=1 deja-dup --backup from

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I have bumped the file limit to 8k from 4k for last night's backup and still got the same error. I am not sure how to run this in verbose mode. I am using it through deja-dup and its plugin that runs the backup every night. What confuses me are the hanging gpg processes that I have reported

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

0.7.15 is coming out soon, .16 down the road. .15 preview is in the daily builds. Were you able to increase ulimit -n? What setting? I'm going to need a debug log. Could you run this with -v9 and supply two files, 1) the first 200 lines of the log 2) the last 200 lines of the log Please

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Tested 0.7.14-bzr1335-0ubuntu3-ubuntu17.10.1 and it produces the following error: Failed with an unknown error. Traceback (innermost last): File "/usr/bin/duplicity", line 1559, in with_tempdir(main) File "/usr/bin/duplicity", line 1545, in with_tempdir fn() File

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

I have tested with 0.7.14+bzr1335-0ubuntu3~ubuntu17.10.1 and it still fails the same. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread

Re: [Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Daily duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/daily On Tue, Oct 31, 2017 at 3:52 PM, Václav Haisman wrote: > Re #8: Is there a PPA for that? > > -- > You received this bug notification because you are a bug assignee. >

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

Re #8: Is there a PPA for that? -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new thread Status in duplicity package in Ubuntu: In Progress

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

If you could, please test the latest bzr trunk version. I just fixed a major bug that was chewing up a lot of memory. That may help with this problem if the fix above does not work. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to

[Desktop-packages] [Bug 1727653] Re: error: can't start new thread

** Summary changed: - duplicity fails with unknown error + error: can't start new thread -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to duplicity in Ubuntu. https://bugs.launchpad.net/bugs/1727653 Title: error: can't start new