Great, I'll try it out. Is the console app in that gist enough for a test case?
@Mike @Jonathan we've faced bugs with previous versions of libc and networking before, also some kernel issues. Update to latest if you can. I can't reproduce with 3.12. I get timeouts but then it recovers when there are available threads unlike with 4.x On Fri, 12 Feb 2016 13:50 Alan <alan.mcgov...@gmail.com> wrote: > It's also worth pointing out that the threadpool implementation has > changed completely since mono 4.0. I believe the new threadpool > implementation shipped as the default starting with mono 4.2 (or > thereabouts). If you're on older Monos the odds are high whatever issue you > have has been fixed already. > > Alan > > On 12 February 2016 at 13:48, Alan <alan.mcgov...@gmail.com> wrote: > >> Hey, >> >> We have just fixed some issues in that area. They are expected to ship as >> part of a the next mono 4.3+ release. If you want to test them out in the >> meantime you could try building mono with this PR [0] and see if it >> resolves all your issues. If it doesn't then a testcase and bug report on >> http://bugzilla.xamarin.com would be awesome! >> >> Thanks! >> >> [0] https://github.com/mono/mono/pull/2576 >> >> On 12 February 2016 at 12:33, Mike Horsley <mhors...@vqcomms.com> wrote: >> >>> we’ve also seen instances of webrequest timeouts that don’t recover (but >>> curl worked) as well but haven’t got to the bottom of it yet. >>> >>> >>> >>> we ran your test app and see the same issue with mono 3.12 on OpenSUSE >>> 13.2 (kernel 3.16.7, libc 2.19). >>> >>> >>> >>> we’ll add the diagnostics from your test app into ours and this will >>> tell us whether we are seeing the same issue with the threadpool. >>> >>> >>> >>> regards >>> >>> Mike >>> >>> >>> >>> *From:* mono-devel-list-boun...@lists.ximian.com [mailto: >>> mono-devel-list-boun...@lists.ximian.com] *On Behalf Of *Seif Attar >>> *Sent:* Friday, February 12, 2016 10:43 AM >>> *To:* mono-devel-list@lists.ximian.com >>> *Subject:* [Mono-dev] WebRequest timeouts after ThreadPool exhaustion >>> >>> >>> >>> Hi, >>> >>> We are running mono 4.2.2 in prod and the VM that the process was >>> running on had SAN failure and after storage recovered, all outgoing >>> requests were timing out, even though doing a curl was working fine. >>> >>> Theory was that thread pool starved and somehow things didn't recover >>> properly. >>> >>> Managed to reproduce with: >>> >>> >>> https://gist.githubusercontent.com/seif/ae2defbfa5afa26fa8f6/raw/bef351eded56c882658a4bad60fa4818ad32d629/timeout.cs >>> >>> Even after ThreadPool finishes the tasks and has available threads, it >>> never recovers and subsequent webrequests all timeout. >>> >>> Running on mono 4.2.2, linux kernel 4.2.0-27 and libc 2.21. >>> >>> Output from gdb is: >>> >>> (gdb) info threads >>> Id Target Id Frame >>> 13 Thread 0x7fca903ff700 (LWP 27944) "cli" >>> pthread_cond_wait@@GLIBC_2.3.2 >>> () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 >>> 12 Thread 0x7fca90b34700 (LWP 27945) "Finalizer" 0x00007fca911d70c9 >>> in futex_abstimed_wait (cancel=true, private=<optimised out>, abstime=0x0, >>> expected=0, futex=0x948ae0) at sem_waitcommon.c:42 >>> 11 Thread 0x7fca8dfff700 (LWP 27946) "Timer-Scheduler" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 10 Thread 0x7fca91ba1700 (LWP 27947) "cli" __clock_nanosleep >>> (clock_id=1, flags=1, req=0x7fca91ba0dc0, rem=0x7fca90f134aa >>> <__clock_nanosleep+138>) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48 >>> 9 Thread 0x7fca8ddfe700 (LWP 27948) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 8 Thread 0x7fca8dbfd700 (LWP 27949) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 7 Thread 0x7fca8d7fb700 (LWP 27951) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 6 Thread 0x7fca8d3f9700 (LWP 27953) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 5 Thread 0x7fca8d1f8700 (LWP 27954) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 4 Thread 0x7fca8cff7700 (LWP 27955) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 3 Thread 0x7fca8cdf6700 (LWP 27956) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> 2 Thread 0x7fca8cbf5700 (LWP 27957) "Threadpool work" >>> pthread_cond_timedwait@@GLIBC_2.3.2 () at >>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 >>> * 1 Thread 0x7fca91cf0780 (LWP 27942) "cli" 0x00007fca911d7e0d in >>> read () at ../sysdeps/unix/syscall-template.S:81 >>> >>> Could not reproduce on mono 3.12, but it happens on 4.0.3 and 4.2.2 >>> >>> Is this a known issue? any workarounds? Tried setting MONO_DNS=1 to use >>> the clr dns, but that didn't help. >>> >>> Let me know if there is any more info I need to provide. >>> >>> Thanks, >>> >>> Seif >>> >>> _______________________________________________ >>> Mono-devel-list mailing list >>> Mono-devel-list@lists.ximian.com >>> http://lists.ximian.com/mailman/listinfo/mono-devel-list >>> >>> >> >
_______________________________________________ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list