Hi Ivan. Apologies for my late reply. I've been caught in other tasks and events, but recently I've encountered the behavior again and also similar pattern was observed in environment of one of our enterprise customer.
All the dumps from svnserve, our application and also the memory dump can be downloaded at https://drive.google.com/open?id=0B-yalijSzAIULWNuRGNGUW9KeWM. I've been using the binaries from http://de.apachehaus.com/downloads/mod_svn-1.9.3-ap24-x64.zip. So to answer your questions.. 1. Do you have debug symbols for Subversion binaries you're using? A: I'm not an expert in C development, so I will need to consult with one of my developers to understand, what you actually ask for. Maybe you can figure the necessary details from the binaries themselves, or maybe you can give me bit more hint in how to get the information you seek. 2. Other threads should have different stack trace, because AFAIK only one thread calls accept(). A: Correct. All the stacktraces are captured in the zip file above. Majority of the threads are in the following state, but our application does not receive any response for a long time. ntoskrnl.exe!KeSynchronizeExecution+0x3de6 ntoskrnl.exe!KeWaitForMutexObject+0xc7a ntoskrnl.exe!KeWaitForMutexObject+0x709 ntoskrnl.exe!KeWaitForMutexObject+0x375 ntoskrnl.exe!IoThreadToProcess+0xff0 ntoskrnl.exe!KeRemoveQueueEx+0x16ba ntoskrnl.exe!KeWaitForMutexObject+0xe8e ntoskrnl.exe!KeWaitForMutexObject+0x709 ntoskrnl.exe!KeWaitForMutexObject+0x375 ntoskrnl.exe!NtWaitForSingleObject+0xf2 ntoskrnl.exe!setjmpex+0x3963 ntdll.dll!NtWaitForSingleObject+0x14 MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155 MSWSOCK.dll!WSPStartup+0x3b1c WS2_32.dll!WSARecv+0x17d libapr-1.dll!apr_socket_recv+0x4a svnserve.exe+0x15895 svnserve.exe+0x10a6d svnserve.exe+0x1175b svnserve.exe+0x1f8f svnserve.exe+0xbeb9 libaprutil-1.dll!apr_thread_pool_top+0x797 MSVCR110.dll!beginthreadex+0x107 MSVCR110.dll!endthreadex+0x192 KERNEL32.DLL!BaseThreadInitThunk+0x22 ntdll.dll!RtlUserThreadStart+0x34 3. Could you please create full memory dump of locked process and send it to me including debug symbols? Dump is part of the zip file and for the debug symbol, I'll follow up on that during the next week with my developers. Thank you, Radek Krotil -----Original Message----- From: Ivan Zhakov [mailto:i...@visualsvn.com] Sent: Friday, April 15, 2016 9:46 AM To: Radek Krotil Cc: dev@subversion.apache.org Subject: Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T) On 14 April 2016 at 19:14, Radek Krotil <radek.kro...@polarion.com> wrote: > Hi all. > > Our application generates lot of concurrent read requests to > subversion using svn: protocol. When we tested the multithreaded mode > of svnserve after upgrade to 1.9.3, we noticed strange 'deadlock-like' > behavior: at some point all the requests are blocked in svnserve and > wait there for a few minutes (3 to 15 minutes, no CPU activity), after > which they continue to work. This is making our application significantly > slower. > > Operating system: Windows 10, CentOS 6.6, CentOS 7.2 > > The release and/or revision of Subversion: 1.9.3 > > The compiler and configuration options you built Subversion with: > WANDisco binaries for CentOS, Apache Haus binaries for Windows > > The workaround on Linux is to run svnserve without -T switch, i.e. not > using multithreaded mode. For Windows, there is no workaround as > svnserve only supports the multi-threaded mode. > > Here is a sample of thread dump of svnserve.exe during the 'deadlock' > obtained on Windows 10 using Process Explorer: > > ntoskrnl.exe!KeSynchronizeExecution+0x3de6 > ntoskrnl.exe!KeWaitForMutexObject+0xc7a > ntoskrnl.exe!KeWaitForMutexObject+0x709 > ntoskrnl.exe!KeWaitForMutexObject+0x375 > ntoskrnl.exe!IoThreadToProcess+0xff0 > ntoskrnl.exe!KeRemoveQueueEx+0x16ba > ntoskrnl.exe!KeWaitForMutexObject+0xe8e > ntoskrnl.exe!KeWaitForMutexObject+0x709 > ntoskrnl.exe!KeWaitForMutexObject+0x375 > ntoskrnl.exe!NtWaitForSingleObject+0xf2 > ntoskrnl.exe!setjmpex+0x3963 > ntdll.dll!NtWaitForSingleObject+0x14 > MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155 > MSWSOCK.dll+0x1bf1 > WS2_32.dll!WSAAccept+0xce > > WS2_32.dll!accept+0x12 > libapr-1.dll!apr_socket_accept+0x46 > svnserve.exe+0xc11c > svnserve.exe+0xbae5 > svnserve.exe+0xaf6c > svnserve.exe+0x13ab > KERNEL32.DLL!BaseThreadInitThunk+0x22 > ntdll.dll!RtlUserThreadStart+0x34 > > The similar stack can be seen with other threads too. > 1. Do you have debug symbols for Subversion binaries you're using? 2. Other threads should have different stack trace, because AFAIK only one thread calls accept(). 3. Could you please create full memory dump of locked process and send it to me including debug symbols? -- Ivan Zhakov