Re: [Mono-dev] Fundamental performance problems with Mono
Hi, Writing my server in a managed language, be it Mono or Java, is a basic choice of effectiveness over cost. I prefer good performance in no time than super-great-performance that takes ages to develop and will almost surely rain upon me bugs from hell. And if you write your managed code correctly (mainly aim towards zero GC, and this includes the framework...) then your performance should be equivalent to a native implementation. If you can point out a managed async socket framework that works I'd be happy to try it. I don't have time to get down and dirty. I got business logic on my hands... Now, from what I understand, the implementation in mono follows the well-known recipe for high-performance socket servers in Linux: event polling (epoll) with I/O Thread pool - you call send()/receive() when the socket is ready. so, yes, eventually you call the simple socket API, but this is how you write high-performance servers in Linux. but more importantly, any socket framework that wants to wrap this model will need primitives like BeginReceive\ReceiveAsync + OnDataReceived callback and BeginSend\SendAsync + OnDataSent calback. can't escape it. so it doesn't matter if the operating system doesn't have native support for async sockets - you end up with more or less the same managed async socket API on Windows and Linux. Therefore, I do blame Mono for the poor performance. If the epoll+I/O Thread pool model is implemented correctly and if the managed async socket API is implemented properly (and there is a lot to benefit from the XXXAsync methods) then there is no reason why a Mono server should not be equal in performance to a .NET on Windows server. And again, this is good enough performance for me. damageboy wrote: Hi Zvika, I'll start by saying that I've been there, I've also seen abysmal performance with the mono async socket implementation. If you'll dig down in the code (I did so last time around Mono 2.2) you should also see that there is no such thing in Mono/Linux at any rate... By this, I mean that a fundamental difference you'll find in the Linux world from the Windows world is that there is no async socket API for Linux. This is a limitation (if you want to call it like that) of the Linux kernel, and in no way related to mono. While calling BeginSend/Receive in Windows + MS.NET is implemented by means of true async sockets on Windows, which ultimately are a Winsock / Windows NT Kernel feature, calling BeginSend on Mono simply queues a work item into the thread pool that will call the normal socket apis. This is a fundamental difference in how Mono/MS.NET work. Feel free to gaze at the code on mcs/class/System/System.Net.Sockets/Socket.cs and see this for yourself... While the Mono people could write two implementations for BeginXXX (one for Windows + async sockets, one for Linux) I don't really blame them for implementing the BeingXXX APIs the way they did. In a way, using a BeginXXX APIs for sockets on Mono generally degrades performance (in terms of overhead and latency for packet send/receive) under heavy load than using the regular non-async apis. This should pretty much leave you asking yourself why would you ever want to use the so-called more advanced XXXAsync Socket API (which was your original intent, as far as I can tell). I personally see very little benefit even if were implemented in Mono. This definitely does not mean that all is lost. On the contrary, you can achieve much higher throughput / lower latency with Mono + Linux, but achieving this with the Microsoft centric APIs / paradigms (as System.Net.Sockets is) is highly unlikely IMO (again, I would like to stress that this is really not Mono's fault). I suggest you read up on the C10K problem either on Wikipedia or Dan Kegel's site: http://www.kegel.com/c10k.html There are many possible solutions, including some that are not mentioned in the C10K page, such as using P/Invoke to call vmsplice/splice for sending/receiving data with Zero Copy networking or, as I've done in the past, wrapping up Evgeniy Polyakov's netchannels and userspace network stack: http://www.ioremap.net/projects/unetstack http://www.ioremap.net/projects/netchannels Although this means getting down and dirty, often using unsafe code and pointers and what not, let me assure you, that you will be able to make a very modest server/desktop machine blow away anything you've ever sen with Windows before. In short, I think you're looking at the wrong problem. Hope this helps. zvikag wrote: Hello all, The bottom line of this message is that I don't see how can one write a high-performance socket server in Mono... Here is the story: I am writing a proxy server using .NET Socket API. This proxy does almost entirely I/O work - copying buffers from one socket to another. Now, Mono doesn't implement the newer http://msdn.microsoft.com/en-us/library
Re: [Mono-dev] Fundamental performance problems with Mono
Did you see this improvement using Mono or .NET on Windows? in your blog post you talk about GC generations that don't exist in Mono GC and OOM exceptions that I don't get. I am also doubtful that the use of pooled buffers can alleviate the problems in Mono since the main problem are the tons of garbage objects that are created during BeginXXX\EndXXX calls, not the pinning of send and receive buffers. Thad wrote: I feel your pain. I pulled a library across that did something similar, and the large GC times had a visible, stop-the-world impact on latency. Switching from garbage collected to managed buffers made a world of difference. For more on this see: http://msdn.microsoft.com/en-us/library/system.servicemodel.channels.buf fermanager.aspx http://codebetter.com/blogs/gregyoung/archive/2007/06/18/async-sockets-a nd-buffer-management.aspx After switching to managed buffers (and replicating some logic similar to the Async socket methods), I've had a pass-through data bridge running for days without a hiccup and with low CPU/memory utilization, even on a fairly underpowered processor. If you do decide to go with managed buffers you'll need to implement your own (or find a good library) as the WCF BufferManager in Mono won't really do what you want. Regards, -Thad -Original Message- From: mono-devel-list-boun...@lists.ximian.com [mailto:mono-devel-list-boun...@lists.ximian.com] On Behalf Of zvikag Sent: Thursday, January 07, 2010 10:38 AM To: mono-devel-list@lists.ximian.com Subject: [Mono-dev] Fundamental performance problems with Mono Hello all, The bottom line of this message is that I don't see how can one write a high-performance socket server in Mono... Here is the story: I am writing a proxy server using .NET Socket API. This proxy does almost entirely I/O work - copying buffers from one socket to another. Now, Mono doesn't implement the newer http://msdn.microsoft.com/en-us/library/system.net.sockets.socketasyncev entargs.aspx XXXAsync Socket API that was introduced in .NET 2.0 SP1 (or more accurately, implements it http://www.mail-archive.com/mono-l...@lists.ximian.com/msg28621.html perfunctorily ). So I was left to use the APM Socket API which produces tons of garbage objects under heavy load. When testing the server on Linux under load we saw very frequent CPU bursts that crippled the throughput of the server. After profiling with the mono built-in profiler I confirmed that the reason for the high CPU usage was the GC collections that got more and more frequent and took more and more time. I then read a little bit and realized that the Mono GC is non-generational which might explain the long GC cycles (if it was generational it could have collected the garbage objects that were created during async socket operations in generation 0 and probably stop there, but it has to traverse the entire managed heap). So the combination of the non-generational GC and the unimplemented XXXAsync Socket API result in very poor performance of the Mono server. The maximum throughput of the server with Mono on Linux is about half of that on Windows using .NET. I attached the GC stats and profiling results of a 15 minute run. http://old.nabble.com/file/p27026906/profile_alloc.log profile_alloc.log http://old.nabble.com/file/p27026906/gc_stats.log gc_stats.log Can you help me out here? -- View this message in context: http://old.nabble.com/Fundamental-performance-problems-with-Mono-tp27026 906p27026906.html Sent from the Mono - Dev mailing list archive at Nabble.com. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list -- View this message in context: http://old.nabble.com/Fundamental-performance-problems-with-Mono-tp27026906p27075167.html Sent from the Mono - Dev mailing list archive at Nabble.com. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Fundamental performance problems with Mono
Robert Jordan wrote: The new GC won't help in this special case. why not? Even when using the Begin\End API, a generational GC could get rid of all the garbage this API creates during the fast gen 0 collections (at least this is what happens with Microsoft's GC on Windows). Robert Jordan wrote: So you're actually blaming a wrapper whose solely intention was to gain .NET 2.0 compatibility. My point is that two things prevent writing a high-performance server in Mono: 1. The current Begin\End async socket API (you call it 1.1) that creates lots of garbage 2. The current GC that cannot handle all this garbage Fix even one of the two (by properly implementing the 2.0 SP1 XXXAsync API to eliminate garbage, or implement a generational GC) and you fixed the problem. But better fix both... :) -- View this message in context: http://old.nabble.com/Fundamental-performance-problems-with-Mono-tp27026906p27078968.html Sent from the Mono - Dev mailing list archive at Nabble.com. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] Fundamental performance problems with Mono
Hello all, The bottom line of this message is that I don't see how can one write a high-performance socket server in Mono... Here is the story: I am writing a proxy server using .NET Socket API. This proxy does almost entirely I/O work - copying buffers from one socket to another. Now, Mono doesn't implement the newer http://msdn.microsoft.com/en-us/library/system.net.sockets.socketasynceventargs.aspx XXXAsync Socket API that was introduced in .NET 2.0 SP1 (or more accurately, implements it http://www.mail-archive.com/mono-l...@lists.ximian.com/msg28621.html perfunctorily ). So I was left to use the APM Socket API which produces tons of garbage objects under heavy load. When testing the server on Linux under load we saw very frequent CPU bursts that crippled the throughput of the server. After profiling with the mono built-in profiler I confirmed that the reason for the high CPU usage was the GC collections that got more and more frequent and took more and more time. I then read a little bit and realized that the Mono GC is non-generational which might explain the long GC cycles (if it was generational it could have collected the garbage objects that were created during async socket operations in generation 0 and probably stop there, but it has to traverse the entire managed heap). So the combination of the non-generational GC and the unimplemented XXXAsync Socket API result in very poor performance of the Mono server. The maximum throughput of the server with Mono on Linux is about half of that on Windows using .NET. I attached the GC stats and profiling results of a 15 minute run. http://old.nabble.com/file/p27026906/profile_alloc.log profile_alloc.log http://old.nabble.com/file/p27026906/gc_stats.log gc_stats.log Can you help me out here? -- View this message in context: http://old.nabble.com/Fundamental-performance-problems-with-Mono-tp27026906p27026906.html Sent from the Mono - Dev mailing list archive at Nabble.com. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-list] SocketAsyncEventArgs Implementation
Hello, The new Socket.XXXAsync APIs were added to the .NET FW to ...provide an alternative asynchronous pattern that can be used by specialized high-performance socket applications. This class was specifically designed for network server applications that require high performance. (MSDN) I was quite disappointed when I saw the mono implementation of this API - create a new thread for each XXXAsync call. This cannot produce a good performance. Did you test that this implementation is really more performant than the BeginXX/EndXX async pattern? Is there any plan to rewrite this code in the future? -- View this message in context: http://www.nabble.com/SocketAsyncEventArgs-Implementation-tp22081313p22081313.html Sent from the Mono - General mailing list archive at Nabble.com. ___ Mono-list maillist - Mono-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-list
