Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I also experienced problems with HttpListener before (on Mono 2.6 and 2.10). Forcing mono to print stacktraces with SIGQUIT signal revealed the fact that HttpListener hungs somewhere in System.Net.HttpConnection.ProcessInput. But in my case it works pretty fine behind nginx. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
I've noticed the same issue with self-hosted ASP.NET WebAPI, which is probably the same underlying cause. On Sun, Aug 4, 2013 at 11:07 AM, Alfred Hall ah...@ahall.org wrote: ** Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Cheers, Alf ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi, Mono had a race in its TCP code for a long time, which caused a lot of troubles for our project because under heavy load TCP async methods eventually stop receiving completion callbacks. But luckily a pull request fixing this problem with epoll/kqueue backends by my colleague was merged in to master, see https://github.com/mono/mono/pull/703 So you can try latest master and see if problem is gone, possibly this bug is the cause of your problem as symptoms are very similar. 4 серп. 2013 04:17, Alfred Hall ah...@ahall.org напис. ** Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Cheers, Alf ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
On 04/08/13 03:07, Alfred Hall wrote: Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Hey Alfred. Could you try mono master (3.3) instead of 3.2? I mention this because this pull request [1] has been merged recently, which could help in this situation (and wouldn't make much difference in Mac since the problem in this platform is worked-around by avoiding kqueue [2]). [1] https://github.com/mono/mono/pull/703 [2] https://github.com/mono/mono/blob/master/configure.in#L1823 ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
I meant to say I tried master too: root@mulder:~# /opt/ahall-mono/bin/mono -V Mono JIT compiler version 3.3.0 (master/2354865 Sun Aug 4 00:42:51 BST 2013) Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com TLS: __thread SIGSEGV: altstack Notifications: epoll Architecture: amd64 Disabled: none Misc: softdebug LLVM: supported, not enabled. GC: sgen That should have the fix in from https://github.com/ysw, but setting MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass the epoll backend. My Nancy service is literally just returning a very simple JSON: public class HelloWorldModule : NancyModule { public HelloWorldModule() { Get[/] = parameters = { return Response.AsJson(new HomeResponse { Message = Test }); }; } } In JMeter I'm using 100 threads and loop count of 100 and it locks up after like 15 seconds even over the network. Very odd. -Original message- From:quot;Andrés G. Aragonesesquot; kno...@gmail.com mailto:kno...@gmail.com Sent: Sunday 4th August 2013 10:03 To: mono-devel-list@lists.ximian.com mailto:mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux On 04/08/13 03:07, Alfred Hall wrote: Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Hey Alfred. Could you try mono master (3.3) instead of 3.2? I mention this because this pull request [1] has been merged recently, which could help in this situation (and wouldn't make much difference in Mac since the problem in this platform is worked-around by avoiding kqueue [2]). [1] https://github.com/mono/mono/pull/703 [2] https://github.com/mono/mono/blob/master/configure.in#L1823 ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com mailto:Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi Alfred Are you sure its not a ulimit issue? I have had this issue before with the nofile limit as each tcp session is a file under /proc and when you have 1024 open (or close wait) sessions the OS does not allow you to open more files and the code seems to hang for a bit until a session is completely gone. Eg. Timed out and you can create files again... -- Esben (on mobile) - Reply message - Fra: Alfred Hall ah...@ahall.org Til: Andrés G. Aragoneses kno...@gmail.com, mono-devel-list@lists.ximian.com, mono-devel-list@lists.ximian.com Emne: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux Dato: søn., aug. 4, 2013 13:17 RE: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux I meant to say I tried master too: root@mulder:~# /opt/ahall-mono/bin/mono -VMono JIT compiler version 3.3.0 (master/2354865 Sun Aug 4 00:42:51 BST 2013)Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com TLS: __thread SIGSEGV: altstack Notifications: epoll Architecture: amd64 Disabled: none Misc: softdebug LLVM: supported, not enabled. GC: sgen That should have the fix in from https://github.com/ysw, but setting MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass the epoll backend.My Nancy service is literally just returning a very simple JSON:public class HelloWorldModule : NancyModule { public HelloWorldModule() { Get[/] = parameters = { return Response.AsJson(new HomeResponse { Message = Test }); }; } }In JMeter I'm using 100 threads and loop count of 100 and it locks up after like 15 seconds even over the network. Very odd. -Original message- From:quot;Andrés G. Aragonesesquot; kno...@gmail.com Sent: Sunday 4th August 2013 10:03 To: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux On 04/08/13 03:07, Alfred Hall wrote: Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Hey Alfred. Could you try mono master (3.3) instead of 3.2? I mention this because this pull request [1] has been merged recently, which could help in this situation (and wouldn't make much difference in Mac since the problem in this platform is worked-around by avoiding kqueue [2]). [1] https://github.com/mono/mono/pull/703 [2] https://github.com/mono/mono/blob/master/configure.in#L1823 ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi Esben. Certainly a good suggestion. I've tried increasing it: nitoback@mulder:~$ ulimit -Sn 20 nitoback@mulder:~$ ulimit -Hn 20 Doesn't really have any effect. Here's what happens when I start hammering it - regarding connections - The self host is running on port 9591: root@mulder:~# lsof -n | grep -i 9591 | wc -l 93 root@mulder:~# lsof -n | grep -i 9591 | wc -l 642 root@mulder:~# lsof -n | grep -i 9591 | wc -l 573 root@mulder:~# lsof -n | grep -i 9591 | wc -l 677 root@mulder:~# lsof -n | grep -i 9591 | wc -l 1270 root@mulder:~# lsof -n | grep -i 9591 | wc -l 1270 The connection count goes up to 1270 and at that point it hangs, this is slightly different each time I run this. After having stopped the tests and 10 minutes after it: root@mulder:~# lsof -n | grep -i 9591 | wc -l 1170 Here is an example of the connections that are in the CLOSE_WAIT state: mono 2788 2840 nitoback 71u IPv4 65081 0t0 TCP 127.0.0.1:9591-127.0.0.1:40372 (CLOSE_WAIT) mono 2788 2840 nitoback 72u IPv4 65082 0t0 TCP 127.0.0.1:9591-127.0.0.1:40373 (CLOSE_WAIT) They seem to be stuck in CLOSE_WAIT which I guess means that there is unread data on the socket. When it gets into this situation its not recoverable, just hangs and keeps the connections in CLOSE_WAIT state. -Original message- From: hy...@hyber.dk hy...@hyber.dk Sent: Sunday 4th August 2013 14:39 To: Alfred Hall ah...@ahall.org Cc: mono-devel-list@lists.ximian.com Subject: SV: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux Hi Alfred Are you sure its not a ulimit issue? I have had this issue before with the nofile limit as each tcp session is a file under /proc and when you have 1024 open (or close wait) sessions the OS does not allow you to open more files and the code seems to hang for a bit until a session is completely gone. Eg. Timed out and you can create files again... -- Esben (on mobile) - Reply message - Fra: Alfred Hall ah...@ahall.org Til: Andrés G. Aragoneses kno...@gmail.com, mono-devel-list@lists.ximian.com, mono-devel-list@lists.ximian.com Emne: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux Dato: søn., aug. 4, 2013 13:17 I meant to say I tried master too: root@mulder:~# /opt/ahall-mono/bin/mono -V Mono JIT compiler version 3.3.0 (master/2354865 Sun Aug 4 00:42:51 BST 2013) Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com TLS: __thread SIGSEGV: altstack Notifications: epoll Architecture: amd64 Disabled: none Misc: softdebug LLVM: supported, not enabled. GC: sgen That should have the fix in from https://github.com/ysw, but setting MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass the epoll backend. My Nancy service is literally just returning a very simple JSON: public class HelloWorldModule : NancyModule { public HelloWorldModule() { Get[/] = parameters = { return Response.AsJson(new HomeResponse { Message = Test }); }; } } In JMeter I'm using 100 threads and loop count of 100 and it locks up after like 15 seconds even over the network. Very odd. -Original message- From:quot;Andrés G. Aragonesesquot; kno...@gmail.com mailto:kno...@gmail.com Sent: Sunday 4th August 2013 10:03 To: mono-devel-list@lists.ximian.com mailto:mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux On 04/08/13 03:07, Alfred Hall wrote: Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Hey Alfred. Could you try mono master (3.3) instead of 3.2? I mention this because this pull request [1] has been merged recently, which could help in this situation
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT {PID}) when it stops processing requests. It will force mono to write thread stack traces to stdout. Grab them and post here, I suspect that the issue is simular to one experienced by me. 2013/8/4 Nikita Tsukanov kek...@gmail.com Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT {PID}) when it stops processing requests. It will force mono to write thread stack traces to stdout. Grab them and post here, I suspect that the issue is simular to one experienced by me. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi Nikita. Full thread dump: threadpool thread tid=0x0x7fc4ad29d700 this=0x0x7fc4ad584c70 thread handle 0x80f state : not waiting owns () IO Threadpool worker tid=0x0x7fc4ad25c700 this=0x0x7fc4ad584dd0 thread handle 0x810 state : interrupted state owns () IO Threadpool worker tid=0x0x7fc4a7567700 this=0x0x7fc4a741d350 thread handle 0x845 state : interrupted state owns () Threadpool worker tid=0x0x7fc4ac39a700 this=0x0x7fc4a6192270 thread handle 0x837 state : interrupted state owns () at unknown 0x at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) 0x at (wrapper alloc) object.AllocVector (intptr,intptr) 0x at System.Net.HttpConnection.BeginReadRequest () 0x0003a at System.Net.EndPointListener.OnAccept (object,System.EventArgs) 0x00357 at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted (System.Net.Sockets.SocketAsyncEventArgs) 0x0002e at System.Net.Sockets.SocketAsyncEventArgs.AcceptCallback (System.IAsyncResult) 0x00336 at System.Net.Sockets.SocketAsyncEventArgs.DispatcherCB (System.IAsyncResult) 0x0010f at (wrapper runtime-invoke) Module.runtime_invoke_void__this___object (object,intptr,intptr,intptr) 0x I then tried the same again and only got this in my trace: Full thread dump: threadpool thread tid=0x0x7f31b8ac5700 this=0x0x7f31b8da4c70 thread handle 0x80e state : not waiting owns () IO Threadpool worker tid=0x0x7f31b8a84700 this=0x0x7f31b8da4dd0 thread handle 0x80f state : interrupted state owns () Not sure why I'm not getting any dump here. Any more debugging I can do on there? What seems to happen is its coping well initially with the requests and then in all of a sudden it stops accepting connections and existing connections dont die off. Cheers, Alf -Original message- From: Nikita Tsukanov kek...@gmail.com Sent: Sunday 4th August 2013 16:13 To: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT {PID}) when it stops processing requests. It will force mono to write thread stack traces to stdout. Grab them and post here, I suspect that the issue is simular to one experienced by me. 2013/8/4 Nikita Tsukanov kek...@gmail.com mailto:kek...@gmail.com Alfred, please, try to send SIGQUIT to mono (i. e. kill -SIGQUIT {PID}) when it stops processing requests. It will force mono to write thread stack traces to stdout. Grab them and post here, I suspect that the issue is simular to one experienced by me. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi, two notes: 1) That should have the fix in from https://github.com/ysw, but setting MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass the epoll backend. The problem we attempted to fix with async sockets is not only with epoll backend. i could reproduce it with plain poll backend as well. Moreover the patch we submitted only addresses it for epoll/kqueue and leaves poll backend unpatched. 2) From what I understand it is very unlikely it is related to the problem described here. The problem we fixed can only be observed if you have parallel read and write operations on the same socket which is unlikely to happen in RPC style protocols like HTTP, unless you do request pipelining from the client. However, if it is the same problem, MONO_DISABLE_AIO won't help as poll backend is not better than epoll/kqueue in this case. -yuriy On Sun, Aug 4, 2013 at 2:17 PM, Alfred Hall ah...@ahall.org wrote: ** I meant to say I tried master too: root@mulder:~# /opt/ahall-mono/bin/mono -V Mono JIT compiler version 3.3.0 (master/2354865 Sun Aug 4 00:42:51 BST 2013) Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com TLS: __thread SIGSEGV: altstack Notifications: epoll Architecture: amd64 Disabled: none Misc: softdebug LLVM: supported, not enabled. GC:sgen That should have the fix in from https://github.com/ysw, but setting MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass the epoll backend. My Nancy service is literally just returning a very simple JSON: public class HelloWorldModule : NancyModule { public HelloWorldModule() { Get[/] = parameters = { return Response.AsJson(new HomeResponse { Message = Test }); }; } } In JMeter I'm using 100 threads and loop count of 100 and it locks up after like 15 seconds even over the network. Very odd. -Original message- From:quot;Andrés G. Aragonesesquot; kno...@gmail.com Sent: Sunday 4th August 2013 10:03 To: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux On 04/08/13 03:07, Alfred Hall wrote: Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Hey Alfred. Could you try mono master (3.3) instead of 3.2? I mention this because this pull request [1] has been merged recently, which could help in this situation (and wouldn't make much difference in Mac since the problem in this platform is worked-around by avoiding kqueue [2]). [1] https://github.com/mono/mono/pull/703 [2] https://github.com/mono/mono/blob/master/configure.in#L1823 ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list -- Yuriy Solodkyy (y.solod...@gmail.com) ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list