Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Hi, Mono had a race in its TCP code for a long time, which caused a lot of troubles for our project because under heavy load TCP async methods eventually stop receiving completion callbacks. But luckily a pull request fixing this problem with epoll/kqueue backends by my colleague was merged in to master, see https://github.com/mono/mono/pull/703 So you can try latest master and see if problem is gone, possibly this bug is the cause of your problem as symptoms are very similar. 4 серп. 2013 04:17, Alfred Hall ah...@ahall.org напис. ** Hi there. I'm running the NancyFX web framework in self hosting mode which is using HttpListener which basically means I deploy an executable and run it and it will start a webserver on localhost on a TCP port of choice. I then use nginx to proxy to it. I first ran my executable on my macbook pro and bombarded it with jmeter and it coped fine and no issues. I then deployed on my debian wheezy 64 bit linux box running on top of KVM using mono 3.2.0 and performed the same jmeter experiment. It locks up fairly quickly and wont take any new requests. I've tried using both sgen and boehm but they behave similarly although it seems to lock up faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it doesn't make any difference. Anyone had similar issues? I tried using self hosted ServiceStack which also uses HttpListener and had similar issues so I'm finding it unlikely that the bug is in NancyFX itself. I tried installing mono 2.10.8 to check if this is a regression, but getting exactly the same results. Any idea how I can debug this further or what could be going on. Cheers, Alf ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] [Mono-Dev] FileStream bug status
Hi, I've submitted a bug about FileStream behavior on combining reads and writes more than 2 month ago: https://bugzilla.xamarin.com/show_bug.cgi?id=11699 Are there any plans on fixing that bug? It seems like FileStream is pretty basic and very important stuff to have working correctly... Best regards, Andrii Nakryiko ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] FileStream write bug
Hi
We've had a problems with FileStream under Mono (different versions, both
2.10.x and 3.0.x). It seems like some internal positioning problem, because
instead of overwriting some parts of file, it just appends written data. It
is very critical that the amount of bytes written is larger than internal
buffer size. I've managed to create small isolated test case that shows
both what is necessary for bug to occur and easy workaround for that. But
hopefully this will be fixed as this is core functionality.
I've filed a bug report here:
https://bugzilla.xamarin.com/show_bug.cgi?id=11699
using System;
using System.IO;
using NUnit.Framework;
[TestFixture]
public class mono_filestream_bug
{
[Test]
public void show_time()
{
const int pos = 1;
const int bufferSize = 128;
var filename = Path.GetTempFileName();
File.WriteAllBytes(filename, new byte[pos + 1]); // init file
with zeros
var bytes = new byte[bufferSize + 1 /* THIS IS WHAT MAKES A BIG
DIFFERENCE */];
new Random().NextBytes(bytes);
using (var file = new FileStream(filename, FileMode.Open,
FileAccess.ReadWrite, FileShare.Read,
bufferSize,
FileOptions.SequentialScan))
{
file.Read(new byte[pos], 0, pos); // THIS READ IS CRITICAL,
WITHOUT IT EVERYTHING WORKS
Assert.AreEqual(pos, file.Position); // !!! here it says
position is correct, but writes at different position !!!
// file.Position = pos; // !!! this fixes test !!!
file.Write(bytes, 0, bytes.Length);
//Assert.AreEqual(pos + bytes.Length, file.Length); -- fails
}
using (var filestream = File.Open(filename, FileMode.Open,
FileAccess.Read))
{
var bb = new byte[bytes.Length];
filestream.Position = pos;
filestream.Read(bb, 0, bb.Length);
Assert.AreEqual(bytes, bb);
}
}
}
Best regards,
Andrii Nakryiko
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] TCP 3.0.1/2
We don't set MONO_DISABLE_AIO=1 and with patch have no problems with async sockets. We haven't seen any deadlocks either. -- Andrii Nakryiko 2013/1/22 Roope Kangas ro...@grandcrugames.com Thanks! Just asking so I get things right... Does this mean that after this patch you can run without MONO_DISABLE_AIO=1? Or do you run into problems with it if you dont have the patch applied? I hope I have time to try the patch with our codebase. -- Roope On Jan 21, 2013, at 11:25 PM, Andrii Nakryiko andrii.nakry...@gmail.com wrote: We use a version of mono with this patch https://github.com/ysw/mono-socket-problem/blob/master/Patches/cb_fix.patchapplied, otherwise we end up with the problem with TCP almost immediately in our test scenarios. -- Andrii Nakryiko 2013/1/21 Roope Kangas ro...@grandcrugames.com Hi! Whats the status of this issue? Is the best way on mono to implement a server by writing it against the *Async API and then settings MONO_DISABLE_AIO=1 Seems odd? -- Roope On Dec 7, 2012, at 6:44 PM, Rodrigo Kumpera kump...@gmail.com wrote: By the way, did you guys tried MONO_DISABLE_AIO=1 by any chance on your tests? It solves the issue for me. On Thu, Dec 6, 2012 at 3:46 AM, Greg Young gregoryyou...@gmail.comwrote: We have code that shows the issue https://github.com/ysw/mono-socket-problem/tree/master/SocketTest Greg On Fri, Nov 30, 2012 at 4:10 PM, Rodrigo Kumpera kump...@gmail.comwrote: Btw, I managed to find a linux bot that shows the io callback issue. The TP test works fine though. On Fri, Nov 30, 2012 at 12:29 AM, Greg Young gregoryyou...@gmail.comwrote: Thats already been provided (though its not hard to do). I will reupload a test that shows beahviour. -- Le doute n'est pas une condition agréable, mais la certitude est absurde. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] TCP 3.0.1/2
We use a version of mono with this patch https://github.com/ysw/mono-socket-problem/blob/master/Patches/cb_fix.patchapplied, otherwise we end up with the problem with TCP almost immediately in our test scenarios. -- Andrii Nakryiko 2013/1/21 Roope Kangas ro...@grandcrugames.com Hi! Whats the status of this issue? Is the best way on mono to implement a server by writing it against the *Async API and then settings MONO_DISABLE_AIO=1 Seems odd? -- Roope On Dec 7, 2012, at 6:44 PM, Rodrigo Kumpera kump...@gmail.com wrote: By the way, did you guys tried MONO_DISABLE_AIO=1 by any chance on your tests? It solves the issue for me. On Thu, Dec 6, 2012 at 3:46 AM, Greg Young gregoryyou...@gmail.comwrote: We have code that shows the issue https://github.com/ysw/mono-socket-problem/tree/master/SocketTest Greg On Fri, Nov 30, 2012 at 4:10 PM, Rodrigo Kumpera kump...@gmail.comwrote: Btw, I managed to find a linux bot that shows the io callback issue. The TP test works fine though. On Fri, Nov 30, 2012 at 12:29 AM, Greg Young gregoryyou...@gmail.comwrote: Thats already been provided (though its not hard to do). I will reupload a test that shows beahviour. -- Le doute n'est pas une condition agréable, mais la certitude est absurde. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] JIT compiler crashes application
Hi guys, I've created a project that reliably reproduces 2 bugs, the mentioned JIT crash, and just plain segmentation fault even without stack traces: https://github.com/anakryiko/mono-jit-crash-repro From readme: mono-jit-crash-repro Reproduction of problems with Mono JIT crashing. To run reproduce, compile the solution and run **run.sh** script which will start the binary in a loop until the program crashes. It usually doesn't take long for crash or segmentation fault to occur, though it can take up to few thousands runs. There are two classes that reproduces two bugs: * **ReproJitCrash.cs** reproduces JIT crash most of the times. Rarely it also causes segmentation fault without any stack traces. * **ReproSegFault.cs** reproduces segmenation fault with no stack trace (as mentioned previously). The only difference between those two cases is in using *struct* vs *class* with **using** statement (see LockReleaserStruct/LockReleaserClass). Also, it seems essential to have Monitor.Enter/Monitor.Exit, without them it's not reproducible (or at least I didn't have patience to wait for crash to happen). Hope that helps to identify and fix the problem. -- Andrii Nakryiko -- Andrii Nakryiko 2012/12/6 Rodrigo Kumpera kump...@gmail.com On Thu, Dec 6, 2012 at 12:38 PM, Andrii Nakryiko andrii.nakry...@gmail.com wrote: It's not about that particular method, we got earlier same kind of crashes on different methods, for instance, ProtoBuf.Serializers.ArrayDecorator:Write () If I remember correctly, I got this crash even for some method on ListT, though I can't locate that log quickly. If I get it, will post here. Also, this bug manifests on different versions of Mono ( 3.0). So the problem is not miscompilation. It might be memory corruption related. Can you try running it under valgrind or any other malloc debug tools? Does it always crash on that particular g_assert? If it does, it's a matter of augmenting it to produce a better crash message. This behavior is very randomly reproducible. To give you some context, we constantly run test scenarios where we start our TestClient in a loop. TestClient does some work and then exits. Then our shell script starts TestClient again. And sometimes TestClient crashes with error I described. What is interesting, crash mostly occurs not on first run of TestClient during this test scenario. Maybe that can help somehow. Then please try to reduce it to a test case that is small enough so we can work on. Can using LLVM back-end help to mitigate this? I have no reason to believe that the LLVM backend would help here. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
[Mono-dev] JIT compiler crashes application
Hi,
We sometimes get runtime crashes during the application run and it seems
that it happens inside JIT compiler. The crash is not happening constantly,
just once in a few runs.
We run under Mono 3.0.1 (no/301b6c6 Tue Dec 4 15:13:23 EET 2012) with SGen.
Here is the stack trace:
Thread 5 (Thread 0x7f26da5f4700 (LWP 13042)):
#0 0x7f26e2fec88d in waitpid () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0049a66b in mono_handle_native_sigsegv (signal=optimized
out, ctx=optimized out) at mini-exceptions.c:2289
#2 0x004ee5ff in mono_arch_handle_altstack_exception
(sigctx=0x7f26e2015ac0, fault_addr=optimized out, stack_ovf=0) at
exceptions-amd64.c:884
#3 0x0041c427 in mono_sigsegv_signal_handler (_dummy=11,
info=0x7f26e2015bf0, context=0x7f26e2015ac0) at mini.c:6066
#4 signal handler called
#5 emit_move_return_value (cfg=0x7f26a8006b10, ins=optimized out,
code=0x7f26a80257d0 \300W\002\250\177) at mini-amd64.c:3552
#6 0x004dd76d in mono_arch_output_basic_block (cfg=0x7f26a8006b10,
bb=0x7f26a8003678) at mini-amd64.c:4853
#7 0x0041d53a in mono_codegen (cfg=0x7f26a8006b10) at mini.c:3727
#8 0x0041e43c in mini_method_compile
(method=EventStore.Transport.Tcp.TcpConnection:EnqueueSend (),
opts=51472895, domain=0xc5cdf0, run_cctors=optimized out, compile_aot=0,
parts=0) at mini.c:5022
#9 0x0041fc42 in mono_jit_compile_method_inner
(jit_ex=0x7f26da5f36b8, opt=51472895, target_domain=0xc5cdf0,
method=EventStore.Transport.Tcp.TcpConnection:EnqueueSend ()) at
mini.c:5304
#10 mono_jit_compile_method_with_opt
(method=EventStore.Transport.Tcp.TcpConnection:EnqueueSend (),
opt=51472895, ex=0x7f26da5f36b8) at mini.c:5558
#11 0x0042062d in mono_jit_compile_method (method=optimized out)
at mini.c:5586
#12 0x0049c228 in common_call_trampoline (regs=0x7f26da5f3988,
code=0x40bd8718 H\203\304, incomplete sequence \303,
m=EventStore.Transport.Tcp.TcpConnection:EnqueueSend (), vt=0x0,
vtable_slot=optimized out, need_rgctx_tramp=0, tramp=optimized out) at
mini-trampolines.c:483
#13 0x40e48186 in ?? ()
#14 0x7f26a8002560 in ?? ()
#15 0x7f26d40025f0 in ?? ()
#16 0x7f26da5f3a70 in ?? ()
#17 0x0056c09a in mono_thread_interruption_checkpoint_request
(bypass_abort_protection=-631293392) at threads.c:4183
#18 0x40e48193 in ?? ()
#19 0x7f26e1c30cd8 in ?? ()
#20 0x in ?? ()
By looking at mini-amd64.c:3552 it seems that some internal assertion is
wrong:
3541: case OP_VCALL: 3542: case OP_VCALL_REG: 3543: case OP_VCALL_MEMBASE:
3544: case OP_VCALL2: 3545: case OP_VCALL2_REG: 3546: case
OP_VCALL2_MEMBASE: 3547: cinfo = get_call_info
(cfg-generic_sharing_context, cfg-mempool,
((MonoCallInst*)ins)-signature); 3548: if (cinfo-ret.storage ==
ArgValuetypeInReg) { 3549: MonoInst *loc = cfg-arch.vret_addr_loc; 3550:
3551: /* Load the destination address */ 3552: g_assert (loc-opcode ==
OP_REGOFFSET);
Any thought on what's wrong? Can we somehow work around this issue?..
-- Andrii Nakryiko
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] JIT compiler crashes application
It's not about that particular method, we got earlier same kind of crashes on different methods, for instance, ProtoBuf.Serializers.ArrayDecorator:Write () If I remember correctly, I got this crash even for some method on ListT, though I can't locate that log quickly. If I get it, will post here. Also, this bug manifests on different versions of Mono ( 3.0). This behavior is very randomly reproducible. To give you some context, we constantly run test scenarios where we start our TestClient in a loop. TestClient does some work and then exits. Then our shell script starts TestClient again. And sometimes TestClient crashes with error I described. What is interesting, crash mostly occurs not on first run of TestClient during this test scenario. Maybe that can help somehow. Can using LLVM back-end help to mitigate this? Thread 2 (Thread 0x7f2c4cfe7700 (LWP 23781)): #0 0x7f2ccf3e488d in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0049996b in mono_handle_native_sigsegv (signal=optimized out, ctx=optimized out) at mini-exceptions.c:2291 #2 0x004ecd7f in mono_arch_handle_altstack_exception (sigctx=0x7f2cbc143ac0, fault_addr=optimized out, stack_ovf=0) at exceptions-amd64.c:884 #3 0x0041c157 in mono_sigsegv_signal_handler (_dummy=11, info=0x7f2cbc143bf0, context=0x7f2cbc143ac0) at mini.c:6046 #4 signal handler called #5 emit_move_return_value (cfg=0x7f2ca800c9b0, ins=optimized out, code=0x7f2ca8021314 ) at mini-amd64.c:3577 #6 0x004dbeed in mono_arch_output_basic_block (cfg=0x7f2ca800c9b0, bb=0x7f2ca8003a28) at mini-amd64.c:4877 #7 0x0041d26a in mono_codegen (cfg=0x7f2ca800c9b0) at mini.c:3707 #8 0x0041e16c in mini_method_compile (method=ProtoBuf.Serializers.ArrayDecorator:Write (), opts=34695679, domain=0x10d8d60, run_cctors=optimized out, compile_aot=0, parts=0) at mini.c:5002 #9 0x0041f972 in mono_jit_compile_method_inner (jit_ex=0x7f2c4cfe64c8, opt=34695679, target_domain=0x10d8d60, method=ProtoBuf.Serializers.ArrayDecorator:Write ()) at mini.c:5284 #10 mono_jit_compile_method_with_opt (method=ProtoBuf.Serializers.ArrayDecorator:Write (), opt=34695679, ex=0x7f2c4cfe64c8) at mini.c:5538 #11 0x0042035d in mono_jit_compile_method (method=optimized out) at mini.c:5566 #12 0x0049b488 in common_call_trampoline (regs=0x7f2c4cfe6798, code=0x415e14c4 L\213,$L\213t$\bH\203?\030?, m=ProtoBuf.Serializers.ArrayDecorator:Write (), vt=vtable(ProtoBuf.Serializers.ArrayDecorator), vtable_slot=optimized out, need_rgctx_tramp=0, tramp=optimized out) at mini-trampolines.c:483 #13 0x41570e16 in ?? () #14 0x7f2cce0687e0 in ?? () #15 0x7f2cce068540 in ?? () #16 0x415e846c in ?? () #17 0x7f2ca800a660 in ?? () #18 0x7f2cce0687e0 in ?? () #19 0x005725c6 in mono_delegate_ctor_with_method (this=0x3, target=0x7f2c4cfe6840, addr=0x0, method=optimized out) at object.c:6092 #20 0x00527f19 in ves_icall_System_Delegate_CreateDelegate_internal (type=0x7f2ccfe55c08, target=0x7f2cce068540, info=optimized out, throwOnBindFailure=optimized out) at icall.c:5911 #21 0x40664333 in ?? () #22 0x7f2ca800af20 in ?? () #23 0x7f2cbf4fd0c8 in ?? () #24 0x7f2ca800af21 in ?? () #25 0x7f2c4cfe7660 in ?? () #26 0x in ?? () -- Andrii Nakryiko 2012/12/6 Rodrigo Kumpera kump...@gmail.com The method in question is EventStore.Transport.Tcp.TcpConnection:EnqueueSend, you can check if calling it directly triggers the bug. On Thu, Dec 6, 2012 at 11:20 AM, Greg Young gregoryyou...@gmail.comwrote: Its not reproducible even in our code base. Sometimes we get a sigfault sometimes not. It may happen 20 times in a row and not happen 20 times on an identical box. On Thu, Dec 6, 2012 at 6:13 PM, Rodrigo Kumpera kump...@gmail.comwrote: Please file a bug report with a reproducible test case. On Thu, Dec 6, 2012 at 10:44 AM, Andrii Nakryiko andrii.nakry...@gmail.com wrote: Hi, We sometimes get runtime crashes during the application run and it seems that it happens inside JIT compiler. The crash is not happening constantly, just once in a few runs. We run under Mono 3.0.1 (no/301b6c6 Tue Dec 4 15:13:23 EET 2012) with SGen. Here is the stack trace: Thread 5 (Thread 0x7f26da5f4700 (LWP 13042)): #0 0x7f26e2fec88d in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0049a66b in mono_handle_native_sigsegv (signal=optimized out, ctx=optimized out) at mini-exceptions.c:2289 #2 0x004ee5ff in mono_arch_handle_altstack_exception (sigctx=0x7f26e2015ac0, fault_addr=optimized out, stack_ovf=0) at exceptions-amd64.c:884 #3 0x0041c427 in mono_sigsegv_signal_handler (_dummy=11, info=0x7f26e2015bf0, context=0x7f26e2015ac0) at mini.c:6066 #4 signal handler called #5 emit_move_return_value (cfg=0x7f26a8006b10, ins=optimized out, code=0x7f26a80257d0 \300W\002\250\177) at mini-amd64.c:3552 #6 0x004dd76d
