[Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic

2015-09-17 Thread Greg Young 
We are running into some interesting regressions with mono. We are
seeing many failures such as:

Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last
time we tested it didn't seem safe to move).

Native stacktrace:

/home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f]
/home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf]
/home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340]
/home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145]
[0x40e21ceb]


[03041,21,13:43:11.379] Global Unhandled Exception occurred.
Object reference not set to an instance of an object
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException:
Object reference not set to an instance of an object
  at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue
(System.Object o) [0x0] in :0
  at System.Threading.Thread.StartInternal () [0x0] in :0

note that this isn't a real nre

We do not see these regressions in Linux ip-172-31-37-60
3.13.0-46-generic #77-Ubuntu SMP

We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu SMP

Anyone have any ideas how to debug this further? These are not
deterministic issues.

Cheers,

Greg

-- 
Studying for the Turing test
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic

2015-09-17 Thread Taloth Saldono 
This Ubuntu bug discusses the issue for 3.13.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584

Please note that later kernel 4.0 versions exhibit similar issues, search
this mailinglist for specifics on that new regression. I gave up on
investigating further due to other obligations and the complexity of the
subject matter.
Op 17 sep. 2015 11:49 schreef "Greg Young" :

> 3.13.0-63-generic does not have the issue as well.
>
> I guess our mitigation will just be to know about the issue and
> suggest downgrade/upgrade. This may however be useful for others in
> the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to
> not be stable. We will also test mono4 now.
>
> Greg
>
> On Thu, Sep 17, 2015 at 11:38 AM, Greg Young 
> wrote:
> > We are running into some interesting regressions with mono. We are
> > seeing many failures such as:
> >
> > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last
> > time we tested it didn't seem safe to move).
> >
> > Native stacktrace:
> >
> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f]
> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf]
> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19]
> > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340]
> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145]
> > [0x40e21ceb]
> >
> >
> > [03041,21,13:43:11.379] Global Unhandled Exception occurred.
> > Object reference not set to an instance of an object
> > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException:
> > Object reference not set to an instance of an object
> >   at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue
> > (System.Object o) [0x0] in :0
> >   at System.Threading.Thread.StartInternal () [0x0] in  unknown>:0
> >
> > note that this isn't a real nre
> >
> > We do not see these regressions in Linux ip-172-31-37-60
> > 3.13.0-46-generic #77-Ubuntu SMP
> >
> > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu SMP
> >
> > Anyone have any ideas how to debug this further? These are not
> > deterministic issues.
> >
> > Cheers,
> >
> > Greg
> >
> > --
> > Studying for the Turing test
>
>
>
> --
> Studying for the Turing test
> ___
> Mono-devel-list mailing list
> Mono-devel-list@lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic

2015-09-17 Thread Greg Young 
I was just actually writing that this seems to be discussed here:
https://bugzilla.xamarin.com/show_bug.cgi?id=29692 :) Thanks for the
additional information.



On Thu, Sep 17, 2015 at 1:01 PM, Taloth Saldono  wrote:
> This Ubuntu bug discusses the issue for 3.13.
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584
>
> Please note that later kernel 4.0 versions exhibit similar issues, search
> this mailinglist for specifics on that new regression. I gave up on
> investigating further due to other obligations and the complexity of the
> subject matter.
>
> Op 17 sep. 2015 11:49 schreef "Greg Young" :
>>
>> 3.13.0-63-generic does not have the issue as well.
>>
>> I guess our mitigation will just be to know about the issue and
>> suggest downgrade/upgrade. This may however be useful for others in
>> the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to
>> not be stable. We will also test mono4 now.
>>
>> Greg
>>
>> On Thu, Sep 17, 2015 at 11:38 AM, Greg Young 
>> wrote:
>> > We are running into some interesting regressions with mono. We are
>> > seeing many failures such as:
>> >
>> > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last
>> > time we tested it didn't seem safe to move).
>> >
>> > Native stacktrace:
>> >
>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f]
>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf]
>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19]
>> > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340]
>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145]
>> > [0x40e21ceb]
>> >
>> >
>> > [03041,21,13:43:11.379] Global Unhandled Exception occurred.
>> > Object reference not set to an instance of an object
>> > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException:
>> > Object reference not set to an instance of an object
>> >   at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue
>> > (System.Object o) [0x0] in :0
>> >   at System.Threading.Thread.StartInternal () [0x0] in > > unknown>:0
>> >
>> > note that this isn't a real nre
>> >
>> > We do not see these regressions in Linux ip-172-31-37-60
>> > 3.13.0-46-generic #77-Ubuntu SMP
>> >
>> > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu
>> > SMP
>> >
>> > Anyone have any ideas how to debug this further? These are not
>> > deterministic issues.
>> >
>> > Cheers,
>> >
>> > Greg
>> >
>> > --
>> > Studying for the Turing test
>>
>>
>>
>> --
>> Studying for the Turing test
>> ___
>> Mono-devel-list mailing list
>> Mono-devel-list@lists.ximian.com
>> http://lists.ximian.com/mailman/listinfo/mono-devel-list



-- 
Studying for the Turing test
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic

2015-09-17 Thread Greg Young 
3.13.0-63-generic does not have the issue as well.

I guess our mitigation will just be to know about the issue and
suggest downgrade/upgrade. This may however be useful for others in
the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to
not be stable. We will also test mono4 now.

Greg

On Thu, Sep 17, 2015 at 11:38 AM, Greg Young  wrote:
> We are running into some interesting regressions with mono. We are
> seeing many failures such as:
>
> Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last
> time we tested it didn't seem safe to move).
>
> Native stacktrace:
>
> /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f]
> /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf]
> /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340]
> /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145]
> [0x40e21ceb]
>
>
> [03041,21,13:43:11.379] Global Unhandled Exception occurred.
> Object reference not set to an instance of an object
> [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException:
> Object reference not set to an instance of an object
>   at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue
> (System.Object o) [0x0] in :0
>   at System.Threading.Thread.StartInternal () [0x0] in  unknown>:0
>
> note that this isn't a real nre
>
> We do not see these regressions in Linux ip-172-31-37-60
> 3.13.0-46-generic #77-Ubuntu SMP
>
> We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu SMP
>
> Anyone have any ideas how to debug this further? These are not
> deterministic issues.
>
> Cheers,
>
> Greg
>
> --
> Studying for the Turing test



-- 
Studying for the Turing test
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list

Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic

2015-09-17 Thread Greg Young 
So to follow up on this.

The fix that was put in for this has since been reverted as mentioned
in the bug above.

https://github.com/torvalds/linux/commit/73459e2a1ada09a68c02cc5b73f3116fc8194b3d

So all 4 kernels will have this issue. I'm not really sure what needs
to be done on the mono side here to avoid this issue but it would seem
to affect any multi-threaded code running on multiple cpu linux
systems (which I would imagine would be a vast majority of mono
installations).

Greg

On Thu, Sep 17, 2015 at 1:05 PM, Greg Young  wrote:
> I was just actually writing that this seems to be discussed here:
> https://bugzilla.xamarin.com/show_bug.cgi?id=29692 :) Thanks for the
> additional information.
>
>
>
> On Thu, Sep 17, 2015 at 1:01 PM, Taloth Saldono  
> wrote:
>> This Ubuntu bug discusses the issue for 3.13.
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584
>>
>> Please note that later kernel 4.0 versions exhibit similar issues, search
>> this mailinglist for specifics on that new regression. I gave up on
>> investigating further due to other obligations and the complexity of the
>> subject matter.
>>
>> Op 17 sep. 2015 11:49 schreef "Greg Young" :
>>>
>>> 3.13.0-63-generic does not have the issue as well.
>>>
>>> I guess our mitigation will just be to know about the issue and
>>> suggest downgrade/upgrade. This may however be useful for others in
>>> the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to
>>> not be stable. We will also test mono4 now.
>>>
>>> Greg
>>>
>>> On Thu, Sep 17, 2015 at 11:38 AM, Greg Young 
>>> wrote:
>>> > We are running into some interesting regressions with mono. We are
>>> > seeing many failures such as:
>>> >
>>> > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last
>>> > time we tested it didn't seem safe to move).
>>> >
>>> > Native stacktrace:
>>> >
>>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f]
>>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf]
>>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19]
>>> > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340]
>>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145]
>>> > [0x40e21ceb]
>>> >
>>> >
>>> > [03041,21,13:43:11.379] Global Unhandled Exception occurred.
>>> > Object reference not set to an instance of an object
>>> > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException:
>>> > Object reference not set to an instance of an object
>>> >   at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue
>>> > (System.Object o) [0x0] in :0
>>> >   at System.Threading.Thread.StartInternal () [0x0] in >> > unknown>:0
>>> >
>>> > note that this isn't a real nre
>>> >
>>> > We do not see these regressions in Linux ip-172-31-37-60
>>> > 3.13.0-46-generic #77-Ubuntu SMP
>>> >
>>> > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu
>>> > SMP
>>> >
>>> > Anyone have any ideas how to debug this further? These are not
>>> > deterministic issues.
>>> >
>>> > Cheers,
>>> >
>>> > Greg
>>> >
>>> > --
>>> > Studying for the Turing test
>>>
>>>
>>>
>>> --
>>> Studying for the Turing test
>>> ___
>>> Mono-devel-list mailing list
>>> Mono-devel-list@lists.ximian.com
>>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>
>
>
> --
> Studying for the Turing test



-- 
Studying for the Turing test
___
Mono-devel-list mailing list
Mono-devel-list@lists.ximian.com
http://lists.ximian.com/mailman/listinfo/mono-devel-list