Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic
3.13.0-63-generic does not have the issue as well. I guess our mitigation will just be to know about the issue and suggest downgrade/upgrade. This may however be useful for others in the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to not be stable. We will also test mono4 now. Greg On Thu, Sep 17, 2015 at 11:38 AM, Greg Young wrote: > We are running into some interesting regressions with mono. We are > seeing many failures such as: > > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last > time we tested it didn't seem safe to move). > > Native stacktrace: > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f] > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf] > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19] > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340] > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145] > [0x40e21ceb] > > > [03041,21,13:43:11.379] Global Unhandled Exception occurred. > Object reference not set to an instance of an object > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: > Object reference not set to an instance of an object > at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue > (System.Object o) [0x0] in :0 > at System.Threading.Thread.StartInternal () [0x0] in unknown>:0 > > note that this isn't a real nre > > We do not see these regressions in Linux ip-172-31-37-60 > 3.13.0-46-generic #77-Ubuntu SMP > > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu SMP > > Anyone have any ideas how to debug this further? These are not > deterministic issues. > > Cheers, > > Greg > > -- > Studying for the Turing test -- Studying for the Turing test ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic
This Ubuntu bug discusses the issue for 3.13. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584 Please note that later kernel 4.0 versions exhibit similar issues, search this mailinglist for specifics on that new regression. I gave up on investigating further due to other obligations and the complexity of the subject matter. Op 17 sep. 2015 11:49 schreef "Greg Young" : > 3.13.0-63-generic does not have the issue as well. > > I guess our mitigation will just be to know about the issue and > suggest downgrade/upgrade. This may however be useful for others in > the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to > not be stable. We will also test mono4 now. > > Greg > > On Thu, Sep 17, 2015 at 11:38 AM, Greg Young > wrote: > > We are running into some interesting regressions with mono. We are > > seeing many failures such as: > > > > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last > > time we tested it didn't seem safe to move). > > > > Native stacktrace: > > > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f] > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf] > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340] > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145] > > [0x40e21ceb] > > > > > > [03041,21,13:43:11.379] Global Unhandled Exception occurred. > > Object reference not set to an instance of an object > > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: > > Object reference not set to an instance of an object > > at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue > > (System.Object o) [0x0] in :0 > > at System.Threading.Thread.StartInternal () [0x0] in unknown>:0 > > > > note that this isn't a real nre > > > > We do not see these regressions in Linux ip-172-31-37-60 > > 3.13.0-46-generic #77-Ubuntu SMP > > > > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu SMP > > > > Anyone have any ideas how to debug this further? These are not > > deterministic issues. > > > > Cheers, > > > > Greg > > > > -- > > Studying for the Turing test > > > > -- > Studying for the Turing test > ___ > Mono-devel-list mailing list > Mono-devel-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-devel-list > ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic
I was just actually writing that this seems to be discussed here: https://bugzilla.xamarin.com/show_bug.cgi?id=29692 :) Thanks for the additional information. On Thu, Sep 17, 2015 at 1:01 PM, Taloth Saldono wrote: > This Ubuntu bug discusses the issue for 3.13. > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584 > > Please note that later kernel 4.0 versions exhibit similar issues, search > this mailinglist for specifics on that new regression. I gave up on > investigating further due to other obligations and the complexity of the > subject matter. > > Op 17 sep. 2015 11:49 schreef "Greg Young" : >> >> 3.13.0-63-generic does not have the issue as well. >> >> I guess our mitigation will just be to know about the issue and >> suggest downgrade/upgrade. This may however be useful for others in >> the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to >> not be stable. We will also test mono4 now. >> >> Greg >> >> On Thu, Sep 17, 2015 at 11:38 AM, Greg Young >> wrote: >> > We are running into some interesting regressions with mono. We are >> > seeing many failures such as: >> > >> > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last >> > time we tested it didn't seem safe to move). >> > >> > Native stacktrace: >> > >> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f] >> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf] >> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19] >> > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340] >> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145] >> > [0x40e21ceb] >> > >> > >> > [03041,21,13:43:11.379] Global Unhandled Exception occurred. >> > Object reference not set to an instance of an object >> > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: >> > Object reference not set to an instance of an object >> > at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue >> > (System.Object o) [0x0] in :0 >> > at System.Threading.Thread.StartInternal () [0x0] in > > unknown>:0 >> > >> > note that this isn't a real nre >> > >> > We do not see these regressions in Linux ip-172-31-37-60 >> > 3.13.0-46-generic #77-Ubuntu SMP >> > >> > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu >> > SMP >> > >> > Anyone have any ideas how to debug this further? These are not >> > deterministic issues. >> > >> > Cheers, >> > >> > Greg >> > >> > -- >> > Studying for the Turing test >> >> >> >> -- >> Studying for the Turing test >> ___ >> Mono-devel-list mailing list >> Mono-devel-list@lists.ximian.com >> http://lists.ximian.com/mailman/listinfo/mono-devel-list -- Studying for the Turing test ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic
So to follow up on this. The fix that was put in for this has since been reverted as mentioned in the bug above. https://github.com/torvalds/linux/commit/73459e2a1ada09a68c02cc5b73f3116fc8194b3d So all 4 kernels will have this issue. I'm not really sure what needs to be done on the mono side here to avoid this issue but it would seem to affect any multi-threaded code running on multiple cpu linux systems (which I would imagine would be a vast majority of mono installations). Greg On Thu, Sep 17, 2015 at 1:05 PM, Greg Young wrote: > I was just actually writing that this seems to be discussed here: > https://bugzilla.xamarin.com/show_bug.cgi?id=29692 :) Thanks for the > additional information. > > > > On Thu, Sep 17, 2015 at 1:01 PM, Taloth Saldono > wrote: >> This Ubuntu bug discusses the issue for 3.13. >> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584 >> >> Please note that later kernel 4.0 versions exhibit similar issues, search >> this mailinglist for specifics on that new regression. I gave up on >> investigating further due to other obligations and the complexity of the >> subject matter. >> >> Op 17 sep. 2015 11:49 schreef "Greg Young" : >>> >>> 3.13.0-63-generic does not have the issue as well. >>> >>> I guess our mitigation will just be to know about the issue and >>> suggest downgrade/upgrade. This may however be useful for others in >>> the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to >>> not be stable. We will also test mono4 now. >>> >>> Greg >>> >>> On Thu, Sep 17, 2015 at 11:38 AM, Greg Young >>> wrote: >>> > We are running into some interesting regressions with mono. We are >>> > seeing many failures such as: >>> > >>> > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last >>> > time we tested it didn't seem safe to move). >>> > >>> > Native stacktrace: >>> > >>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f] >>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf] >>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19] >>> > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340] >>> > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145] >>> > [0x40e21ceb] >>> > >>> > >>> > [03041,21,13:43:11.379] Global Unhandled Exception occurred. >>> > Object reference not set to an instance of an object >>> > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: >>> > Object reference not set to an instance of an object >>> > at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue >>> > (System.Object o) [0x0] in :0 >>> > at System.Threading.Thread.StartInternal () [0x0] in >> > unknown>:0 >>> > >>> > note that this isn't a real nre >>> > >>> > We do not see these regressions in Linux ip-172-31-37-60 >>> > 3.13.0-46-generic #77-Ubuntu SMP >>> > >>> > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu >>> > SMP >>> > >>> > Anyone have any ideas how to debug this further? These are not >>> > deterministic issues. >>> > >>> > Cheers, >>> > >>> > Greg >>> > >>> > -- >>> > Studying for the Turing test >>> >>> >>> >>> -- >>> Studying for the Turing test >>> ___ >>> Mono-devel-list mailing list >>> Mono-devel-list@lists.ximian.com >>> http://lists.ximian.com/mailman/listinfo/mono-devel-list > > > > -- > Studying for the Turing test -- Studying for the Turing test ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic
I am surprised to see no one else post on this. I would think a massive regression that prevents mono from running multi-threaded code on a multi-cpu system (including in the future on any v4 kernel) might be an issue that people would be concerned about?! On Thu, Sep 17, 2015 at 4:03 PM, Greg Young wrote: > So to follow up on this. > > The fix that was put in for this has since been reverted as mentioned > in the bug above. > > https://github.com/torvalds/linux/commit/73459e2a1ada09a68c02cc5b73f3116fc8194b3d > > So all 4 kernels will have this issue. I'm not really sure what needs > to be done on the mono side here to avoid this issue but it would seem > to affect any multi-threaded code running on multiple cpu linux > systems (which I would imagine would be a vast majority of mono > installations). > > Greg > > On Thu, Sep 17, 2015 at 1:05 PM, Greg Young wrote: >> I was just actually writing that this seems to be discussed here: >> https://bugzilla.xamarin.com/show_bug.cgi?id=29692 :) Thanks for the >> additional information. >> >> >> >> On Thu, Sep 17, 2015 at 1:01 PM, Taloth Saldono >> wrote: >>> This Ubuntu bug discusses the issue for 3.13. >>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584 >>> >>> Please note that later kernel 4.0 versions exhibit similar issues, search >>> this mailinglist for specifics on that new regression. I gave up on >>> investigating further due to other obligations and the complexity of the >>> subject matter. >>> >>> Op 17 sep. 2015 11:49 schreef "Greg Young" : 3.13.0-63-generic does not have the issue as well. I guess our mitigation will just be to know about the issue and suggest downgrade/upgrade. This may however be useful for others in the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to not be stable. We will also test mono4 now. Greg On Thu, Sep 17, 2015 at 11:38 AM, Greg Young wrote: > We are running into some interesting regressions with mono. We are > seeing many failures such as: > > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last > time we tested it didn't seem safe to move). > > Native stacktrace: > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f] > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf] > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19] > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340] > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145] > [0x40e21ceb] > > > [03041,21,13:43:11.379] Global Unhandled Exception occurred. > Object reference not set to an instance of an object > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: > Object reference not set to an instance of an object > at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue > (System.Object o) [0x0] in :0 > at System.Threading.Thread.StartInternal () [0x0] in >>> > unknown>:0 > > note that this isn't a real nre > > We do not see these regressions in Linux ip-172-31-37-60 > 3.13.0-46-generic #77-Ubuntu SMP > > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu > SMP > > Anyone have any ideas how to debug this further? These are not > deterministic issues. > > Cheers, > > Greg > > -- > Studying for the Turing test -- Studying for the Turing test ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list >> >> >> >> -- >> Studying for the Turing test > > > > -- > Studying for the Turing test -- Studying for the Turing test ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic
So we have just seen an issue that looks oddly familiar to this in kernel 4.4.5 On Mon, Sep 21, 2015 at 10:03 PM, Greg Young wrote: > I am surprised to see no one else post on this. > > I would think a massive regression that prevents mono from running > multi-threaded code on a multi-cpu system (including in the future on > any v4 kernel) might be an issue that people would be concerned > about?! > > > > On Thu, Sep 17, 2015 at 4:03 PM, Greg Young wrote: >> So to follow up on this. >> >> The fix that was put in for this has since been reverted as mentioned >> in the bug above. >> >> https://github.com/torvalds/linux/commit/73459e2a1ada09a68c02cc5b73f3116fc8194b3d >> >> So all 4 kernels will have this issue. I'm not really sure what needs >> to be done on the mono side here to avoid this issue but it would seem >> to affect any multi-threaded code running on multiple cpu linux >> systems (which I would imagine would be a vast majority of mono >> installations). >> >> Greg >> >> On Thu, Sep 17, 2015 at 1:05 PM, Greg Young wrote: >>> I was just actually writing that this seems to be discussed here: >>> https://bugzilla.xamarin.com/show_bug.cgi?id=29692 :) Thanks for the >>> additional information. >>> >>> >>> >>> On Thu, Sep 17, 2015 at 1:01 PM, Taloth Saldono >>> wrote: This Ubuntu bug discusses the issue for 3.13. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584 Please note that later kernel 4.0 versions exhibit similar issues, search this mailinglist for specifics on that new regression. I gave up on investigating further due to other obligations and the complexity of the subject matter. Op 17 sep. 2015 11:49 schreef "Greg Young" : > > 3.13.0-63-generic does not have the issue as well. > > I guess our mitigation will just be to know about the issue and > suggest downgrade/upgrade. This may however be useful for others in > the future. Mono 3.12.1 on 3.13.0-46-generic #77-Ubuntu SMP appears to > not be stable. We will also test mono4 now. > > Greg > > On Thu, Sep 17, 2015 at 11:38 AM, Greg Young > wrote: > > We are running into some interesting regressions with mono. We are > > seeing many failures such as: > > > > Mono v3.12.1 (we have considered moving up to 4.1 or 4.2 but the last > > time we tested it didn't seem safe to move). > > > > Native stacktrace: > > > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x609a4f] > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x5bdecf] > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x453b19] > > /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd9709b7340] > > /home/ubuntu/EventStore-OSS-Ubuntu-v3.2.1/eventstored() [0x4e6145] > > [0x40e21ceb] > > > > > > [03041,21,13:43:11.379] Global Unhandled Exception occurred. > > Object reference not set to an instance of an object > > [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: > > Object reference not set to an instance of an object > > at EventStore.Core.Bus.QueuedHandlerAutoReset.ReadFromQueue > > (System.Object o) [0x0] in :0 > > at System.Threading.Thread.StartInternal () [0x0] in > unknown>:0 > > > > note that this isn't a real nre > > > > We do not see these regressions in Linux ip-172-31-37-60 > > 3.13.0-46-generic #77-Ubuntu SMP > > > > We do see them in Linux ip-172-31-40-115 3.13.0-48-generic #80-Ubuntu > > SMP > > > > Anyone have any ideas how to debug this further? These are not > > deterministic issues. > > > > Cheers, > > > > Greg > > > > -- > > Studying for the Turing test > > > > -- > Studying for the Turing test > ___ > Mono-devel-list mailing list > Mono-devel-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-devel-list >>> >>> >>> >>> -- >>> Studying for the Turing test >> >> >> >> -- >> Studying for the Turing test > > > > -- > Studying for the Turing test -- Studying for the Turing test ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Regressions on 3.13.0-46-generic -> 3.13.0-48-generic (Mono-devel-list Digest, Vol 125, Issue 17)
On 21/09/15 22:03, Greg Young wrote: > I am surprised to see no one else post on this. > > I would think a massive regression that prevents mono from running > multi-threaded code on a multi-cpu system (including in the future on > any v4 kernel) might be an issue that people would be concerned > about?! The original kernel change was reverted precisely because it caused problems with apps like Mono. Which is why it's not in current 3.13 Trusty kernels. Have you confirmed that the new version of the change, in 4.1+, reintroduces the issue? Because I run a mainline 4.2 on my laptop, and haven't had it happen. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
