If the C code overruns a stack allocated variable couldn’t it easily corrupt the saved registers ?
> On Sep 9, 2021, at 2:14 PM, Ian Lance Taylor <i...@golang.org> wrote: > > On Thu, Sep 9, 2021 at 10:29 AM Varun V <varun.1...@gmail.com> wrote: >> >> @Brian, We did not try that as the issue is happening with go 1.13.7 as well > > The "fatal: morestack on gsignal" error is more or less impossible. > Some things that can cause it to happen are > > 1) C code calls sigaltstack, but the alternate signal stack is too small. > 2) C code trashes the TLS slot that stores the value of G during a cgo call. > 3) C code fails to correctly preserve registers across a cgo call. > > None of these are at all likely. > > Ian > > >>> On Thu, 9 Sep, 2021, 10:48 pm Brian Candler, <b.cand...@pobox.com> wrote: >>> >>> Just a random thought, but have you tried 1.16 with >>> GODEBUG=asyncpreemptoff=1 ? The preemptive scheduling stuff was introduced >>> in 1.14 I believe. >>> >>> On Thursday, 9 September 2021 at 16:28:38 UTC+1 varun...@gmail.com wrote: >>>> >>>> @Kurtis, Thanks for the reply. >>>> >>>> Yes. We upgraded to 1.16.5 due to this issue and had to revert back to >>>> 1.13.7 (over earlier working version) as we are frequently seeing these >>>> errors in 1.16.5. >>>> >>>> I looked into all the related issues on golang forums. We are not on >>>> NetBSD (We are on CentOS7.8) >>>> >>>> We also suspect it is a bug in CGO code (specifically some memory >>>> violation) but finding the bug is like finding needle in a haystack. The >>>> SIGSEGV CGO signal handler did not catch anything at the time of this >>>> error. The error message by itself does not mention any thing debuggable. >>>> There is no stack or core dump. We tried working with GOTRACEBACK but that >>>> did not generate any core dump as well. >>>> >>>> Thanks, >>>> Varun >>>> >>>> >>>> On Thursday, September 9, 2021 at 8:07:00 PM UTC+5:30 Kurtis Rader wrote: >>>>> >>>>> Googling "go morestack on gsignal" turns up quite a few reports. Some of >>>>> which involved kernel bugs (e.g., >>>>> https://github.com/golang/go/issues/19652) but most seem to involve >>>>> SIGSEGV errors in non-kernel code; sometimes the Go runtime or stdlib >>>>> (e.g., https://github.com/golang/go/issues/35235) and sometimes user >>>>> code. Given that you're using CGO a likely explanation is a bug in your C >>>>> code. However, you say you're using Go 1.13.7 which is getting long in >>>>> the tooth. Are you sure you're seeing the same failure with Go 1.16.5? If >>>>> yes then I would bet the bug is in your C code. If no then it could be a >>>>> Go bug that has likely already been fixed. >>>>> >>>>> On Thu, Sep 9, 2021 at 4:15 AM varun...@gmail.com <varun...@gmail.com> >>>>> wrote: >>>>>> >>>>>> Minor update: >>>>>> >>>>>> a. The crash with "morestack on gsignal" happens to be independent of >>>>>> the kernel versions mentioned earlier >>>>>> b. We implemented a signal handler to catch SIGSEGV from CGO. That is >>>>>> not helping either >>>>>> c. From system audit logs, we can see the following message: >>>>>> >>>>>> ./audit/audit.log.2:22944:type=ANOM_ABEND >>>>>> msg=audit(1630376735.155:5486532): auid=4294967295 uid=996 gid=994 >>>>>> ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=26422 >>>>>> comm="indexer" reason="memory violation" sig=5 >>>>>> >>>>>> Sig=5 is SIGTRAP. At this point, we suspect there is a SIGSEGV in CGO >>>>>> layer and that is being returned as SIGTRAP by runtime. >>>>>> >>>>>> I am still clueless as to why a process can crash with "morestack on >>>>>> gsignal". Any pointers to debug this further would be of great help. >>>>>> >>>>>> Thanks, >>>>>> Varun >>>>>> On Wednesday, September 1, 2021 at 2:44:52 PM UTC+5:30 >>>>>> varun...@gmail.com wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> Recently we have been seeing multiple occurrences of "fatal: morestack >>>>>>> on gsignal" error in our system tests. The issue happens with golang >>>>>>> 1.13.7 and golang 1.16.5 runtimes. The process restarts after this >>>>>>> error and no stack trace is dumped. Our process uses CGO calls, I/O ops >>>>>>> etc. >>>>>>> >>>>>>> So far, the issue seem to happen only on CentOS7 with linux kernel >>>>>>> version: 3.10.0-1127.19.1.el7.x86_64. The issue is not seen on: >>>>>>> 3.10.0-693.5.2.el7.x86_64 >>>>>>> >>>>>>> At this point, we have no clue as to why this is happening and no stack >>>>>>> trace is making it difficult to debug. Any pointers on debugging the >>>>>>> issue would be of great help. >>>>>>> >>>>>>> Thanks, >>>>>>> Varun >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to golang-nuts...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com. >>>>> >>>>> >>>>> >>>>> -- >>>>> Kurtis Rader >>>>> Caretaker of the exceptional canines Junior and Hank >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "golang-nuts" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/golang-nuts/msVmzxWjIOI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> golang-nuts+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/4e6c8f7d-86be-4ecf-b4f5-19554f3d1379n%40googlegroups.com. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/CAMzZVzBrwcw%2BMJOjvXgg_YhFA0ZgYD%3DiLhYJyYHCxzXFKYD%2BeQ%40mail.gmail.com. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX%3DDSaZBZVa69xM4RNxVhfV2KianmVOfmXm2Bs_3bUcxA%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/BBBF032D-99C5-48CD-BC6F-FD86C5B579E2%40ix.netcom.com.
