On Fri, Aug 23, 2019 at 2:12 PM Robert Engels <reng...@ix.netcom.com> wrote:
> If you write an long running non pre-emptable all go routines are > effected. > > Yeah, I've come across the case before where GC would begin a stop the world but get stuck waiting for a non-preemtable goroutine. I don't think that's happening here because when the program gets "unstuck" as the error gets printed, there is no gctrace output that indicated a GC happened. I can make sure by adding a dummy goroutine that just logs output every second. That would make it more obvious that more than one goroutine is being affected. Thanks for the suggestion. Are you suggesting that there might be enough unpreemtable goroutines to fill all the P's? I do have several cgo goroutines sitting in syscalls, but my understanding was that the scheduler would "preempt" that by moving the M off the P and creating a new M. Thanks > On Aug 23, 2019, at 4:03 PM, Michael Andersen <mich...@steelcode.com> > wrote: > > > > On Fri, Aug 23, 2019 at 1:23 PM Ronny Bangsund <ronny.bangs...@gmail.com> > wrote: > >> On Friday, August 23, 2019 at 5:58:44 PM UTC+2, Michael Andersen wrote: >>> >>> It can take a while to reproduce, so it might be a week or so before I >>> have the results. >>> >> Can you think of ways to cause the same memory pressure quicker? Mocking >> data, running the functions the number of times they would have run in a >> two-week period etc. >> >> > > It's not memory pressure, there's a ton of free memory and I'm not GC'ing > particularly often. I've tried to create a smaller reproducing example but > haven't had any luck so far. I have the servers hooked up to a synthetic > load, so there are always exactly the same number of requests being made, > at the same rate, and it sometimes happens in a day, sometimes in two > weeks. Fortunately this is in a development cluster, not production. > Incidentally, it's happened in several clusters, some in the cloud some on > bare metal, so it's unlikely to be a hardware problem. > > I've added the stack print, as well as a runtime.schedtrace(true) at the > point of the error, so hopefully I will have something to work off the next > time it happens. > > -- >> You received this message because you are subscribed to a topic in the >> Google Groups "golang-nuts" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/golang-nuts/RT0vrWEc5fw/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/d3d080e0-30d5-4a5c-b5fd-5014071f3bfc%40googlegroups.com >> <https://groups.google.com/d/msgid/golang-nuts/d3d080e0-30d5-4a5c-b5fd-5014071f3bfc%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAPLpPrtFXzN9itXStxCW-j8N172h1LrGsZxOqU9G5QwQsOZJyQ%40mail.gmail.com > <https://groups.google.com/d/msgid/golang-nuts/CAPLpPrtFXzN9itXStxCW-j8N172h1LrGsZxOqU9G5QwQsOZJyQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAPLpPruKr0C3122ymyV1VCdJYiZDiNQzRqX-R3oeUpqOrKmMvw%40mail.gmail.com.
