And I’ll repeat, you probably need a pool to limit the number of concurrent processes/connections. If you had that you would probably of detected the problem sooner.
> On Feb 5, 2020, at 6:32 PM, Robert Engels <reng...@ix.netcom.com> wrote: > > > I am not familiar with YouTube-dl but if you can pass a timeout on the > command line that would be safest. > > Using CommandContext will forcibly kill the process. > >>> On Feb 5, 2020, at 4:13 PM, jnaderl...@gmail.com wrote: >>> >> >> So someone answered on the SO question I had posted, explaining how to check >> the number of processes, they suggest I add a timeout, since it seems >> youtube-dl, after succesffully running, sometimes hangs, so it seems golang >> isn't closing it sometimes. >> >> Should I do a timeout, orinstead the CommandContext? >> >>> On Wednesday, February 5, 2020 at 2:20:25 PM UTC-7, Johnathan Nader wrote: >>> How can I debug the number of threads and go routines running and checking >>> if the Wait()'s finish? Because I believe that may be problem, that they >>> hang. >>> >>> And when you say append the output, are you saying make a go routine to >>> write to the headers? If you have an example I would appreciate it >>> >>>> On Wednesday, February 5, 2020 at 8:47:05 AM UTC-7, Robert Engels wrote: >>>> I think your problem may be >>>> >>>> "Depending on the HTTP protocol version and the client, calling >>>> // Write or WriteHeader may prevent future reads on the >>>> // Request.Body. For HTTP/1.x requests, handlers should read any >>>> // needed request body data before writing the response. Once the >>>> // headers have been flushed (due to either an explicit Flusher.Flush >>>> // call or writing enough data to trigger a flush), the request body >>>> // may be unavailable. For HTTP/2 requests, the Go HTTP server permits >>>> // handlers to continue to read the request body while concurrently >>>> // writing the response. However, such behavior may not be supported >>>> // by all HTTP/2 clients. Handlers should read before writing if >>>> // possible to maximize compatibility." >>>> >>>> You may need to write the ResponseHeader as a final stage and append the >>>> output - if you write the header you may be hanging the input stages. If >>>> the input stage hangs (you tube download hangs, etc.), the whole process >>>> is going to hang. >>>> >>>> Did you debug the number of threads and go routines the process has while >>>> running? I am betting these are continually increasing. (Another check >>>> would be that all Waits() complete). >>>> >>>> Finally, I would use a CommandContext with a Deadline to ensure stragglers >>>> are cleaned-up. >>>> >>>> >>>> >>>> -----Original Message----- >>>> >From: Ian Lance Taylor <ia...@golang.org> >>>> >Sent: Feb 5, 2020 8:32 AM >>>> >To: jnade...@gmail.com >>>> >Cc: golang-nuts <golan...@googlegroups.com> >>>> >Subject: Re: [go-nuts] runtime/cgo: pthread_create failed: Resource >>>> >temporarily unavailable >>>> > >>>> >On Tue, Feb 4, 2020 at 11:22 PM <jnade...@gmail.com> wrote: >>>> >> >>>> >> I don't think that is the issue. I have tried it on a few different >>>> >> servers. Most recent one with 100 gig's of ram and 50 cores. The load >>>> >> average never goes above 9, but the ram slowly but surely on htop >>>> >> starts to go up. The go binary ends up climbing slowly in it's ram use >>>> >> over time, then after a hour or so, it reaches around 30 gigs of ram >>>> >> and then crashes, and restarts. >>>> >> >>>> >> I have it under supervisor. >>>> > >>>> >That is not inconsistent with Robert's suggestion. If you are >>>> >starting C threads that don't do any work but never exit, that is >>>> >exactly what you would see. >>>> > >>>> >It's not the only possible cause of this. There could also be a space >>>> >leak, either in C code with memory that is malloced but never freed, >>>> >or in Go code with memory that something keeps a permanent reference >>>> >to. >>>> > >>>> >Ian >>>> > >>>> > >>>> >> On Tuesday, February 4, 2020 at 2:00:55 PM UTC-7, Robert Engels wrote: >>>> >>> >>>> >>> Are you certain you are not just starting too many processes? Ie use a >>>> >>> “worker pool” so you have at most N conversions happening at the same >>>> >>> time. >>>> >>> >>>> >>> On Feb 4, 2020, at 2:34 PM, Robert Engels <ren...@ix.netcom.com> >>>> >>> wrote: >>>> >>> >>>> >>> >>>> >>> I will take a more in-depth look this evening. >>>> >>> >>>> >>> On Feb 4, 2020, at 2:19 PM, jnade...@gmail.com wrote: >>>> >>> >>>> >>> >>>> >>> I also thought the Wait() took care of closing the file descriptors? >>>> >>> Are you saying I should add a pipe3.Close()? Or a youtube.Close()? >>>> >>> >>>> >>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> >>> Groups "golang-nuts" group. >>>> >>> To unsubscribe from this group and stop receiving emails from it, send >>>> >>> an email to golan...@googlegroups.com. >>>> >>> To view this discussion on the web visit >>>> >>> https://groups.google.com/d/msgid/golang-nuts/2cbf0d51-0b36-4c37-a324-8a343193769a%40googlegroups.com. >>>> >>> >>>> >>> >>>> >>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> >>> Groups "golang-nuts" group. >>>> >>> To unsubscribe from this group and stop receiving emails from it, send >>>> >>> an email to golan...@googlegroups.com. >>>> >>> To view this discussion on the web visit >>>> >>> https://groups.google.com/d/msgid/golang-nuts/A56FB7F5-95D2-4B0E-B90C-B335B8714009%40ix.netcom.com. >>>> >>> >>>> >> >>>> >> -- >>>> >> You received this message because you are subscribed to the Google >>>> >> Groups "golang-nuts" group. >>>> >> To unsubscribe from this group and stop receiving emails from it, send >>>> >> an email to golan...@googlegroups.com. >>>> >> To view this discussion on the web visit >>>> >> https://groups.google.com/d/msgid/golang-nuts/0e7e67a6-6ad4-4187-9f15-0d9f278b55a2%40googlegroups.com. >>>> >> >>>> > >>>> >-- >>>> >You received this message because you are subscribed to the Google Groups >>>> >"golang-nuts" group. >>>> >To unsubscribe from this group and stop receiving emails from it, send an >>>> >email to golan...@googlegroups.com. >>>> >To view this discussion on the web visit >>>> >https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVeSsLHe92m8eQue4Lfyk3uf%2B%3D94Qy7ZapHdjsgA1RjhA%40mail.gmail.com. >>>> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/d4e3fdc3-4373-43a0-b902-4115e1656c64%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1C14A52E-69B8-4B86-BDE9-C87D3DB1DDCB%40ix.netcom.com.
