> > You need to write to a shared writable path that is global so when the pod > terminated the file is avail.
That's a good idea, we can write to to a config map. The http should work too if you map the ports properly. You can be able to > access the port from outside the pod. This might not work. When the issue happened last time, prometheus was unable to pull any metrics from the pod. For the entire period prometheus output was blank which told us that the pod was non-responsive. On Fri, Sep 11, 2020 at 5:59 PM Robert Engels <reng...@ix.netcom.com> wrote: > You need to write to a shared writable path that is global so when the pod > terminated the file is avail. > > The http should work too if you map the ports properly. You can be able to > access the port from outside the pod. > > On Sep 11, 2020, at 7:14 PM, Siddhesh Divekar <siddhesh.dive...@gmail.com> > wrote: > > > I went through the suggested approaches and here are my thoughts. > > - runtime/trace.Start - The example in the doc > https://golang.org/pkg/runtime/trace/ suggests adding > os.Create("trace.out"), trace.Start & > defer trace.Stop in the main before the application program gets started. > In an environment like k8s if the executing program is the pod main > program then the pod will terminate and the trace.out will be lost. > > - net/http/pprof package - This has the same problem where requests might > not make to the http server. > > - go test -trace - Works with only tests > > Any thoughts/ideas of what we can do to get a trace file avoiding above > issues ? > > On Fri, Sep 11, 2020 at 12:06 PM Robert Engels <reng...@ix.netcom.com> > wrote: > >> Please read golang.org/cmd/trace >> >> You need a trace file. There are many ways to capture one. >> >> On Sep 11, 2020, at 12:09 PM, Siddhesh Divekar < >> siddhesh.dive...@gmail.com> wrote: >> >> >> Ok, let me get them working first. >> >> My concern with pprof is it has to be made part of my process. >> We had an issue where user requests were not reaching our http server >> itself. >> >> In this case what are my options if pprof server is not reachable when we >> hit the same issue again. >> Are there any other tools which can be run independently? >> >> On Fri, Sep 11, 2020 at 5:20 AM Robert Engels <reng...@ix.netcom.com> >> wrote: >> >>> I would start with making sure you can get the standard ‘go tool trace’ >>> and ‘pprof’ working. Once you have those working ‘goanalyzer’ is an >>> enhanced version. >>> >>> The docs are far more complete and substantial on the standard tools. >>> >>> On Sep 11, 2020, at 1:37 AM, Siddhesh Divekar < >>> siddhesh.dive...@gmail.com> wrote: >>> >>> >>> Is there any dependency on GOPATH while running `./goanalyzer binary >>> trace-file` ? >>> Also my goanalyzer is built on mac and am trying to look at trace file >>> generated on ubuntu (shouldn't be a problem). >>> >>> ./goanalyzer ~/workspace/binary ~/workspace/trace >>> 2020/09/10 23:27:32 Parsing trace... >>> 2020/09/10 23:27:32 Splitting trace... >>> 2020/09/10 23:27:32 Opening browser. Trace viewer is listening on >>> http://127.0.0.1:55520 >>> >>> If I try to go to a particular go routing eg >>> http://127.0.0.1:55520/trace?goid=9084 I see nothing on the browser. >>> >>> Clicking on any of the graphs for a goroutine says the following. >>> >>> failed to execute go tool pprof: exit status 1 >>> failed to execute dot. Is Graphviz installed? Error: exec: "dot": >>> executable file not found in $PATH >>> >>> >>> On Thu, Sep 10, 2020 at 4:49 PM robert engels <reng...@ix.netcom.com> >>> wrote: >>> >>>> goanalyzer = go tool trace >>>> >>>> The options to goanalyzer are the same as ‘go tool trace’ - the usage >>>> message is misleading in this way. >>>> >>>> By ‘map’ I mean expose the pprof port as you would any other port, and >>>> yes you use the net/http/pprof to start the internal webserver - you don’t >>>> need to create your own. >>>> >>>> >>>> >>>> On Sep 10, 2020, at 6:38 PM, Siddhesh Divekar < >>>> siddhesh.dive...@gmail.com> wrote: >>>> >>>> Hi Robert, >>>> >>>> Laying down the steps to make sure I understood it correctly. >>>> >>>> You can map the port >>>>> >>>> What did you mean by above ? >>>> >>>> but might be easier to capture to a file via code and use goanalyzer on >>>>> the file >>>>> >>>> In this case I will do the following. >>>> a) Set up a webserver in my program for getting Go profiles (with >>>> import _ "net/http/pprof") >>>> b) map a port from outside k8s so that i can run curl >>>> localhost:$PORT/debug/pprof/$PROFILE_TYPE to save a profile (say >>>> profile-file) >>>> c) Use go tool pprof/goanalyzer to analyze given profile from the >>>> profile-file >>>> >>>> I would prefer to run this outside my process as the issue we expect to >>>> happen is when my program is hung. >>>> I am not sure if the http server would respond & generate profile files. >>>> >>>> From the help of goanalyzer it's not very clear how I would pass the >>>> profile-file to it. >>>> It's showing how to generate profile-file using go test/tool but not >>>> how to pass them to goanalyzer. >>>> >>>> # ./goanalyzer -h >>>> Usage of 'go tool trace': >>>> Given a trace file produced by 'go test': >>>> go test -trace=trace.out pkg >>>> >>>> Open a web browser displaying trace: >>>> go tool trace [flags] [pkg.test] trace.out >>>> >>>> Generate a pprof-like profile from the trace: >>>> go tool trace -pprof=TYPE [pkg.test] trace.out >>>> >>>> [pkg.test] argument is required for traces produced by Go 1.6 and below. >>>> Go 1.7 does not require the binary argument. >>>> >>>> Supported profile types are: >>>> - net: network blocking profile >>>> - sync: synchronization blocking profile >>>> - syscall: syscall blocking profile >>>> - sched: scheduler latency profile >>>> >>>> Flags: >>>> -http=addr: HTTP service address (e.g., ':6060') >>>> -pprof=type: print a pprof-like profile instead >>>> -d: print debug info such as parsed events >>>> >>>> >>>> On Thu, Sep 10, 2020 at 3:22 PM Robert Engels <reng...@ix.netcom.com> >>>> wrote: >>>> >>>>> You can map the port but might be easier to capture to a file via code >>>>> and use goanalyzer on the file. >>>>> >>>>> On Sep 10, 2020, at 4:53 PM, Siddhesh Divekar < >>>>> siddhesh.dive...@gmail.com> wrote: >>>>> >>>>> >>>>> Hi, >>>>> >>>>> Has anyone tried running goanalyzer on golang process running inside >>>>> k8s pod. >>>>> >>>>> If so can you point me to the steps. >>>>> >>>>> -- >>>>> -Siddhesh. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "golang-nuts" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to golang-nuts+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>> >>>> -- >>>> -Siddhesh. >>>> >>>> >>>> >>> >>> -- >>> -Siddhesh. >>> >>> >> >> -- >> -Siddhesh. >> >> > > -- > -Siddhesh. > > -- -Siddhesh. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BjSCa-brgvC1dUAChG2gdV2aGnRy1E-%2Bzp%3D2EBm8J6xQg%40mail.gmail.com.
