In writing to a file option, we would periodically write the file as it might turn out to be a huge file for the entire life of the pod to flush at once.
On Fri, Sep 11, 2020 at 6:32 PM Siddhesh Divekar <siddhesh.dive...@gmail.com> wrote: > You need to write to a shared writable path that is global so when the pod >> terminated the file is avail. > > That's a good idea, we can write to to a config map. > > The http should work too if you map the ports properly. You can be able to >> access the port from outside the pod. > > This might not work. When the issue happened last time, prometheus was > unable to pull any metrics from the pod. > For the entire period prometheus output was blank which told us that the > pod was non-responsive. > > On Fri, Sep 11, 2020 at 5:59 PM Robert Engels <reng...@ix.netcom.com> > wrote: > >> You need to write to a shared writable path that is global so when the >> pod terminated the file is avail. >> >> The http should work too if you map the ports properly. You can be able >> to access the port from outside the pod. >> >> On Sep 11, 2020, at 7:14 PM, Siddhesh Divekar <siddhesh.dive...@gmail.com> >> wrote: >> >> >> I went through the suggested approaches and here are my thoughts. >> >> - runtime/trace.Start - The example in the doc >> https://golang.org/pkg/runtime/trace/ suggests adding >> os.Create("trace.out"), trace.Start & >> defer trace.Stop in the main before the application program gets started. >> In an environment like k8s if the executing program is the pod main >> program then the pod will terminate and the trace.out will be lost. >> >> - net/http/pprof package - This has the same problem where requests might >> not make to the http server. >> >> - go test -trace - Works with only tests >> >> Any thoughts/ideas of what we can do to get a trace file avoiding above >> issues ? >> >> On Fri, Sep 11, 2020 at 12:06 PM Robert Engels <reng...@ix.netcom.com> >> wrote: >> >>> Please read golang.org/cmd/trace >>> >>> You need a trace file. There are many ways to capture one. >>> >>> On Sep 11, 2020, at 12:09 PM, Siddhesh Divekar < >>> siddhesh.dive...@gmail.com> wrote: >>> >>> >>> Ok, let me get them working first. >>> >>> My concern with pprof is it has to be made part of my process. >>> We had an issue where user requests were not reaching our http server >>> itself. >>> >>> In this case what are my options if pprof server is not reachable when >>> we hit the same issue again. >>> Are there any other tools which can be run independently? >>> >>> On Fri, Sep 11, 2020 at 5:20 AM Robert Engels <reng...@ix.netcom.com> >>> wrote: >>> >>>> I would start with making sure you can get the standard ‘go tool trace’ >>>> and ‘pprof’ working. Once you have those working ‘goanalyzer’ is an >>>> enhanced version. >>>> >>>> The docs are far more complete and substantial on the standard tools. >>>> >>>> On Sep 11, 2020, at 1:37 AM, Siddhesh Divekar < >>>> siddhesh.dive...@gmail.com> wrote: >>>> >>>> >>>> Is there any dependency on GOPATH while running `./goanalyzer binary >>>> trace-file` ? >>>> Also my goanalyzer is built on mac and am trying to look at trace file >>>> generated on ubuntu (shouldn't be a problem). >>>> >>>> ./goanalyzer ~/workspace/binary ~/workspace/trace >>>> 2020/09/10 23:27:32 Parsing trace... >>>> 2020/09/10 23:27:32 Splitting trace... >>>> 2020/09/10 23:27:32 Opening browser. Trace viewer is listening on >>>> http://127.0.0.1:55520 >>>> >>>> If I try to go to a particular go routing eg >>>> http://127.0.0.1:55520/trace?goid=9084 I see nothing on the browser. >>>> >>>> Clicking on any of the graphs for a goroutine says the following. >>>> >>>> failed to execute go tool pprof: exit status 1 >>>> failed to execute dot. Is Graphviz installed? Error: exec: "dot": >>>> executable file not found in $PATH >>>> >>>> >>>> On Thu, Sep 10, 2020 at 4:49 PM robert engels <reng...@ix.netcom.com> >>>> wrote: >>>> >>>>> goanalyzer = go tool trace >>>>> >>>>> The options to goanalyzer are the same as ‘go tool trace’ - the usage >>>>> message is misleading in this way. >>>>> >>>>> By ‘map’ I mean expose the pprof port as you would any other port, and >>>>> yes you use the net/http/pprof to start the internal webserver - you don’t >>>>> need to create your own. >>>>> >>>>> >>>>> >>>>> On Sep 10, 2020, at 6:38 PM, Siddhesh Divekar < >>>>> siddhesh.dive...@gmail.com> wrote: >>>>> >>>>> Hi Robert, >>>>> >>>>> Laying down the steps to make sure I understood it correctly. >>>>> >>>>> You can map the port >>>>>> >>>>> What did you mean by above ? >>>>> >>>>> but might be easier to capture to a file via code and use goanalyzer >>>>>> on the file >>>>>> >>>>> In this case I will do the following. >>>>> a) Set up a webserver in my program for getting Go profiles (with >>>>> import _ "net/http/pprof") >>>>> b) map a port from outside k8s so that i can run curl >>>>> localhost:$PORT/debug/pprof/$PROFILE_TYPE to save a profile (say >>>>> profile-file) >>>>> c) Use go tool pprof/goanalyzer to analyze given profile from the >>>>> profile-file >>>>> >>>>> I would prefer to run this outside my process as the issue we expect >>>>> to happen is when my program is hung. >>>>> I am not sure if the http server would respond & generate profile >>>>> files. >>>>> >>>>> From the help of goanalyzer it's not very clear how I would pass the >>>>> profile-file to it. >>>>> It's showing how to generate profile-file using go test/tool but not >>>>> how to pass them to goanalyzer. >>>>> >>>>> # ./goanalyzer -h >>>>> Usage of 'go tool trace': >>>>> Given a trace file produced by 'go test': >>>>> go test -trace=trace.out pkg >>>>> >>>>> Open a web browser displaying trace: >>>>> go tool trace [flags] [pkg.test] trace.out >>>>> >>>>> Generate a pprof-like profile from the trace: >>>>> go tool trace -pprof=TYPE [pkg.test] trace.out >>>>> >>>>> [pkg.test] argument is required for traces produced by Go 1.6 and >>>>> below. >>>>> Go 1.7 does not require the binary argument. >>>>> >>>>> Supported profile types are: >>>>> - net: network blocking profile >>>>> - sync: synchronization blocking profile >>>>> - syscall: syscall blocking profile >>>>> - sched: scheduler latency profile >>>>> >>>>> Flags: >>>>> -http=addr: HTTP service address (e.g., ':6060') >>>>> -pprof=type: print a pprof-like profile instead >>>>> -d: print debug info such as parsed events >>>>> >>>>> >>>>> On Thu, Sep 10, 2020 at 3:22 PM Robert Engels <reng...@ix.netcom.com> >>>>> wrote: >>>>> >>>>>> You can map the port but might be easier to capture to a file via >>>>>> code and use goanalyzer on the file. >>>>>> >>>>>> On Sep 10, 2020, at 4:53 PM, Siddhesh Divekar < >>>>>> siddhesh.dive...@gmail.com> wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> Has anyone tried running goanalyzer on golang process running inside >>>>>> k8s pod. >>>>>> >>>>>> If so can you point me to the steps. >>>>>> >>>>>> -- >>>>>> -Siddhesh. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> >>>>> >>>>> -- >>>>> -Siddhesh. >>>>> >>>>> >>>>> >>>> >>>> -- >>>> -Siddhesh. >>>> >>>> >>> >>> -- >>> -Siddhesh. >>> >>> >> >> -- >> -Siddhesh. >> >> > > -- > -Siddhesh. > -- -Siddhesh. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BivMe9ZM53nVo5Pnfp-6Kdb1p_%3DPfhX_c5JTTy9wi%3DsdQ%40mail.gmail.com.
