Re: [go-nuts] Running goanalyzer on golang process running inside kubernetes pod

[Siddhesh Divekar]

Is there any dependency on GOPATH while running `./goanalyzer binary
trace-file` ?
Also my goanalyzer is built on mac and am trying to look at trace file
generated on ubuntu (shouldn't be a problem).

./goanalyzer ~/workspace/binary  ~/workspace/trace
2020/09/10 23:27:32 Parsing trace...
2020/09/10 23:27:32 Splitting trace...
2020/09/10 23:27:32 Opening browser. Trace viewer is listening on
http://127.0.0.1:55520

If I try to go to a particular go routing eg
http://127.0.0.1:55520/trace?goid=9084 I see nothing on the browser.

Clicking on any of the graphs for a goroutine says the following.

failed to execute go tool pprof: exit status 1
failed to execute dot. Is Graphviz installed? Error: exec: "dot":
executable file not found in $PATH


On Thu, Sep 10, 2020 at 4:49 PM robert engels  wrote:

> goanalyzer = go tool trace
>
> The options to goanalyzer are the same as ‘go tool trace’ - the usage
> message is misleading in this way.
>
> By ‘map’ I mean expose the pprof port as you would any other port, and yes
> you use the net/http/pprof to start the internal webserver - you don’t need
> to create your own.
>
>
>
> On Sep 10, 2020, at 6:38 PM, Siddhesh Divekar 
> wrote:
>
> Hi Robert,
>
> Laying down the steps to make sure I understood it correctly.
>
> You can map the port
>>
> What did you mean by above ?
>
> but might be easier to capture to a file via code and use goanalyzer on
>> the file
>>
> In this case I will do the following.
> a) Set up a webserver in my program for getting Go profiles (with import _
> "net/http/pprof")
> b) map a port from outside k8s so that i can run curl
> localhost:$PORT/debug/pprof/$PROFILE_TYPE to save a profile (say
> profile-file)
> c) Use go tool pprof/goanalyzer to analyze given profile from the
> profile-file
>
> I would prefer to run this outside my process as the issue we expect to
> happen is when my program is hung.
> I am not sure if the http server would respond & generate profile files.
>
> From the help of goanalyzer it's not very clear how I would pass the
> profile-file to it.
> It's showing how to generate profile-file using go test/tool but not how
> to pass them to goanalyzer.
>
> # ./goanalyzer -h
> Usage of 'go tool trace':
> Given a trace file produced by 'go test':
> go test -trace=trace.out pkg
>
> Open a web browser displaying trace:
> go tool trace [flags] [pkg.test] trace.out
>
> Generate a pprof-like profile from the trace:
> go tool trace -pprof=TYPE [pkg.test] trace.out
>
> [pkg.test] argument is required for traces produced by Go 1.6 and below.
> Go 1.7 does not require the binary argument.
>
> Supported profile types are:
> - net: network blocking profile
> - sync: synchronization blocking profile
> - syscall: syscall blocking profile
> - sched: scheduler latency profile
>
> Flags:
> -http=addr: HTTP service address (e.g., ':6060')
> -pprof=type: print a pprof-like profile instead
> -d: print debug info such as parsed events
>
>
> On Thu, Sep 10, 2020 at 3:22 PM Robert Engels 
> wrote:
>
>> You can map the port but might be easier to capture to a file via code
>> and use goanalyzer on the file.
>>
>> On Sep 10, 2020, at 4:53 PM, Siddhesh Divekar 
>> wrote:
>>
>> 
>> Hi,
>>
>> Has anyone tried running goanalyzer on golang process running inside k8s
>> pod.
>>
>> If so can you point me to the steps.
>>
>> --
>> -Siddhesh.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com
>> 
>> .
>>
>>
>
> --
> -Siddhesh.
>
>
>

-- 
-Siddhesh.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2Bh%2BxrRr9PqoZEAsjv-QVJOi8pzP4gk7VUFfnfXw-Je51Q%40mail.gmail.com.

Re: [go-nuts] zombie parent scenario with golang

[Bakul Shah]

1. Looks like*something* in ps reports process/thread state incorrectly. It 
should not report  until all the pthreads have exited and the parent 
has not picked up the status. The runtime will call exit() when the last thread 
terminates (exit() in turn will call the _exit syscall).

2. If any thread calls _exit(), the system will clean up everything. It doesn't 
matter how many threads may  be active. You can see this for yourself if you 
replaced pthread_exit() in main() with _exit(0).

3. stacktrace within the kernel mode is irrelevant. You are merely confusing 
yourself.

4. Go runtime doesn't use pthread so not sure testing pthread based C program 
is relevant. Exiting from func main() will kill all goroutines. Copy 
https://play.golang.org/p/zRfhvfYt_oE locally and see for yourself.

5. Looking at your original message, it *seems* like a parent is not picking up 
the child's status. But I can't be sure.

I suspect you are on a wild goose chase. Possibly confused by ps. You may wish 
to backtrack to whatever you were looking at before this  came up. Or 
try explaining what is going on with your go program and what you expect it 
should do, without stacktrace or C programs etc.

I wouldn't switch to a newer kernel if I were you. When debugging you should 
keep everything else fixed or else you may end up chasing something different 
or the symptom may change or disappear.

> On Sep 10, 2020, at 10:08 PM, Uday Kiran Jonnala  wrote:
> 
> Thanks Kurtis for the reply. I understand defunct process mechanism. 
> 
> As I mentioned in the initial mail, [Correct me if I am wrong here], In a 
> process if there is main thread and a detached thread created by main thread, 
> when the main thread exits the process is kept in defunct state, since the 
> created thread is still
> executing, I was thinking if we have such scenario in go runtime. That could 
> be the reason I see this thread is waiting on futex and holding the file 
> handles and causing the go process (kernel) not to send SIGCHLD to parent 
> process.
> 
> For example below case
> 
> #include 
> #include 
> #include 
> #include 
> 
> void *thread_function(void *args)
> {
> printf("The is new thread! Sleep 20 seconds...\n");
> sleep(100);
> printf("Exit from thread\n");
> pthread_exit(0);
> }
> 
> int main(int argc, char **argv)
> {
>  pthread_t thrd;
>  pthread_attr_t attr;
>  int res = 0;
>  res = pthread_attr_init(&attr);
>  res = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
>  res = pthread_create(&thrd, &attr, thread_function, NULL);
>  res = pthread_attr_destroy(&attr);
>  printf("Main thread. Sleep 5 seconds\n");
>  sleep(5);
>  printf("Exit from main process\n");
>  pthread_exit(0);
> }
> 
> ujonnala@ ~/mycode/go () $ ps -T
>PID   SPID TTY  TIME CMD
>  43635  43635 pts/29   00:00:00 a.out 
>  43635  43638 pts/29   00:00:00 a.out
> 
> Due to the detached thread still executing the process left in defunt state. 
> 
> Thanks for checking on this, I will see if we can reproduce my situation on a 
> newer kernel.
> 
> Thanks & Regards,
> Uday Kiran
> 
> On Thursday, September 10, 2020 at 9:49:06 PM UTC-7 Kurtis Rader wrote:
> On Thu, Sep 10, 2020 at 9:25 PM Uday Kiran Jonnala  > wrote:
> Thanks for the reply. We are fixing the issue. But the point I wanted to 
> bring it up here is the issue of a thread causing the go process to be in 
> defunct state.
> 
> Any thread can cause the go process to enter the "defunct" state. For 
> example, by calling os.Exit(), or panic(), or causing a signal to be 
> delivered that terminates the process (e.g., SIGSEGV).
>  
> My kernel version is 
> Linux version 4.14.175-1.nutanix.20200709.el7.x86_64 (dev@ca4b0551898c) (gcc 
> version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)) #1 SMP Fri Jul 10 02:17:54 
> UTC 2020
> 
> Is that the output of `uname -a`? It seems to suggest you're using CentOS 
> provided by the https://www.nutanix.com/go/linux-on-ahv 
>  cloud environment. So we've 
> established you are using Linux with kernel version 4.14. A kernel that is 
> now three years old. I don't have anything like it installed on any of my 
> virtual machines so I can't explore how it handles defunct processes. But my 
> prior point stands: A "defunct" process is one that has been terminated but 
> whose parent process has not reaped its exit status. Either that parent 
> process has a bug (the most likely explanation) or your OS has a bug.
> 
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/ad4843e1-f7d1-43ae-8091-579bc61527fdn%40googlegroups.com
>  

Re: [go-nuts] zombie parent scenario with golang

[Kurtis Rader]

Your example is a C program. I'm guessing you're using gccgo to link with
equivalent C code. In which case your question has almost nothing to do
with Go. You need to ask the Linux community why your example results in a
defunct process that appears to have a live thread.

I do not believe you "understand defunct process mechanism". Because a
defunct process is one that does not have any executing threads. Yet you
still seem to think that Go, somehow, creates a process that is both
defunct and has an executing thread.

On Thu, Sep 10, 2020 at 10:08 PM Uday Kiran Jonnala 
wrote:

> Thanks Kurtis for the reply. I understand defunct process mechanism.
>
> As I mentioned in the initial mail, [Correct me if I am wrong here], In a
> process if there is main thread and a detached thread created by main
> thread, when the main thread exits the process is kept in defunct state,
> since the created thread is still
> executing, I was thinking if we have such scenario in go runtime. That
> could be the reason I see this thread is waiting on futex and holding the
> file handles and causing the go process (kernel) not to send SIGCHLD to
> parent process.
>
> For example below case
>
> #include 
> #include 
> #include 
> #include 
>
> void *thread_function(void *args)
> {
> printf("The is new thread! Sleep 20 seconds...\n");
> sleep(100);
> printf("Exit from thread\n");
> pthread_exit(0);
> }
>
> int main(int argc, char **argv)
> {
>  pthread_t thrd;
>  pthread_attr_t attr;
>  int res = 0;
>  res = pthread_attr_init(&attr);
>  res = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
>  res = pthread_create(&thrd, &attr, thread_function, NULL);
>  res = pthread_attr_destroy(&attr);
>  printf("Main thread. Sleep 5 seconds\n");
>  sleep(5);
>  printf("Exit from main process\n");
>  pthread_exit(0);
> }
>
> ujonnala@ ~/mycode/go () $ ps -T
>PID   SPID TTY  TIME CMD
>  43635  43635 pts/29   00:00:00 a.out 
>  43635  43638 pts/29   00:00:00 a.out
>
> Due to the detached thread still executing the process left in defunt
> state.
>
> Thanks for checking on this, I will see if we can reproduce my situation
> on a newer kernel.
>
> Thanks & Regards,
> Uday Kiran
>
> On Thursday, September 10, 2020 at 9:49:06 PM UTC-7 Kurtis Rader wrote:
>
>> On Thu, Sep 10, 2020 at 9:25 PM Uday Kiran Jonnala 
>> wrote:
>>
>>> Thanks for the reply. We are fixing the issue. But the point I wanted to
>>> bring it up here is the issue of a thread causing the go process to be in
>>> defunct state.
>>>
>>
>> Any thread can cause the go process to enter the "defunct" state. For
>> example, by calling os.Exit(), or panic(), or causing a signal to be
>> delivered that terminates the process (e.g., SIGSEGV).
>>
>>
>>> My kernel version is
>>> Linux version 4.14.175-1.nutanix.20200709.el7.x86_64 (dev@ca4b0551898c)
>>> (gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)) #1 SMP Fri Jul 10
>>> 02:17:54 UTC 2020
>>>
>>
>> Is that the output of `uname -a`? It seems to suggest you're using CentOS
>> provided by the https://www.nutanix.com/go/linux-on-ahv cloud
>> environment. So we've established you are using Linux with kernel version
>> 4.14. A kernel that is now three years old. I don't have anything like it
>> installed on any of my virtual machines so I can't explore how it handles
>> defunct processes. But my prior point stands: A "defunct" process is one
>> that has been terminated but whose parent process has not reaped its exit
>> status. Either that parent process has a bug (the most likely explanation)
>> or your OS has a bug.
>>
>> --
>> Kurtis Rader
>> Caretaker of the exceptional canines Junior and Hank
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/ad4843e1-f7d1-43ae-8091-579bc61527fdn%40googlegroups.com
> 
> .
>


-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-AA6N7jw2%3DR5gamWoewpokuRzVz_OSo70jGrWQ%2Bpj7DA%40mail.gmail.com.

Re: [go-nuts] zombie parent scenario with golang

[Uday Kiran Jonnala]

Thanks Kurtis for the reply. I understand defunct process mechanism. 

As I mentioned in the initial mail, [Correct me if I am wrong here], In a 
process if there is main thread and a detached thread created by main 
thread, when the main thread exits the process is kept in defunct state, 
since the created thread is still
executing, I was thinking if we have such scenario in go runtime. That 
could be the reason I see this thread is waiting on futex and holding the 
file handles and causing the go process (kernel) not to send SIGCHLD to 
parent process.

For example below case

#include 
#include 
#include 
#include 

void *thread_function(void *args)
{
printf("The is new thread! Sleep 20 seconds...\n");
sleep(100);
printf("Exit from thread\n");
pthread_exit(0);
}

int main(int argc, char **argv)
{
 pthread_t thrd;
 pthread_attr_t attr;
 int res = 0;
 res = pthread_attr_init(&attr);
 res = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
 res = pthread_create(&thrd, &attr, thread_function, NULL);
 res = pthread_attr_destroy(&attr);
 printf("Main thread. Sleep 5 seconds\n");
 sleep(5);
 printf("Exit from main process\n");
 pthread_exit(0);
}

ujonnala@ ~/mycode/go () $ ps -T
   PID   SPID TTY  TIME CMD
 43635  43635 pts/29   00:00:00 a.out 
 43635  43638 pts/29   00:00:00 a.out

Due to the detached thread still executing the process left in defunt 
state. 

Thanks for checking on this, I will see if we can reproduce my situation on 
a newer kernel.

Thanks & Regards,
Uday Kiran

On Thursday, September 10, 2020 at 9:49:06 PM UTC-7 Kurtis Rader wrote:

> On Thu, Sep 10, 2020 at 9:25 PM Uday Kiran Jonnala  
> wrote:
>
>> Thanks for the reply. We are fixing the issue. But the point I wanted to 
>> bring it up here is the issue of a thread causing the go process to be in 
>> defunct state.
>>
>
> Any thread can cause the go process to enter the "defunct" state. For 
> example, by calling os.Exit(), or panic(), or causing a signal to be 
> delivered that terminates the process (e.g., SIGSEGV).
>  
>
>> My kernel version is 
>> Linux version 4.14.175-1.nutanix.20200709.el7.x86_64 (dev@ca4b0551898c) 
>> (gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)) #1 SMP Fri Jul 10 
>> 02:17:54 UTC 2020
>>
>
> Is that the output of `uname -a`? It seems to suggest you're using CentOS 
> provided by the https://www.nutanix.com/go/linux-on-ahv cloud 
> environment. So we've established you are using Linux with kernel version 
> 4.14. A kernel that is now three years old. I don't have anything like it 
> installed on any of my virtual machines so I can't explore how it handles 
> defunct processes. But my prior point stands: A "defunct" process is one 
> that has been terminated but whose parent process has not reaped its exit 
> status. Either that parent process has a bug (the most likely explanation) 
> or your OS has a bug.
>
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ad4843e1-f7d1-43ae-8091-579bc61527fdn%40googlegroups.com.

Re: [go-nuts] zombie parent scenario with golang

[Kurtis Rader]

On Thu, Sep 10, 2020 at 9:25 PM Uday Kiran Jonnala 
wrote:

> Thanks for the reply. We are fixing the issue. But the point I wanted to
> bring it up here is the issue of a thread causing the go process to be in
> defunct state.
>

Any thread can cause the go process to enter the "defunct" state. For
example, by calling os.Exit(), or panic(), or causing a signal to be
delivered that terminates the process (e.g., SIGSEGV).


> My kernel version is
> Linux version 4.14.175-1.nutanix.20200709.el7.x86_64 (dev@ca4b0551898c)
> (gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)) #1 SMP Fri Jul 10
> 02:17:54 UTC 2020
>

Is that the output of `uname -a`? It seems to suggest you're using CentOS
provided by the https://www.nutanix.com/go/linux-on-ahv cloud environment.
So we've established you are using Linux with kernel version 4.14. A kernel
that is now three years old. I don't have anything like it installed on any
of my virtual machines so I can't explore how it handles defunct processes.
But my prior point stands: A "defunct" process is one that has been
terminated but whose parent process has not reaped its exit status. Either
that parent process has a bug (the most likely explanation) or your OS has
a bug.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-VA7DOcAMkGmMSiH%2Bs-iugfzCU-4dgr_PnmqkjdbnjrQ%40mail.gmail.com.

Re: [go-nuts] zombie parent scenario with golang

[Uday Kiran Jonnala]

Hi Ian, Kurtis,

Thanks for the reply. We are fixing the issue. But the point I wanted to 
bring it up here is the issue of a thread causing the go process to be in 
defunct state.
My kernel version is 
Linux version 4.14.175-1.nutanix.20200709.el7.x86_64 (dev@ca4b0551898c) 
(gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)) #1 SMP Fri Jul 10 
02:17:54 UTC 2020

Thanks & Regards,
Uday Kiran

On Thursday, September 10, 2020 at 6:42:06 PM UTC-7 Ian Lance Taylor wrote:

> On Thu, Sep 10, 2020 at 5:09 PM Kurtis Rader  wrote:
> >
> > A defunct process is a process that has terminated but whose parent 
> process has not called wait() or one of its variants. I don't know why lsof 
> still reports open files. It shouldn't since a dead process should have its 
> resources, such as its file descriptor table, freed by the kernel even if 
> the parent hasn't called wait(). You didn't tell us the details of the OS 
> you're using so I would simply assume it's a quirk of your OS. It might be 
> more productive to look into why your program is panicing at 
> map_faststr.go:275. A likely explanation is you have a race in your program 
> that is causing it to attempt to mutate a map concurrently or you're trying 
> to insert into a nil map.
>
> That's a good point. What OS are you using? I don't think you said.
>
> Ian
>
>
> > On Thu, Sep 10, 2020 at 4:43 PM Uday Kiran Jonnala  
> wrote:
> >>
> >> Hi Ian,
> >>
> >> Again. Thanks for the reply. Problem here is we see go process is in 
> defunt process and sure parent process did not get SIGCHILD and looking 
> deeper,
> >> I see a thread in futex_wait_queue_me. If we think we are just getting 
> the stack trace and the go process actually got killed, why would I see
> >> associated fd's in file table and fd table is still intact (see lsof 
> information)
> >>
> >> Process which is in defunt state which got panic is <87548>, checking 
> for threads in this which is 87548
> >>
> >> bash-4.2# cat /proc/87548/status
> >> Name: replicator
> >> State: Z (zombie)
> >>
> >> bash-4.2# ls -Fl /proc/87548/task/87561/fd | grep 606649
> >> l-wx--. 1 root root 64 Aug 25 10:59 1 -> pipe:[606649]
> >> l-wx--. 1 root root 64 Aug 25 10:59 2 -> pipe:[606649]
> >>
> >> Listing the threads
> >>
> >> bash-4.2# ps -aefT | grep 87548
> >> root 87548 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
> >> root 87548 87561 87507 0 Aug23 ? 00:00:00 [replicator] 
> >> root 112448 112448 42566 0 17:13 pts/0 00:00:00 grep 87548
> >>
> >> bash-4.2# lsof | grep 606649
> >> replicato 87548 87561 root 1w FIFO 0,11 0t0 606649 pipe
> >> replicato 87548 87561 root 2w FIFO 0,11 0t0 606649 pipe
> >>
> >> Why does lsof show the entry for the FIFO file of this process?
> >>
> >> So I feel we have a scenario the thread which is sleeping on 
> futex_wait_queue_me is not cleanup during panic() and causing the main
> >> thread to be exited leaving detached thread which waiting in 
> futex_wait_queue_me is still present.
> >>
> >> The main issue is I am not able to reproduce this, since this go 
> process is very big.
> >>
> >> Any way to verify this OR take it further.
> >>
> >> Thanks & Regards,
> >> Uday Kiran
> >> On Monday, September 7, 2020 at 12:05:05 PM UTC-7 Ian Lance Taylor 
> wrote:
> >>>
> >>> On Mon, Sep 7, 2020 at 12:03 AM Uday Kiran Jonnala  
> wrote:
> >>> >
> >>> > Thanks for the reply, I get the point on zombie, I do not think the 
> issue here is parent not reaping child, seems like go process has not 
> finished execution of some
> >>> > internal threads (waiting on some futex) and causing SIGCHILD not to 
> be sent to parent.
> >>> >
> >>> > go process named  hit with panic and I see this went 
> into zombie state
> >>> >
> >>> > $ ps -ef | grep replicator
> >>> > root 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
> >>> >
> >>> > Now looking at the tasks within the process
> >>> >
> >>> > I see the stack trace of the threads within the process still stuck 
> on following
> >>> >
> >>> > bash-4.2# cat /proc/87548/task/87561/stack
> >>> > [] futex_wait_queue_me+0xc4/0x120
> >>> > [] futex_wait+0x10a/0x250
> >>> > [] do_futex+0x35e/0x5b0
> >>> > [] SyS_futex+0x13b/0x180
> >>> > [] do_syscall_64+0x79/0x1b0
> >>> > [] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> >>> > [] 0x
> >>> >
> >>> > From the above example if we are creating some internal threads and 
> main thread is excited due to panic and left some detached threads, process 
> will be in zombie state until the threads
> >>> > within the process completes.
> >>> >
> >>> > It appears there is some run away threads hung state scenario 
> causing this. I am not able to reproduce it with main go routine explict 
> panic and some go routine still executing.
> >>> >
> >>> > Does the above stack trace sound familiar wrt internal threads of Go 
> runtime ?
> >>>
> >>> If the process is defunct, then none of the thread stacks matter.
> >>> They are just where the thread happened to be when the process exited.
> >>>
> >>> What is the rea

Re: [go-nuts] unit test for function which has goroutine

[Yvonne Zhang]

Thank you Gregor. As this is just to make unit test work, if I add 
waitgroup in the code, will this affect the real streaming logic?
Is that OK, I check if is it a mock function functionB called, then I tell 
it to sleep for a sec before return. 

On Thursday, September 10, 2020 at 5:42:21 PM UTC+10 be...@pferdewetten.de 
wrote:

> Right, I wanted to write something about the goroutine and forgot about 
> it. Sorry about that.
>
> The problem here could be that in the code you posted there's nothing 
> waiting on any result from the goroutine, and if the test exits quickly 
> enough, it might not get scheduled.
>
> In general, whenever a goroutine is created, you should have some idea 
> of when and how it gets to run and when it stops running, and how other 
> goroutines wait for it to do its job.
>
> You could do this for example by controlling the lifetime of the 
> goroutine with a sync.WaitGroup. Pass in a WaitGroup and call Done() on 
> it when the goroutine returns (probably as a deferred call), then call 
> Wait() in some other goroutine (for example in your main function or in 
> the TestFoo function) so there's some sort of ordering enforced.
>
> In general, you'll need some way of signalling "I'm done copying stuff 
> from zipReader to zipWriter" in the goroutine, and you'll need to 
> arrange for some other, longer lived, goroutine to wait for that signal.
>
>
> On 10.09.20 09:34, Yvonne Zhang wrote:
> > Thank you Gregor for your quick reply!
> > 
> > Anything special I need to think about this go routine logic in my unit 
> > test. It might be still running after test returns..?
> > 
> > On Thursday, September 10, 2020 at 5:26:13 PM UTC+10 
> > be...@pferdewetten.de wrote:
> > 
> > If I were you, I'd pass in an interface value to functionA with the
> > signature
> > 
> > type Ber interface{
> > functionB(io.Reader, io.Writer)
> > }
> > 
> > (assuming that zipReader is an io.Reader, zipWriter is an io.Writer,
> > you
> > may have to adjust this a bit).
> > 
> > Then make your mock functionB a method on a type that implements the
> > Ber
> > interface, the same for the real functionB. In unit tests, pass in an
> > instance of the type with the mock method, in real life, pass in an
> > instance of the type with the real method.
> > 
> > Note that the types could be as simple as
> > 
> > type mockB struct{}
> > func (m mockB) functionB(...)
> > 
> > On 10.09.20 09:15, Yvonne Zhang wrote:
> > > Hi,
> > > I have a function streaming a zipreader to browser. It is like this.
> > > func functionA()(body io.ReadCloser, err error){
> > >
> > >// some logic to get a zipreader
> > >
> > >body, pipeWriter := io.Pipe()
> > >zipWriter := zip.NewWriter(pipeWriter)
> > >
> > >go func(){
> > >   // err := functionB(zipReader, zipWriter)call another
> > function to
> > > prepare files and write to zipwriter
> > >If err != nil{
> > >zipWriter.Close()
> > > _ = pipeWriter.CloseWithError(err)
> > > return
> > >}
> > >   zipWriter.Close()
> > >pipeWriter.Close()
> > >}()
> > >return
> > > }
> > >
> > > My question is about unit test about this functionA. I have mock for
> > > functionB. How do I write unit test for functionA with the mock
> > > functionB. It seems my unit test does not work well when it goes
> > into
> > > the goroutine.
> > > Many thanks!
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "golang-nuts" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > send
> > > an email to golang-nuts...@googlegroups.com
> > > .
> > > To view this discussion on the web visit
> > >
> > 
> https://groups.google.com/d/msgid/golang-nuts/4ca0817a-f1f7-44a9-be3d-3584bcb61b8an%40googlegroups.com
> > 
> > >
> > <
> https://groups.google.com/d/msgid/golang-nuts/4ca0817a-f1f7-44a9-be3d-3584bcb61b8an%40googlegroups.com?utm_medium=email&utm_source=footer
> >.
> > 
> > 
> > -- 
> > Gregor Best
> > be...@pferdewetten.de
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to golang-nuts...@googlegroups.com 
> > .
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/golang-nuts/33648e9a-4011-4d02-a07b-01b506cf5cd1n%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/golang-nuts/33648e9a-4011-4d02-a07b-01b506cf5cd1n%40googlegroups.com?utm_medium=email&utm_source=footer
> >.
>
> -- 
> Gregor Best
> be...@pferdewetten.de
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups

Re: [go-nuts] Running golang C-shared library in an OS that uses cooperative scheduling.

[Yonatan Gizachew]

Thank you!

On Friday, September 11, 2020 at 3:52:43 AM UTC+9 Ian Lance Taylor wrote:

> On Wed, Sep 9, 2020 at 11:58 PM Yonatan Gizachew  wrote:
> >
> > Also, is there a way to print the values of runtime.g?
>
> In what sense? Code running the runtime package can just print it.
> Or if you run the program under gdb, you should be able to "print
> 'runtime.g'" (you need the quotes because otherwise gdb will think it
> is a structure field reference). At least, gdb works that way on
> GNU/Linux, not sure about your operating system.
>
> Ian
>
> > On Thursday, September 10, 2020 at 12:15:04 PM UTC+9 Ian Lance Taylor 
> wrote:
> >>
> >> On Wed, Sep 9, 2020 at 7:39 PM Yonatan Gizachew  
> wrote:
> >> >
> >> > This might be trivial, but could you explain me the relationship 
> between the following?
> >> > 1. value returned by __tls_get_addr
> >>
> >> This is the address of the TLS control block. The exact definition
> >> depends on the architecture. All TLS variables are at some offset
> >> from this value.
> >>
> >> > 2. runtime.g
> >>
> >> A TLS variable that holds the current goroutine, a pointer to a g 
> struct.
> >>
> >> > 3. runtime.g0
> >>
> >> A global variable that holds the g struct used for the initial 
> goroutine.
> >>
> >> > 4. runtime.m0
> >>
> >> A global variable that holds the m struct used for the initial thread.
> >>
> >> > 5. runtime.m
> >>
> >> gccgo no longer uses this. In earlier releases it was a TLS variable
> >> that holds the current thread, a pointer to an m struct. In current
> >> releases gccgo uses g->m for this.
> >>
> >> Ian
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golang-nuts...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/0935d7c3-d2de-49cd-8d25-196b276c5ca9n%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5ad20ec3-6492-447c-bfa8-b01307a548b3n%40googlegroups.com.

Re: [go-nuts] zombie parent scenario with golang

[Ian Lance Taylor]

On Thu, Sep 10, 2020 at 5:09 PM Kurtis Rader  wrote:
>
> A defunct process is a process that has terminated but whose parent process 
> has not called wait() or one of its variants. I don't know why lsof still 
> reports open files. It shouldn't since a dead process should have its 
> resources, such as its file descriptor table, freed by the kernel even if the 
> parent hasn't called wait(). You didn't tell us the details of the OS you're 
> using so I would simply assume it's a quirk of your OS. It might be more 
> productive to look into why your program is panicing at map_faststr.go:275. A 
> likely explanation is you have a race in your program that is causing it to 
> attempt to mutate a map concurrently or you're trying to insert into a nil 
> map.

That's a good point.  What OS are you using?  I don't think you said.

Ian


> On Thu, Sep 10, 2020 at 4:43 PM Uday Kiran Jonnala  
> wrote:
>>
>> Hi Ian,
>>
>> Again. Thanks for the reply. Problem here is we see go process is in defunt 
>> process and sure parent process did not get SIGCHILD and looking deeper,
>> I see a thread in  futex_wait_queue_me. If we think we are just getting the 
>> stack trace and the go process actually got killed, why would I see
>> associated fd's in file table and fd table is still intact (see lsof 
>> information)
>>
>> Process which is in defunt state which got panic is <87548>, checking for 
>> threads in this which is 87548
>>
>> bash-4.2# cat /proc/87548/status
>>  Name: replicator
>>  State: Z (zombie)
>>
>> bash-4.2# ls -Fl /proc/87548/task/87561/fd | grep 606649
>> l-wx--. 1 root root 64 Aug 25 10:59 1 -> pipe:[606649]
>> l-wx--. 1 root root 64 Aug 25 10:59 2 -> pipe:[606649]
>>
>> Listing the threads
>>
>> bash-4.2# ps -aefT | grep 87548
>> root 87548 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
>> root 87548 87561 87507 0 Aug23 ? 00:00:00 [replicator] 
>> root 112448 112448 42566 0 17:13 pts/0 00:00:00 grep 87548
>>
>> bash-4.2# lsof | grep 606649
>> replicato  87548  87561root1w FIFO   0,11   0t0  
>>606649 pipe
>> replicato  87548  87561root2w FIFO   0,11   0t0  
>>606649 pipe
>>
>> Why does lsof show the entry for the FIFO file of this process?
>>
>> So I feel we have a scenario the thread which is sleeping on 
>> futex_wait_queue_me is not cleanup during panic() and causing the main
>> thread to be exited leaving detached thread which waiting in 
>> futex_wait_queue_me is still present.
>>
>> The main issue is I am not able to reproduce this, since this go process is 
>> very big.
>>
>> Any way to verify this OR  take it further.
>>
>> Thanks & Regards,
>> Uday Kiran
>> On Monday, September 7, 2020 at 12:05:05 PM UTC-7 Ian Lance Taylor wrote:
>>>
>>> On Mon, Sep 7, 2020 at 12:03 AM Uday Kiran Jonnala  
>>> wrote:
>>> >
>>> > Thanks for the reply, I get the point on zombie, I do not think the issue 
>>> > here is parent not reaping child, seems like go process has not finished 
>>> > execution of some
>>> > internal threads (waiting on some futex) and causing SIGCHILD not to be 
>>> > sent to parent.
>>> >
>>> > go process named  hit with panic and I see this went into 
>>> > zombie state
>>> >
>>> > $ ps -ef | grep replicator
>>> > root 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
>>> >
>>> > Now looking at the tasks within the process
>>> >
>>> > I see the stack trace of the threads within the process still stuck on 
>>> > following
>>> >
>>> > bash-4.2# cat /proc/87548/task/87561/stack
>>> > [] futex_wait_queue_me+0xc4/0x120
>>> > [] futex_wait+0x10a/0x250
>>> > [] do_futex+0x35e/0x5b0
>>> > [] SyS_futex+0x13b/0x180
>>> > [] do_syscall_64+0x79/0x1b0
>>> > [] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
>>> > [] 0x
>>> >
>>> > From the above example if we are creating some internal threads and main 
>>> > thread is excited due to panic and left some detached threads, process 
>>> > will be in zombie state until the threads
>>> > within the process completes.
>>> >
>>> > It appears there is some run away threads hung state scenario causing 
>>> > this. I am not able to reproduce it with main go routine explict panic 
>>> > and some go routine still executing.
>>> >
>>> > Does the above stack trace sound familiar wrt internal threads of Go 
>>> > runtime ?
>>>
>>> If the process is defunct, then none of the thread stacks matter.
>>> They are just where the thread happened to be when the process exited.
>>>
>>> What is the real problem you are seeing?
>>>
>>> Ian
>>>
>>>
>>>
>>>
>>> > On Thursday, August 27, 2020 at 1:43:39 PM UTC-7 Ian Lance Taylor wrote:
>>> >>
>>> >> On Thu, Aug 27, 2020 at 10:01 AM Uday Kiran Jonnala
>>> >>  wrote:
>>> >> >
>>> >> > I have a situation on zombie parent scenario with golang
>>> >> >
>>> >> > A process (in the case replicator) has many goroutines internally
>>> >> >
>>> >> > We hit into panic() and I see the replicator process is in Zombie state
>>> >> >
>>> >> > <<>>>:~$ ps -ef | grep replica

Re: [go-nuts] alloc vs totalalloc in memory

[Ian Lance Taylor]

On Thu, Sep 10, 2020 at 4:39 PM Alexander Mills
 wrote:
>
> I have this helper func to print memory usage (detect a memory leak?)
>
>
> func PrintMemUsage() {
> var m runtime.MemStats
> runtime.ReadMemStats(&m)
> // For info on each, see: https://golang.org/pkg/runtime/#MemStats
> fmt.Printf("Alloc = %v MiB", bToMb(m.Alloc))
> fmt.Printf("\tTotalAlloc = %v MiB", bToMb(m.TotalAlloc))
> fmt.Printf("\tSys = %v MiB", bToMb(m.Sys))
> fmt.Printf("\tNumGC = %v\n", m.NumGC)
> }
>
> I am seeing output like this:
> Alloc = 2481 MiB TotalAlloc = 24038 MiB Sys = 2752 MiB NumGC = 114
>
> what is the difference between Alloc and TotalAlloc .. ?

The link above, after "For info on each", explains the difference.  If
the explanation is unclear, tell us more.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVsLoMxgnHbf%2BizcB27f-jfwqxsL80p4ya%2Bb_bp1Wp5AQ%40mail.gmail.com.

Re: [go-nuts] zombie parent scenario with golang

[Kurtis Rader]

A defunct process is a process that has terminated but whose parent process
has not called wait() or one of its variants. I don't know why lsof still
reports open files. It shouldn't since a dead process should have its
resources, such as its file descriptor table, freed by the kernel even if
the parent hasn't called wait(). You didn't tell us the details of the OS
you're using so I would simply assume it's a quirk of your OS. It might be
more productive to look into why your program is panicing at
map_faststr.go:275. A likely explanation is you have a race in your program
that is causing it to attempt to mutate a map concurrently or you're trying
to insert into a nil map.

On Thu, Sep 10, 2020 at 4:43 PM Uday Kiran Jonnala 
wrote:

> Hi Ian,
>
> Again. Thanks for the reply. Problem here is we see go process is in
> defunt process and sure parent process did not get SIGCHILD and looking
> deeper,
> I see a thread in  futex_wait_queue_me. If we think we are just getting
> the stack trace and the go process actually got killed, why would I see
> associated fd's in file table and fd table is still intact (see lsof
> information)
>
> Process which is in defunt state which got panic is <87548>, checking for
> threads in this which is 87548
>
> bash-4.2# cat /proc/*87548*/status
>  Name: replicator
>  State: Z (zombie)
>
> bash-4.2# ls -Fl /proc/*87548*/task/*87561*/fd | grep 606649
> l-wx--. 1 root root 64 Aug 25 10:59 1 -> pipe:[606649]
> l-wx--. 1 root root 64 Aug 25 10:59 2 -> pipe:[606649]
>
> Listing the threads
>
> bash-4.2# ps -aefT | grep 87548
> root 87548 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
> root 87548 87561 87507 0 Aug23 ? 00:00:00 [replicator] 
> root 112448 112448 42566 0 17:13 pts/0 00:00:00 grep 87548
>
> bash-4.2# lsof | grep 606649
> replicato  87548  87561root1w FIFO   0,11
> 0t0 606649 pipe
> replicato  87548  87561root2w FIFO   0,11
> 0t0 606649 pipe
>
> Why does lsof show the entry for the FIFO file of this process?
>
> So I feel we have a scenario the thread which is sleeping on
> futex_wait_queue_me is not cleanup during panic() and causing the main
> thread to be exited leaving detached thread which waiting in
> futex_wait_queue_me is still present.
>
> The main issue is I am not able to reproduce this, since this go process
> is very big.
>
> Any way to verify this OR  take it further.
>
> Thanks & Regards,
> Uday Kiran
> On Monday, September 7, 2020 at 12:05:05 PM UTC-7 Ian Lance Taylor wrote:
>
>> On Mon, Sep 7, 2020 at 12:03 AM Uday Kiran Jonnala 
>> wrote:
>> >
>> > Thanks for the reply, I get the point on zombie, I do not think the
>> issue here is parent not reaping child, seems like go process has not
>> finished execution of some
>> > internal threads (waiting on some futex) and causing SIGCHILD not to be
>> sent to parent.
>> >
>> > go process named  hit with panic and I see this went into
>> zombie state
>> >
>> > $ ps -ef | grep replicator
>> > root 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
>> >
>> > Now looking at the tasks within the process
>> >
>> > I see the stack trace of the threads within the process still stuck on
>> following
>> >
>> > bash-4.2# cat /proc/87548/task/87561/stack
>> > [] futex_wait_queue_me+0xc4/0x120
>> > [] futex_wait+0x10a/0x250
>> > [] do_futex+0x35e/0x5b0
>> > [] SyS_futex+0x13b/0x180
>> > [] do_syscall_64+0x79/0x1b0
>> > [] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
>> > [] 0x
>> >
>> > From the above example if we are creating some internal threads and
>> main thread is excited due to panic and left some detached threads, process
>> will be in zombie state until the threads
>> > within the process completes.
>> >
>> > It appears there is some run away threads hung state scenario causing
>> this. I am not able to reproduce it with main go routine explict panic and
>> some go routine still executing.
>> >
>> > Does the above stack trace sound familiar wrt internal threads of Go
>> runtime ?
>>
>> If the process is defunct, then none of the thread stacks matter.
>> They are just where the thread happened to be when the process exited.
>>
>> What is the real problem you are seeing?
>>
>> Ian
>>
>>
>>
>>
>> > On Thursday, August 27, 2020 at 1:43:39 PM UTC-7 Ian Lance Taylor
>> wrote:
>> >>
>> >> On Thu, Aug 27, 2020 at 10:01 AM Uday Kiran Jonnala
>> >>  wrote:
>> >> >
>> >> > I have a situation on zombie parent scenario with golang
>> >> >
>> >> > A process (in the case replicator) has many goroutines internally
>> >> >
>> >> > We hit into panic() and I see the replicator process is in Zombie
>> state
>> >> >
>> >> > <<>>>:~$ ps -ef | grep replicator
>> >> >
>> >> > root 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
>> >> >
>> >> >
>> >> >
>> >> > Main go routine (or the supporting P) excited, but panic left the
>> other P thread to be still in executing state (main P could be 87548 and
>> supporting P thread 87561 is still there) in blocked state
>> >> >
>> >>

Re: [go-nuts] Running goanalyzer on golang process running inside kubernetes pod

[robert engels]

goanalyzer = go tool trace

The options to goanalyzer are the same as ‘go tool trace’ - the usage message 
is misleading in this way.

By ‘map’ I mean expose the pprof port as you would any other port, and yes you 
use the net/http/pprof to start the internal webserver - you don’t need to 
create your own.



> On Sep 10, 2020, at 6:38 PM, Siddhesh Divekar  
> wrote:
> 
> Hi Robert,
> 
> Laying down the steps to make sure I understood it correctly.
> 
> You can map the port 
> What did you mean by above ? 
> 
> but might be easier to capture to a file via code and use goanalyzer on the 
> file
> In this case I will do the following.
> a) Set up a webserver in my program for getting Go profiles (with import _ 
> "net/http/pprof")
> b) map a port from outside k8s so that i can run curl 
> localhost:$PORT/debug/pprof/$PROFILE_TYPE to save a profile (say profile-file)
> c) Use go tool pprof/goanalyzer to analyze given profile from the profile-file
> 
> I would prefer to run this outside my process as the issue we expect to 
> happen is when my program is hung.
> I am not sure if the http server would respond & generate profile files.
> 
> From the help of goanalyzer it's not very clear how I would pass the 
> profile-file to it.
> It's showing how to generate profile-file using go test/tool but not how to 
> pass them to goanalyzer.
> 
> # ./goanalyzer -h
> Usage of 'go tool trace':
> Given a trace file produced by 'go test':
> go test -trace=trace.out pkg
> 
> Open a web browser displaying trace:
> go tool trace [flags] [pkg.test] trace.out
> 
> Generate a pprof-like profile from the trace:
> go tool trace -pprof=TYPE [pkg.test] trace.out
> 
> [pkg.test] argument is required for traces produced by Go 1.6 and below.
> Go 1.7 does not require the binary argument.
> 
> Supported profile types are:
> - net: network blocking profile
> - sync: synchronization blocking profile
> - syscall: syscall blocking profile
> - sched: scheduler latency profile
> 
> Flags:
> -http=addr: HTTP service address (e.g., ':6060')
> -pprof=type: print a pprof-like profile instead
> -d: print debug info such as parsed events
>  
> 
> On Thu, Sep 10, 2020 at 3:22 PM Robert Engels  > wrote:
> You can map the port but might be easier to capture to a file via code and 
> use goanalyzer on the file. 
> 
>> On Sep 10, 2020, at 4:53 PM, Siddhesh Divekar > > wrote:
>> 
>> 
>> Hi,
>> 
>> Has anyone tried running goanalyzer on golang process running inside k8s pod.
>> 
>> If so can you point me to the steps.
>> 
>> -- 
>> -Siddhesh.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com
>>  
>> .
> 
> 
> -- 
> -Siddhesh.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C7C0ED08-7809-47DC-9B4F-3D91E827D7F4%40ix.netcom.com.

Re: [go-nuts] zombie parent scenario with golang

[Uday Kiran Jonnala]

Hi Ian,

Again. Thanks for the reply. Problem here is we see go process is in defunt 
process and sure parent process did not get SIGCHILD and looking deeper,
I see a thread in  futex_wait_queue_me. If we think we are just getting the 
stack trace and the go process actually got killed, why would I see
associated fd's in file table and fd table is still intact (see lsof 
information)

Process which is in defunt state which got panic is <87548>, checking for 
threads in this which is 87548

bash-4.2# cat /proc/*87548*/status 
 Name: replicator 
 State: Z (zombie) 

bash-4.2# ls -Fl /proc/*87548*/task/*87561*/fd | grep 606649 
l-wx--. 1 root root 64 Aug 25 10:59 1 -> pipe:[606649] 
l-wx--. 1 root root 64 Aug 25 10:59 2 -> pipe:[606649]  

Listing the threads

bash-4.2# ps -aefT | grep 87548 
root 87548 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
root 87548 87561 87507 0 Aug23 ? 00:00:00 [replicator]  
root 112448 112448 42566 0 17:13 pts/0 00:00:00 grep 87548  

bash-4.2# lsof | grep 606649 
replicato  87548  87561root1w FIFO   0,11   0t0 
606649 pipe 
replicato  87548  87561root2w FIFO   0,11   0t0 
606649 pipe  

Why does lsof show the entry for the FIFO file of this process?

So I feel we have a scenario the thread which is sleeping on 
futex_wait_queue_me is not cleanup during panic() and causing the main
thread to be exited leaving detached thread which waiting in 
futex_wait_queue_me is still present.

The main issue is I am not able to reproduce this, since this go process is 
very big.

Any way to verify this OR  take it further.

Thanks & Regards,
Uday Kiran
On Monday, September 7, 2020 at 12:05:05 PM UTC-7 Ian Lance Taylor wrote:

> On Mon, Sep 7, 2020 at 12:03 AM Uday Kiran Jonnala  
> wrote:
> >
> > Thanks for the reply, I get the point on zombie, I do not think the 
> issue here is parent not reaping child, seems like go process has not 
> finished execution of some
> > internal threads (waiting on some futex) and causing SIGCHILD not to be 
> sent to parent.
> >
> > go process named  hit with panic and I see this went into 
> zombie state
> >
> > $ ps -ef | grep replicator
> > root 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
> >
> > Now looking at the tasks within the process
> >
> > I see the stack trace of the threads within the process still stuck on 
> following
> >
> > bash-4.2# cat /proc/87548/task/87561/stack
> > [] futex_wait_queue_me+0xc4/0x120
> > [] futex_wait+0x10a/0x250
> > [] do_futex+0x35e/0x5b0
> > [] SyS_futex+0x13b/0x180
> > [] do_syscall_64+0x79/0x1b0
> > [] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> > [] 0x
> >
> > From the above example if we are creating some internal threads and main 
> thread is excited due to panic and left some detached threads, process will 
> be in zombie state until the threads
> > within the process completes.
> >
> > It appears there is some run away threads hung state scenario causing 
> this. I am not able to reproduce it with main go routine explict panic and 
> some go routine still executing.
> >
> > Does the above stack trace sound familiar wrt internal threads of Go 
> runtime ?
>
> If the process is defunct, then none of the thread stacks matter.
> They are just where the thread happened to be when the process exited.
>
> What is the real problem you are seeing?
>
> Ian
>
>
>
>
> > On Thursday, August 27, 2020 at 1:43:39 PM UTC-7 Ian Lance Taylor wrote:
> >>
> >> On Thu, Aug 27, 2020 at 10:01 AM Uday Kiran Jonnala
> >>  wrote:
> >> >
> >> > I have a situation on zombie parent scenario with golang
> >> >
> >> > A process (in the case replicator) has many goroutines internally
> >> >
> >> > We hit into panic() and I see the replicator process is in Zombie 
> state
> >> >
> >> > <<>>>:~$ ps -ef | grep replicator
> >> >
> >> > root 87548 87507 0 Aug23 ? 00:00:00 [replicator] 
> >> >
> >> >
> >> >
> >> > Main go routine (or the supporting P) excited, but panic left the 
> other P thread to be still in executing state (main P could be 87548 and 
> supporting P thread 87561 is still there) in blocked state
> >> >
> >> > bash-4.2# ls -Fl /proc/87548/task/87561/fd | grep 606649l-wx--. 1 
> root root 64 Aug 25 10:59 1 -> pipe:[606649]l-wx--. 1 root root 64 Aug 
> 25 10:59 2 -> pipe:[606649]
> >> >
> >> > Stack trace
> >> >
> >> > bash-4.2# cat /proc/87548/task/87561/stack[] 
> futex_wait_queue_me+0xc4/0x120[] 
> futex_wait+0x10a/0x250[] 
> do_futex+0x35e/0x5b0[] 
> SyS_futex+0x13b/0x180[] 
> do_syscall_64+0x79/0x1b0[] 
> entry_SYSCALL_64_after_hwframe+0x3d/0xa2[] 
> 0x
> >> >
> >> >
> >> >
> >> > We have panic internally from main go routine
> >> >
> >> > fatal error: concurrent map writes
> >> >
> >> > goroutine 666359 [running]:
> >> > runtime.throw(0x101d6ae, 0x15)
> >> > 
> /home/ll/ntnx/toolchain-builds/78ae837ba07c8ef8f0ea782407d8d4626815552b.x86_64/go/src/runtime/panic.go:608
>  
> +0x72 fp=0xc00374b6f0 sp=0xc00374b6c0 pc=0x42da62

Re: [go-nuts] alloc vs totalalloc in memory

[Alexander Mills]

I stole the memory code from here:
https://golangcode.com/print-the-current-memory-usage/

On Thu, Sep 10, 2020 at 4:39 PM Alexander Mills
 wrote:
>
> I have this helper func to print memory usage (detect a memory leak?)
>
>
> func PrintMemUsage() {
> var m runtime.MemStats
> runtime.ReadMemStats(&m)
> // For info on each, see: https://golang.org/pkg/runtime/#MemStats
> fmt.Printf("Alloc = %v MiB", bToMb(m.Alloc))
> fmt.Printf("\tTotalAlloc = %v MiB", bToMb(m.TotalAlloc))
> fmt.Printf("\tSys = %v MiB", bToMb(m.Sys))
> fmt.Printf("\tNumGC = %v\n", m.NumGC)
> }
>
> I am seeing output like this:
> Alloc = 2481 MiB TotalAlloc = 24038 MiB Sys = 2752 MiB NumGC = 114
>
> what is the difference between Alloc and TotalAlloc .. ?
> thx
>
> --
> You received this message because you are subscribed to a topic in the Google 
> Groups "golang-nuts" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/golang-nuts/zLxLN03h6LQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/20f0b65f-a420-4ab1-9a44-57e7414c7bd5n%40googlegroups.com.



-- 
Alexander D. Mills
New cell phone # (415)730-1805
linkedin.com/in/alexanderdmills

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BKyZp40b7waZymt%3DVw6Wd%2BVzUOG9RCDckzoYb7OLNOKPnWAEg%40mail.gmail.com.

[go-nuts] alloc vs totalalloc in memory

[Alexander Mills]

I have this helper func to print memory usage (detect a memory leak?)


func PrintMemUsage() {
var m runtime.MemStats
runtime.ReadMemStats(&m)
// For info on each, see: https://golang.org/pkg/runtime/#MemStats
fmt.Printf("Alloc = %v MiB", bToMb(m.Alloc))
fmt.Printf("\tTotalAlloc = %v MiB", bToMb(m.TotalAlloc))
fmt.Printf("\tSys = %v MiB", bToMb(m.Sys))
fmt.Printf("\tNumGC = %v\n", m.NumGC)
}

I am seeing output like this:
Alloc = 2481 MiB TotalAlloc = 24038 MiB Sys = 2752 MiB NumGC = 114

what is the difference between Alloc and TotalAlloc .. ?
thx

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20f0b65f-a420-4ab1-9a44-57e7414c7bd5n%40googlegroups.com.

Re: [go-nuts] Running goanalyzer on golang process running inside kubernetes pod

[Siddhesh Divekar]

Hi Robert,

Laying down the steps to make sure I understood it correctly.

You can map the port
>
What did you mean by above ?

but might be easier to capture to a file via code and use goanalyzer on the
> file
>
In this case I will do the following.
a) Set up a webserver in my program for getting Go profiles (with import _
"net/http/pprof")
b) map a port from outside k8s so that i can run curl
localhost:$PORT/debug/pprof/$PROFILE_TYPE to save a profile (say
profile-file)
c) Use go tool pprof/goanalyzer to analyze given profile from the
profile-file

I would prefer to run this outside my process as the issue we expect to
happen is when my program is hung.
I am not sure if the http server would respond & generate profile files.

>From the help of goanalyzer it's not very clear how I would pass the
profile-file to it.
It's showing how to generate profile-file using go test/tool but not how to
pass them to goanalyzer.

# ./goanalyzer -h
Usage of 'go tool trace':
Given a trace file produced by 'go test':
go test -trace=trace.out pkg

Open a web browser displaying trace:
go tool trace [flags] [pkg.test] trace.out

Generate a pprof-like profile from the trace:
go tool trace -pprof=TYPE [pkg.test] trace.out

[pkg.test] argument is required for traces produced by Go 1.6 and below.
Go 1.7 does not require the binary argument.

Supported profile types are:
- net: network blocking profile
- sync: synchronization blocking profile
- syscall: syscall blocking profile
- sched: scheduler latency profile

Flags:
-http=addr: HTTP service address (e.g., ':6060')
-pprof=type: print a pprof-like profile instead
-d: print debug info such as parsed events


On Thu, Sep 10, 2020 at 3:22 PM Robert Engels  wrote:

> You can map the port but might be easier to capture to a file via code and
> use goanalyzer on the file.
>
> On Sep 10, 2020, at 4:53 PM, Siddhesh Divekar 
> wrote:
>
> 
> Hi,
>
> Has anyone tried running goanalyzer on golang process running inside k8s
> pod.
>
> If so can you point me to the steps.
>
> --
> -Siddhesh.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com
> 
> .
>
>

-- 
-Siddhesh.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2Biwf21YOz%3Dbv-ej9qwYzdATFoHSjC_%3D_80CPm-zXqm%3DiQ%40mail.gmail.com.

Re: [go-nuts] godoc-static - Generate static documentation for your packages

[Trevor Slocum]

godoc-static v0.1.8 is now available featuring improved support for Go 
modules.

On Wednesday, February 26, 2020 at 3:45:49 PM UTC-8 Trevor Slocum wrote:

> I've had some time to polish this project somewhat. It should be fairly 
> usable.
>
> I've added ZIP archive generation to allow browsing offline without godoc.
>
> Any additional feedback is welcome.
>
> On Monday, February 10, 2020 at 8:22:51 AM UTC-8, Trevor Slocum wrote:
>>
>> Hey Nick, I did consider that and hope to still visit it.  I chose to 
>> scrape instead because the output requires only minimal alteration to 
>> achieve the desired result. Example code expansion is rewritten to not 
>> require JavaScript by using  and  and not much more.  
>> Large changes to output from the godoc tool can certainly break 
>> godoc-static in its current state.  Assistance with exploring generating 
>> similar output without using godoc would be appreciated.
>>
>>
>> On Monday, February 10, 2020 at 7:14:51 AM UTC-8, Nick wrote:
>>>
>>> Quoth tsl...@gmail.com: 
>>> > Inspired by the recent news of godoc.org shutting down I have created 
>>> > godoc-static.  While we could all migrate to pkg.go.dev 
>>> ,
>>>  
>>> we could also host and 
>>> > update our documentation ourselves. 
>>>
>>> Cool, sounds interesting, I like the idea of more tools to make more 
>>> distributed hosting of go stuff easier! 
>>>
>>> > When invoked, "godoc -http=localhost:" is started, relevant 
>>> package 
>>> > documentation pages and source code is scraped and rewritten to fix 
>>> styling, 
>>> > expanding content, etc., then godoc is terminated. 
>>>
>>> Interesting approach. Did you consider taking the code that godoc 
>>> uses (either direct copying or including it somehow) and generating 
>>> the HTML directly? I can imagine that might not be straightforward, 
>>> but I'm interested to hear why you chose this approach instead.   
>>> Without having looked at the code, I would imagine it would be more 
>>> liable to break with future versions of godoc which change the 
>>> output. 
>>>
>>> Anyway, good job, I may use it myself soon, and will let you know 
>>> how I get on. 
>>>
>>> Nick 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/59d45342-c8ae-4dc7-8b2d-96910b9906d8n%40googlegroups.com.

Re: [go-nuts] why is wait group necessary here?

[burak serdar]

On Thu, Sep 10, 2020 at 4:43 PM Alexander Mills
 wrote:
>
> It was behaving strangely before adding the waitgroup, my script was
> getting stuck

Waitgroup is unnecessary. There's a few problems: when you detect an
error you should stop writing, but the program "continue"s. You set
the content-type to json, but this is technically not valid json.

If it was getting stuck, then it may help to find out where it is getting stuck.

>
> On Thu, Sep 10, 2020 at 11:14 AM burak serdar  wrote:
> >
> > On Thu, Sep 10, 2020 at 11:46 AM Alexander Mills
> >  wrote:
> > >
> > > I tried writing this http post request which streams the request body 
> > > line-by-line as JSON.
> > > (This is for an ElasticSearch bulk api request, newline separate JSON in 
> > > the request body). I didn't think the waitgroup was necessary, but 
> > > apparently the waitgroup is necessary. Does anyone know why? I assume the 
> > > need for the waitgroup might represent a bug in my code.
> >
> >
> > What makes you think waitgroup is necessary?
> >
> >
> >
> > >
> > > Here is the same code in a github gist: 
> > > https://gist.github.com/ORESoftware/06341057ca39e1b2a47dc8a8e4e4b55f
> > >
> > >
> > > func doBulkUpsertWithResponse(baseUrl string, z []UpsertDoc) 
> > > (*map[string]interface{}, error) {
> > >
> > > var m *map[string]interface{}
> > >
> > > if len(z) < 1 {
> > > return m, nil
> > > }
> > >
> > > rd, wr := io.Pipe() // we stream json lines to post request
> > >
> > > wg := sync.WaitGroup{}
> > >
> > > wg.Add(1)
> > > go func() {
> > >
> > > defer func() {
> > > // close the stream after writing is finished
> > > wg.Done()
> > > wr.Close();
> > > }()
> > >
> > > for _, v := range z {
> > >
> > > var doc = ElasticBulkUpsertable{
> > > &v.Doc, true,
> > > }
> > >
> > > jsonStr, err := json.Marshal(&doc)
> > >
> > > if err != nil {
> > > continue
> > > }
> > >
> > > updateLine := fmt.Sprintf(`{"update":{"_id":"%s"}}`, v.Id) + "\n"
> > > if _, err := wr.Write([]byte(updateLine)); err != nil {
> > >   continue
> > > }
> > >
> > > docLine := string(jsonStr) + "\n"
> > > if _, err := wr.Write([]byte(docLine)); err != nil {
> > > continue
> > > }
> > > }
> > >
> > > }()
> > >
> > > fullUrl := fmt.Sprintf("%s/%s", baseUrl, "_bulk")
> > > req, err := http.NewRequest("POST", fullUrl, rd)
> > >
> > > if err != nil {
> > > log.Println(fmt.Errorf("cm:3ec045d3-4790-4d7a-aab8-65949933d263: '%v'", 
> > > err))
> > > return nil, err
> > > }
> > >
> > > req.Header.Set("Content-Type", "application/json")
> > >
> > > resp, err := client.Do(req)
> > >
> > > wg.Wait()
> > >
> > > if err != nil {
> > > return m, err
> > > }
> > >
> > > defer func() {
> > > resp.Body.Close()
> > > }()
> > >
> > > if resp.StatusCode != 200 {
> > >   return m, errors.New("request failed with status code: " + resp.Status)
> > > }
> > >
> > > if err := json.NewDecoder(resp.Body).Decode(&m); err != nil {
> > > return nil, err
> > > }
> > >
> > > return m, nil
> > >
> > > }
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups 
> > > "golang-nuts" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an 
> > > email to golang-nuts+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit 
> > > https://groups.google.com/d/msgid/golang-nuts/4d6b60cd-4b9e-4bce-aa6f-2d2bfbefbd98n%40googlegroups.com.
>
>
>
> --
> Alexander D. Mills
> New cell phone # (415)730-1805
> linkedin.com/in/alexanderdmills

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMV2Rqo5ZJZOz32m%3DJ0iHdWLM4kHhqc0ECwZ6hXV1EzT9%3DgqZg%40mail.gmail.com.

Re: [go-nuts] add method to interface via reflect

[Kurtis Rader]

On Thu, Sep 10, 2020 at 3:44 PM Vasiliy Tolstov  wrote:

> Hi! Does go support adding method to exiting interface via reflect?
> Mostly I need to add method to struct pointer.
>

No, and this sounds like an XY problem . It would
help if you described the problem rather than the solution.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD9Q%3Dwm%2Bot58VrxWPeg2j7gMsvuiTTfEvKNPzHjCuqZZuw%40mail.gmail.com.

[go-nuts] add method to interface via reflect

[Vasiliy Tolstov]

Hi! Does go support adding method to exiting interface via reflect?
Mostly I need to add method to struct pointer.

-- 
Vasiliy Tolstov,
e-mail: v.tols...@selfip.ru

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CACaajQs62OAkU7OpFpQz9n3zZQPnzrcrAsWtFL78fgjfPv_%3Dkw%40mail.gmail.com.

Re: [go-nuts] why is wait group necessary here?

[Alexander Mills]

It was behaving strangely before adding the waitgroup, my script was
getting stuck

On Thu, Sep 10, 2020 at 11:14 AM burak serdar  wrote:
>
> On Thu, Sep 10, 2020 at 11:46 AM Alexander Mills
>  wrote:
> >
> > I tried writing this http post request which streams the request body 
> > line-by-line as JSON.
> > (This is for an ElasticSearch bulk api request, newline separate JSON in 
> > the request body). I didn't think the waitgroup was necessary, but 
> > apparently the waitgroup is necessary. Does anyone know why? I assume the 
> > need for the waitgroup might represent a bug in my code.
>
>
> What makes you think waitgroup is necessary?
>
>
>
> >
> > Here is the same code in a github gist: 
> > https://gist.github.com/ORESoftware/06341057ca39e1b2a47dc8a8e4e4b55f
> >
> >
> > func doBulkUpsertWithResponse(baseUrl string, z []UpsertDoc) 
> > (*map[string]interface{}, error) {
> >
> > var m *map[string]interface{}
> >
> > if len(z) < 1 {
> > return m, nil
> > }
> >
> > rd, wr := io.Pipe() // we stream json lines to post request
> >
> > wg := sync.WaitGroup{}
> >
> > wg.Add(1)
> > go func() {
> >
> > defer func() {
> > // close the stream after writing is finished
> > wg.Done()
> > wr.Close();
> > }()
> >
> > for _, v := range z {
> >
> > var doc = ElasticBulkUpsertable{
> > &v.Doc, true,
> > }
> >
> > jsonStr, err := json.Marshal(&doc)
> >
> > if err != nil {
> > continue
> > }
> >
> > updateLine := fmt.Sprintf(`{"update":{"_id":"%s"}}`, v.Id) + "\n"
> > if _, err := wr.Write([]byte(updateLine)); err != nil {
> >   continue
> > }
> >
> > docLine := string(jsonStr) + "\n"
> > if _, err := wr.Write([]byte(docLine)); err != nil {
> > continue
> > }
> > }
> >
> > }()
> >
> > fullUrl := fmt.Sprintf("%s/%s", baseUrl, "_bulk")
> > req, err := http.NewRequest("POST", fullUrl, rd)
> >
> > if err != nil {
> > log.Println(fmt.Errorf("cm:3ec045d3-4790-4d7a-aab8-65949933d263: '%v'", 
> > err))
> > return nil, err
> > }
> >
> > req.Header.Set("Content-Type", "application/json")
> >
> > resp, err := client.Do(req)
> >
> > wg.Wait()
> >
> > if err != nil {
> > return m, err
> > }
> >
> > defer func() {
> > resp.Body.Close()
> > }()
> >
> > if resp.StatusCode != 200 {
> >   return m, errors.New("request failed with status code: " + resp.Status)
> > }
> >
> > if err := json.NewDecoder(resp.Body).Decode(&m); err != nil {
> > return nil, err
> > }
> >
> > return m, nil
> >
> > }
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to golang-nuts+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/golang-nuts/4d6b60cd-4b9e-4bce-aa6f-2d2bfbefbd98n%40googlegroups.com.



-- 
Alexander D. Mills
New cell phone # (415)730-1805
linkedin.com/in/alexanderdmills

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BKyZp7rWF4q5B%3D2eUmkuyAEVsFL%3DX5xcNQ%3DkVsq-_BSPCzbUQ%40mail.gmail.com.

Re: [go-nuts] Running goanalyzer on golang process running inside kubernetes pod

[Robert Engels]

You can map the port but might be easier to capture to a file via code and use 
goanalyzer on the file. 

> On Sep 10, 2020, at 4:53 PM, Siddhesh Divekar  
> wrote:
> 
> 
> Hi,
> 
> Has anyone tried running goanalyzer on golang process running inside k8s pod.
> 
> If so can you point me to the steps.
> 
> -- 
> -Siddhesh.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3A348919-5B47-474B-8DFB-363FF9935D8C%40ix.netcom.com.

[go-nuts] Running goanalyzer on golang process running inside kubernetes pod

[Siddhesh Divekar]

Hi,

Has anyone tried running goanalyzer on golang process running inside k8s
pod.

If so can you point me to the steps.

-- 
-Siddhesh.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2BhUDqMRmE0mDp_gMTKWKT0Be8KgtBxuF2fYAGCs-dNoog%40mail.gmail.com.

Re: [go-nuts] Running golang C-shared library in an OS that uses cooperative scheduling.

[Ian Lance Taylor]

On Wed, Sep 9, 2020 at 11:58 PM Yonatan Gizachew  wrote:
>
> Also, is there a way to print the values of runtime.g?

In what sense?  Code running the runtime package can just print it.
Or if you run the program under gdb, you should be able to "print
'runtime.g'" (you need the quotes because otherwise gdb will think it
is a structure field reference).  At least, gdb works that way on
GNU/Linux, not sure about your operating system.

Ian

> On Thursday, September 10, 2020 at 12:15:04 PM UTC+9 Ian Lance Taylor wrote:
>>
>> On Wed, Sep 9, 2020 at 7:39 PM Yonatan Gizachew  wrote:
>> >
>> > This might be trivial, but could you explain me the relationship between 
>> > the following?
>> > 1. value returned by __tls_get_addr
>>
>> This is the address of the TLS control block. The exact definition
>> depends on the architecture. All TLS variables are at some offset
>> from this value.
>>
>> > 2. runtime.g
>>
>> A TLS variable that holds the current goroutine, a pointer to a g struct.
>>
>> > 3. runtime.g0
>>
>> A global variable that holds the g struct used for the initial goroutine.
>>
>> > 4. runtime.m0
>>
>> A global variable that holds the m struct used for the initial thread.
>>
>> > 5. runtime.m
>>
>> gccgo no longer uses this. In earlier releases it was a TLS variable
>> that holds the current thread, a pointer to an m struct. In current
>> releases gccgo uses g->m for this.
>>
>> Ian
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/0935d7c3-d2de-49cd-8d25-196b276c5ca9n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcW1KzAdxYxfv4Ge5zFoioO3XTmDwVLeCxaC5dxobavkDw%40mail.gmail.com.

Re: [go-nuts] Running golang C-shared library in an OS that uses cooperative scheduling.

[Ian Lance Taylor]

On Wed, Sep 9, 2020 at 11:55 PM Yonatan Gizachew  wrote:
>
> Thank you for the explanation. For example, please have a look at the this bt.
> __tls_get_addr is called after runtime.getg. So, what does __tls_get_addr 
> return at this time, a pointer to runtime.g or runtime.g itself?

If I recall correctly, __tls_get_addr (which is of course a general
utility that has nothing particularly to do with Go) will return the
address of runtime.g for the thread that calls it.

Ian



> On Thursday, September 10, 2020 at 12:15:04 PM UTC+9 Ian Lance Taylor wrote:
>>
>> On Wed, Sep 9, 2020 at 7:39 PM Yonatan Gizachew  wrote:
>> >
>> > This might be trivial, but could you explain me the relationship between 
>> > the following?
>> > 1. value returned by __tls_get_addr
>>
>> This is the address of the TLS control block. The exact definition
>> depends on the architecture. All TLS variables are at some offset
>> from this value.
>>
>> > 2. runtime.g
>>
>> A TLS variable that holds the current goroutine, a pointer to a g struct.
>>
>> > 3. runtime.g0
>>
>> A global variable that holds the g struct used for the initial goroutine.
>>
>> > 4. runtime.m0
>>
>> A global variable that holds the m struct used for the initial thread.
>>
>> > 5. runtime.m
>>
>> gccgo no longer uses this. In earlier releases it was a TLS variable
>> that holds the current thread, a pointer to an m struct. In current
>> releases gccgo uses g->m for this.
>>
>> Ian
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/20dd490b-f972-4af7-bf4c-f2ab8188c5c1n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXTEX8i6j6qdcr4jEdOo%2Beix8zoepn4by8nuO%2BirqXF8w%40mail.gmail.com.

Re: [go-nuts] why is wait group necessary here?

[burak serdar]

On Thu, Sep 10, 2020 at 11:46 AM Alexander Mills
 wrote:
>
> I tried writing this http post request which streams the request body 
> line-by-line as JSON.
> (This is for an ElasticSearch bulk api request, newline separate JSON in the 
> request body). I didn't think the waitgroup was necessary, but apparently the 
> waitgroup is necessary. Does anyone know why? I assume the need for the 
> waitgroup might represent a bug in my code.


What makes you think waitgroup is necessary?



>
> Here is the same code in a github gist: 
> https://gist.github.com/ORESoftware/06341057ca39e1b2a47dc8a8e4e4b55f
>
>
> func doBulkUpsertWithResponse(baseUrl string, z []UpsertDoc) 
> (*map[string]interface{}, error) {
>
> var m *map[string]interface{}
>
> if len(z) < 1 {
> return m, nil
> }
>
> rd, wr := io.Pipe() // we stream json lines to post request
>
> wg := sync.WaitGroup{}
>
> wg.Add(1)
> go func() {
>
> defer func() {
> // close the stream after writing is finished
> wg.Done()
> wr.Close();
> }()
>
> for _, v := range z {
>
> var doc = ElasticBulkUpsertable{
> &v.Doc, true,
> }
>
> jsonStr, err := json.Marshal(&doc)
>
> if err != nil {
> continue
> }
>
> updateLine := fmt.Sprintf(`{"update":{"_id":"%s"}}`, v.Id) + "\n"
> if _, err := wr.Write([]byte(updateLine)); err != nil {
>   continue
> }
>
> docLine := string(jsonStr) + "\n"
> if _, err := wr.Write([]byte(docLine)); err != nil {
> continue
> }
> }
>
> }()
>
> fullUrl := fmt.Sprintf("%s/%s", baseUrl, "_bulk")
> req, err := http.NewRequest("POST", fullUrl, rd)
>
> if err != nil {
> log.Println(fmt.Errorf("cm:3ec045d3-4790-4d7a-aab8-65949933d263: '%v'", err))
> return nil, err
> }
>
> req.Header.Set("Content-Type", "application/json")
>
> resp, err := client.Do(req)
>
> wg.Wait()
>
> if err != nil {
> return m, err
> }
>
> defer func() {
> resp.Body.Close()
> }()
>
> if resp.StatusCode != 200 {
>   return m, errors.New("request failed with status code: " + resp.Status)
> }
>
> if err := json.NewDecoder(resp.Body).Decode(&m); err != nil {
> return nil, err
> }
>
> return m, nil
>
> }
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/4d6b60cd-4b9e-4bce-aa6f-2d2bfbefbd98n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMV2RqpUQ3rZS0YBADG%3DdWz%3DVrQ77vP%2BrWuvAYz_bDyQd8Vrug%40mail.gmail.com.

[go-nuts] why is wait group necessary in streaming post request to elasticsearch

[Alexander Mills]

I am trying to figure out why the wg is seemingly necessary as part of this 
code to stream JSON to a post request to ElasticSearch:

https://gist.github.com/ORESoftware/06341057ca39e1b2a47dc8a8e4e4b55f

Any tips appreciated. If the waitgroup is not used, it doesn't appear to 
wait until the full request body is received? Not sure.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6d7974ce-8c83-45b6-a5c7-10a2d9185bdbn%40googlegroups.com.

[go-nuts] why is wait group necessary here?

[Alexander Mills]

I tried writing this http post request which streams the request body 
line-by-line as JSON.
(This is for an ElasticSearch bulk api request, newline separate JSON in 
the request body). I didn't think the waitgroup was necessary, but 
apparently the waitgroup is necessary. Does anyone know why? I assume the 
need for the waitgroup might represent a bug in my code.

Here is the same code in a github 
gist: https://gist.github.com/ORESoftware/06341057ca39e1b2a47dc8a8e4e4b55f


func doBulkUpsertWithResponse(baseUrl string, z []UpsertDoc) 
(*map[string]interface{}, error) {

var m *map[string]interface{}

if len(z) < 1 {
return m, nil
}

rd, wr := io.Pipe() // we stream json lines to post request

wg := sync.WaitGroup{}

wg.Add(1)
go func() {

defer func() {
// close the stream after writing is finished
wg.Done()
wr.Close();
}()

for _, v := range z {

var doc = ElasticBulkUpsertable{
&v.Doc, true,
}

jsonStr, err := json.Marshal(&doc)

if err != nil {
continue
}

updateLine := fmt.Sprintf(`{"update":{"_id":"%s"}}`, v.Id) + "\n"
if _, err := wr.Write([]byte(updateLine)); err != nil {
  continue
}

docLine := string(jsonStr) + "\n"
if _, err := wr.Write([]byte(docLine)); err != nil {
continue
}
}

}()

fullUrl := fmt.Sprintf("%s/%s", baseUrl, "_bulk")
req, err := http.NewRequest("POST", fullUrl, rd)

if err != nil {
log.Println(fmt.Errorf("cm:3ec045d3-4790-4d7a-aab8-65949933d263: '%v'", 
err))
return nil, err
}

req.Header.Set("Content-Type", "application/json")

resp, err := client.Do(req)

wg.Wait()

if err != nil {
return m, err
}

defer func() {
resp.Body.Close()
}()

if resp.StatusCode != 200 {
  return m, errors.New("request failed with status code: " + resp.Status)
}

if err := json.NewDecoder(resp.Body).Decode(&m); err != nil {
return nil, err
}

return m, nil

}

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4d6b60cd-4b9e-4bce-aa6f-2d2bfbefbd98n%40googlegroups.com.

Re: [go-nuts] [generics] Allowing interfaces with type constraints

[Viktor Kojouharov]

I see,

I guess the compiler error message needs improvement here. It could mention 
that interfaces with type lists cannot be used.

On Thursday, September 10, 2020 at 5:17:57 PM UTC+2 rog wrote:

> You can definitely use parametric interfaces in type assertion 
> expressions, but you can't use interfaces with type lists.
> I'm not sure exactly what you're trying to do in your example, but ISTM 
> that it could be somewhat simpler. You
> only need one type parameter: https://go2goplay.golang.org/p/4E0ZnVxJwL9
>
>
> On Thu, 10 Sep 2020 at 15:54, Viktor Kojouharov  
> wrote:
>
>>
>> Would using parametric interfaces be allowed in type assertion 
>> expressions / type switches?
>>
>> The current behavior in the go2go playground is for the compiler to 
>> return the error: "interface contains type constraints (T)" (
>> https://go2goplay.golang.org/p/O1Un2Vm9Dh0)
>> I was under the impression that since the outer function is already 
>> parametrised with the same type, this should work without a problem.
>>
>> Perhaps I'm missing something, but I currently don't see how one can 
>> check for optional interface implementations of generic types, since this 
>> error will pretty much stop them.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/fbe50aaf-1794-4a17-9321-2b923f546d13n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/10d006bf-2fa7-4d09-bcfc-1b9c0e0eb525n%40googlegroups.com.

Re: [go-nuts] [generics] Allowing interfaces with type constraints

[roger peppe]

You can definitely use parametric interfaces in type assertion expressions,
but you can't use interfaces with type lists.
I'm not sure exactly what you're trying to do in your example, but ISTM
that it could be somewhat simpler. You
only need one type parameter: https://go2goplay.golang.org/p/4E0ZnVxJwL9


On Thu, 10 Sep 2020 at 15:54, Viktor Kojouharov 
wrote:

>
> Would using parametric interfaces be allowed in type assertion expressions
> / type switches?
>
> The current behavior in the go2go playground is for the compiler to return
> the error: "interface contains type constraints (T)" (
> https://go2goplay.golang.org/p/O1Un2Vm9Dh0)
> I was under the impression that since the outer function is already
> parametrised with the same type, this should work without a problem.
>
> Perhaps I'm missing something, but I currently don't see how one can check
> for optional interface implementations of generic types, since this error
> will pretty much stop them.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/fbe50aaf-1794-4a17-9321-2b923f546d13n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAJhgach9vJ_6Tcu8vceMBGyE9L26e20skKQykybuawY7az5Prg%40mail.gmail.com.

Re: [go-nuts] Is there any lib like CMSSignedData in GOLANG?

[liuhanlin]

https://github.com/jack0liu/pbkdf2


发自我的iPhone

> 在 2020年9月10日，上午2:00，Somporn pongpan  写道：
> 
> Can you share the code or any info for decryption with pkcs7?
> 
>> On Wednesday, June 8, 2016 at 7:51:40 AM UTC+7 18126...@163.com wrote:
>> It's great. 
>> 
>> In fact , I have found some code using pkcs7 to decode,  and it works very 
>> well now.
>> 
>> Thank you very much.
>> 
>> 在 2016年6月8日星期三 UTC+8上午1:59:10，Konstantin Khomoutov写道：
>>> On Tue, 7 Jun 2016 01:21:26 -0700 (PDT) 
>>> 18126...@163.com wrote: 
>>> 
>>> > I don't know the server how to encrypt and signed data, but i have 
>>> > the CMSSignedData class in Java to decode data, the CMSSignedData 
>>> > class is from org.bouncycastle.cms.*, it's contained in the jar 
>>> > "bcpkix-jdk15on-1.52.jar". 
>>> > Is there similar lib in GOLANG, then i can use it to decode the 
>>> > server data easily. 
>>> 
>>> Out of curiosity, I have googled for you, and judging from [1], 
>>> [2] might be a way to go. 
>>> 
>>> 1. 
>>> https://www.bouncycastle.org/docs/pkixdocs1.4/org/bouncycastle/cms/CMSSignedData.html
>>>  
>>> 2. https://godoc.org/github.com/fullsailor/pkcs7
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "golang-nuts" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/golang-nuts/nacJKaX_THw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/97a53860-5580-4b8f-a12f-120e8882fd6fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3B6B3C56-D480-44D9-8585-78F3230BDBE0%40163.com.

[go-nuts] [generics] Allowing interfaces with type constraints

[Viktor Kojouharov]

Would using parametric interfaces be allowed in type assertion expressions 
/ type switches?

The current behavior in the go2go playground is for the compiler to return 
the error: "interface contains type constraints (T)" 
(https://go2goplay.golang.org/p/O1Un2Vm9Dh0)
I was under the impression that since the outer function is already 
parametrised with the same type, this should work without a problem.

Perhaps I'm missing something, but I currently don't see how one can check 
for optional interface implementations of generic types, since this error 
will pretty much stop them.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/fbe50aaf-1794-4a17-9321-2b923f546d13n%40googlegroups.com.

Re: [go-nuts] Is there any lib like CMSSignedData in GOLANG?

[liuhanlin]

https://github.com/jack0liu/pbkdf2

if it works，thanks for giving a star

发自我的iPhone

> 在 2020年9月10日，上午2:00，Somporn pongpan  写道：
> 
> Can you share the code or any info for decryption with pkcs7?
> 
>> On Wednesday, June 8, 2016 at 7:51:40 AM UTC+7 18126...@163.com wrote:
>> It's great. 
>> 
>> In fact , I have found some code using pkcs7 to decode,  and it works very 
>> well now.
>> 
>> Thank you very much.
>> 
>> 在 2016年6月8日星期三 UTC+8上午1:59:10，Konstantin Khomoutov写道：
>>> On Tue, 7 Jun 2016 01:21:26 -0700 (PDT) 
>>> 18126...@163.com wrote: 
>>> 
>>> > I don't know the server how to encrypt and signed data, but i have 
>>> > the CMSSignedData class in Java to decode data, the CMSSignedData 
>>> > class is from org.bouncycastle.cms.*, it's contained in the jar 
>>> > "bcpkix-jdk15on-1.52.jar". 
>>> > Is there similar lib in GOLANG, then i can use it to decode the 
>>> > server data easily. 
>>> 
>>> Out of curiosity, I have googled for you, and judging from [1], 
>>> [2] might be a way to go. 
>>> 
>>> 1. 
>>> https://www.bouncycastle.org/docs/pkixdocs1.4/org/bouncycastle/cms/CMSSignedData.html
>>>  
>>> 2. https://godoc.org/github.com/fullsailor/pkcs7
> 
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "golang-nuts" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/golang-nuts/nacJKaX_THw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/97a53860-5580-4b8f-a12f-120e8882fd6fn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/493288F3-7800-4E4B-9599-8B9AEFC07C47%40163.com.

Re: [go-nuts] Workflow and tools for JSON Schema generation

[Paul Jolly]

CUE (https://cuelang.org/) will (in the near future) be able to help
with the Go -> JSON Schema part (or indeed JSON Schema -> Go).

Please feel free to ask any questions of the community via
https://cuelang.org/community/

On Thu, 3 Sep 2020 at 08:05, Johann Höchtl  wrote:
>
>
> Hi,
> I would like to accomplish the following:
>
> An existing golang package on github provides a struct definition I am 
> interested in to be used as part of a REST API interface I design
>
> I would like to automatically go generate json schema for this struct
> I would like to incorporate this generated schema into an OpenAPI definition 
> and generate go boilerplate for that
>
>
> For step 2 I am using https://github.com/deepmap/oapi-codegen which works 
> great. It also supports $ref to external schemas.
>
> Still looking to accomplish step 1: I am searching for a tool which go 
> generates JSON schema from a golang source file or package. Any help?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/f0e099de-7008-478d-ba92-f2febd0c40bdn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CACoUkn4EgxJJc1qA53a7kZVeeXrzb%2BYvU-vgGr%2BgcxJakajhjA%40mail.gmail.com.

Re: [go-nuts] unit test for function which has goroutine

[Gregor Best]

Right, I wanted to write something about the goroutine and forgot about 
it. Sorry about that.


The problem here could be that in the code you posted there's nothing 
waiting on any result from the goroutine, and if the test exits quickly 
enough, it might not get scheduled.


In general, whenever a goroutine is created, you should have some idea 
of when and how it gets to run and when it stops running, and how other 
goroutines wait for it to do its job.


You could do this for example by controlling the lifetime of the 
goroutine with a sync.WaitGroup. Pass in a WaitGroup and call Done() on 
it when the goroutine returns (probably as a deferred call), then call 
Wait() in some other goroutine (for example in your main function or in 
the TestFoo function) so there's some sort of ordering enforced.


In general, you'll need some way of signalling "I'm done copying stuff 
from zipReader to zipWriter" in the goroutine, and you'll need to 
arrange for some other, longer lived, goroutine to wait for that signal.



On 10.09.20 09:34, Yvonne Zhang wrote:

Thank you Gregor for your quick reply!

Anything special I need to think about this go routine logic in my unit 
test. It might be still running after test returns..?


On Thursday, September 10, 2020 at 5:26:13 PM UTC+10 
be...@pferdewetten.de wrote:


If I were you, I'd pass in an interface value to functionA with the
signature

type Ber interface{
functionB(io.Reader, io.Writer)
}

(assuming that zipReader is an io.Reader, zipWriter is an io.Writer,
you
may have to adjust this a bit).

Then make your mock functionB a method on a type that implements the
Ber
interface, the same for the real functionB. In unit tests, pass in an
instance of the type with the mock method, in real life, pass in an
instance of the type with the real method.

Note that the types could be as simple as

type mockB struct{}
func (m mockB) functionB(...)

On 10.09.20 09:15, Yvonne Zhang wrote:
 > Hi,
 > I have a function streaming a zipreader to browser. It is like this.
 > func functionA()(body io.ReadCloser, err error){
 >
 >    // some logic to get a zipreader
 >
 >    body, pipeWriter := io.Pipe()
 >    zipWriter := zip.NewWriter(pipeWriter)
 >
 >    go func(){
 >       // err := functionB(zipReader, zipWriter)call another
function to
 > prepare files and write to zipwriter
 >    If err != nil{
 >        zipWriter.Close()
 > _ = pipeWriter.CloseWithError(err)
 >         return
 >    }
 >       zipWriter.Close()
 >        pipeWriter.Close()
 >    }()
 >    return
 > }
 >
 > My question is about unit test about this functionA. I have mock for
 > functionB. How do I write unit test for functionA with the mock
 > functionB. It seems my unit test does not work well when it goes
into
 > the goroutine.
 > Many thanks!
 >
 > --
 > You received this message because you are subscribed to the Google
 > Groups "golang-nuts" group.
 > To unsubscribe from this group and stop receiving emails from it,
send
 > an email to golang-nuts...@googlegroups.com
 > .
 > To view this discussion on the web visit
 >

https://groups.google.com/d/msgid/golang-nuts/4ca0817a-f1f7-44a9-be3d-3584bcb61b8an%40googlegroups.com

 >

.


-- 
Gregor Best

be...@pferdewetten.de

--
You received this message because you are subscribed to the Google 
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to golang-nuts+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/33648e9a-4011-4d02-a07b-01b506cf5cd1n%40googlegroups.com 
.


--
Gregor Best
  b...@pferdewetten.de

--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/746e54f8-cefc-9fee-c4e0-8724833e38e8%40pferdewetten.de.

Re: [go-nuts] unit test for function which has goroutine

[Yvonne Zhang]

Thank you Gregor for your quick reply!

Anything special I need to think about this go routine logic in my unit 
test. It might be still running after test returns..?

On Thursday, September 10, 2020 at 5:26:13 PM UTC+10 be...@pferdewetten.de 
wrote:

> If I were you, I'd pass in an interface value to functionA with the 
> signature
>
> type Ber interface{
> functionB(io.Reader, io.Writer)
> }
>
> (assuming that zipReader is an io.Reader, zipWriter is an io.Writer, you 
> may have to adjust this a bit).
>
> Then make your mock functionB a method on a type that implements the Ber 
> interface, the same for the real functionB. In unit tests, pass in an 
> instance of the type with the mock method, in real life, pass in an 
> instance of the type with the real method.
>
> Note that the types could be as simple as
>
> type mockB struct{}
> func (m mockB) functionB(...)
>
> On 10.09.20 09:15, Yvonne Zhang wrote:
> > Hi,
> > I have a function streaming a zipreader to browser. It is like this.
> > func functionA()(body io.ReadCloser, err error){
> > 
> >// some logic to get a zipreader
> > 
> >body, pipeWriter := io.Pipe()
> >zipWriter := zip.NewWriter(pipeWriter)
> > 
> >go func(){
> >   // err := functionB(zipReader, zipWriter)call another function to 
> > prepare files and write to zipwriter
> >If err != nil{
> >zipWriter.Close()
> > _ = pipeWriter.CloseWithError(err)
> > return
> >}
> >   zipWriter.Close()
> >pipeWriter.Close()
> >}()
> >return
> > }
> > 
> > My question is about unit test about this functionA. I have mock for 
> > functionB. How do I write unit test for functionA with the mock 
> > functionB. It seems my unit test does not work well when it goes into 
> > the goroutine.
> > Many thanks!
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to golang-nuts...@googlegroups.com 
> > .
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/golang-nuts/4ca0817a-f1f7-44a9-be3d-3584bcb61b8an%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/golang-nuts/4ca0817a-f1f7-44a9-be3d-3584bcb61b8an%40googlegroups.com?utm_medium=email&utm_source=footer
> >.
>
> -- 
> Gregor Best
> be...@pferdewetten.de
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/33648e9a-4011-4d02-a07b-01b506cf5cd1n%40googlegroups.com.

Re: [go-nuts] unit test for function which has goroutine

[Gregor Best]

If I were you, I'd pass in an interface value to functionA with the 
signature


type Ber interface{
functionB(io.Reader, io.Writer)
}

(assuming that zipReader is an io.Reader, zipWriter is an io.Writer, you 
may have to adjust this a bit).


Then make your mock functionB a method on a type that implements the Ber 
interface, the same for the real functionB. In unit tests, pass in an 
instance of the type with the mock method, in real life, pass in an 
instance of the type with the real method.


Note that the types could be as simple as

type mockB struct{}
func (m mockB) functionB(...)

On 10.09.20 09:15, Yvonne Zhang wrote:

Hi,
I have a function streaming a zipreader to browser. It is like this.
func functionA()(body io.ReadCloser, err error){

    // some logic to get a zipreader

    body, pipeWriter := io.Pipe()
    zipWriter := zip.NewWriter(pipeWriter)

    go func(){
       // err := functionB(zipReader, zipWriter)call another function to 
prepare files and write to zipwriter

    If err != nil{
        zipWriter.Close()
 _ = pipeWriter.CloseWithError(err)
         return
    }
       zipWriter.Close()
        pipeWriter.Close()
    }()
    return
}

My question is about unit test about this functionA. I have mock for 
functionB. How do I write unit test for functionA with the mock 
functionB. It seems my unit test does not work well when it goes into 
the goroutine.

Many thanks!

--
You received this message because you are subscribed to the Google 
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to golang-nuts+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4ca0817a-f1f7-44a9-be3d-3584bcb61b8an%40googlegroups.com 
.


--
  Gregor Best
  b...@pferdewetten.de

--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/eaeee7dd-8ff0-4dbf-3b5f-bce7617afa18%40pferdewetten.de.

[go-nuts] unit test for function which has goroutine

[Yvonne Zhang]

Hi,
I have a function streaming a zipreader to browser. It is like this.
func functionA()(body io.ReadCloser, err error){

   // some logic to get a zipreader

   body, pipeWriter := io.Pipe()
   zipWriter := zip.NewWriter(pipeWriter)

   go func(){
  // err := functionB(zipReader, zipWriter)call another function to 
prepare files and write to zipwriter
   If err != nil{
   zipWriter.Close()
_ = pipeWriter.CloseWithError(err)
return
   }
  zipWriter.Close()
   pipeWriter.Close()
   }()
   return 
}

My question is about unit test about this functionA. I have mock for 
functionB. How do I write unit test for functionA with the mock functionB. 
It seems my unit test does not work well when it goes into the goroutine.
Many thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4ca0817a-f1f7-44a9-be3d-3584bcb61b8an%40googlegroups.com.