Re: [go-nuts] Re: Go called by C: is the goroutine locked to a thread?

[Domenico Andreoli]

On Sun, Oct 29, 2023 at 11:32 PM Jason E. Aten  wrote:

> You could also look at the existing Go <-> Python interfaces and see how
> they handle such issues. Might give you hints.
>
> https://github.com/qur/gopy. (python 3.11)
> https://github.com/glycerine/pyg (python 3.7.1)
>
> Thanks for the pointers, I'll check what they do.

-- 
rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGH6_pqkYp_-HxhBHZnkpPdWKG4DMo%3DUSrJ%3DveNQ8Ua7h9%3D4YA%40mail.gmail.com.

[go-nuts] Re: Go called by C: is the goroutine locked to a thread?

[Jason E. Aten]

You could also look at the existing Go <-> Python interfaces and see how 
they handle such issues. Might give you hints.

https://github.com/qur/gopy. (python 3.11)
https://github.com/glycerine/pyg (python 3.7.1)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/82d6388a-7694-4a8e-bc65-750178052371n%40googlegroups.com.

Re: [go-nuts] Re: Go called by C: is the goroutine locked to a thread?

['Bryan C. Mills' via golang-nuts]

On Wed, Oct 25, 2023 at 9:58 AM Domenico Andreoli <
domenico.andre...@linux.com> wrote:

> On Tue, Oct 24, 2023 at 09:58:22AM -0700, 'Bryan C. Mills' via golang-nuts
> wrote:
> > If a C thread calls into Go, the goroutine processing that call (and
> only
> > that goroutine) will run on the C thread. The Go runtime will initialize
>
> Is this thanks to the `lockOSThread` call in function `cgocallbackg` of
> https://github.com/golang/go/blob/master/src/runtime/cgocall.go?
>

I'm not sure, to be honest. I think so, though. 

> any of its own per-thread state for that thread as needed. If a goroutine
> > calls into C, and that C thread calls back into Go, I believe that the
> > C-to-Go call will be on the same goroutine as the Go-to-C call.
>
> This is great news.
>
> > So if you have a chain of cross-language calls (say, Go to C to Go to C
> to
> > Go), that should still only consume one OS thread in total.
>
> That's good. I found it also explained here
> https://groups.google.com/g/golang-nuts/c/8Lx2TUzeXQE/m/3yl0A-wPEAAJ.
>
> >
> > On the other hand, if you have a goroutine without any C call frames,
> and
> > that goroutine may have thread-local data for the Python interpreter,
> you
> > should explicitly call LockOSThread on that goroutine — and never unlock
> > it! — to avoid that data migrating unexpectedly to another goroutine.
> > See https://go.dev/issue/20395.
>
> That's a very interesting and eye opening link, thanks a lot.
>
> Dom
>
> >
> > On Monday, October 23, 2023 at 8:34:28 AM UTC-4 Domenico Andreoli wrote:
> >
> > > Hi,
> > >
> > >   I'm writing a Go library to embed/extend CPython (Pygolo
> > > ). CPython uses thread local storage
> and
> > > therefore I need to care about pinning goroutines to OS threads
> accordingly.
> > >
> > > It's pretty much clear that when the Python interpreter is embedded it
> can
> > > be accessed only from one goroutine and that such goroutine must be
> locked
> > > to an OS thread all the time.
> > >
> > > It's possible to create Python thread states and allow multiple
> threads to
> > > access the interpreter, therefore other goroutines - if locked to an
> OS
> > > thread - can access the interpreter (modulo Python GIL).
> > >
> > > It's also clear that if the Python interpreter calls back a Go
> function it
> > > will happen from one of the locked threads of above and therefore
> nothing
> > > special needs to be done on the Go side.
> > >
> > > Or not? If a thread is locked no other goroutines are allowed to run
> as
> > > per documentation of LockOSThread, so on which goroutine the Go
> callback
> > > will actually run?
> > >
> > > Different story is when Go is used to extend the Python interpreter,
> when
> > > Go calls are happening in a thread not owned by the Go runtime. Here I
> have
> > > some doubts, I think I read something about cgo locking threads of Go
> > > callbacks but I can't find it any more.
> > >
> > > I guess that at the first Go call cgo initializes the Go runtime and
> > > suddenly a few threads and goroutines spawn to life but how the
> goroutine
> > > scheduling works when C is in control of the calling threads?
> > >
> > > Is it possible that multiple goroutines get scheduled on the caller C
> > > thread?
> > >
> > > Are Go callbacks goroutines reused? If not and I leave a Go callback
> > > goroutine locked to a thread, is the calling C thread going to be
> killed,
> > > as per documentation of LockOSThread?
> > >
> > > Thanks in advance for any clarification.
> > >
> > > Regards,
> > > Dom
>
> --
> rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
> ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKWVi_SRKmzeqLuG5Cpr6jzKDzbZ1ODEw%2B1YmNZtjqL-p%2BPLrA%40mail.gmail.com.

Re: [go-nuts] Re: Go called by C: is the goroutine locked to a thread?

[Domenico Andreoli]

On Tue, Oct 24, 2023 at 09:58:22AM -0700, 'Bryan C. Mills' via golang-nuts 
wrote:
> If a C thread calls into Go, the goroutine processing that call (and only 
> that goroutine) will run on the C thread. The Go runtime will initialize 

Is this thanks to the `lockOSThread` call in function `cgocallbackg` of
https://github.com/golang/go/blob/master/src/runtime/cgocall.go?

> any of its own per-thread state for that thread as needed. If a goroutine 
> calls into C, and that C thread calls back into Go, I believe that the 
> C-to-Go call will be on the same goroutine as the Go-to-C call.

This is great news.

> So if you have a chain of cross-language calls (say, Go to C to Go to C to 
> Go), that should still only consume one OS thread in total.

That's good. I found it also explained here
https://groups.google.com/g/golang-nuts/c/8Lx2TUzeXQE/m/3yl0A-wPEAAJ.

> 
> On the other hand, if you have a goroutine without any C call frames, and 
> that goroutine may have thread-local data for the Python interpreter, you 
> should explicitly call LockOSThread on that goroutine — and never unlock 
> it! — to avoid that data migrating unexpectedly to another goroutine. 
> See https://go.dev/issue/20395.

That's a very interesting and eye opening link, thanks a lot.

Dom

> 
> On Monday, October 23, 2023 at 8:34:28 AM UTC-4 Domenico Andreoli wrote:
> 
> > Hi,
> >
> >   I'm writing a Go library to embed/extend CPython (Pygolo 
> > ). CPython uses thread local storage and 
> > therefore I need to care about pinning goroutines to OS threads accordingly.
> >
> > It's pretty much clear that when the Python interpreter is embedded it can 
> > be accessed only from one goroutine and that such goroutine must be locked 
> > to an OS thread all the time.
> >
> > It's possible to create Python thread states and allow multiple threads to 
> > access the interpreter, therefore other goroutines - if locked to an OS 
> > thread - can access the interpreter (modulo Python GIL).
> >
> > It's also clear that if the Python interpreter calls back a Go function it 
> > will happen from one of the locked threads of above and therefore nothing 
> > special needs to be done on the Go side.
> >
> > Or not? If a thread is locked no other goroutines are allowed to run as 
> > per documentation of LockOSThread, so on which goroutine the Go callback 
> > will actually run?
> >
> > Different story is when Go is used to extend the Python interpreter, when 
> > Go calls are happening in a thread not owned by the Go runtime. Here I have 
> > some doubts, I think I read something about cgo locking threads of Go 
> > callbacks but I can't find it any more. 
> >
> > I guess that at the first Go call cgo initializes the Go runtime and 
> > suddenly a few threads and goroutines spawn to life but how the goroutine 
> > scheduling works when C is in control of the calling threads?
> >
> > Is it possible that multiple goroutines get scheduled on the caller C 
> > thread?
> >
> > Are Go callbacks goroutines reused? If not and I leave a Go callback 
> > goroutine locked to a thread, is the calling C thread going to be killed, 
> > as per documentation of LockOSThread?
> >
> > Thanks in advance for any clarification.
> >
> > Regards,
> > Dom

-- 
rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ZTkfF4XpOdTpQzA4%40localhost.

[go-nuts] Re: Go called by C: is the goroutine locked to a thread?

['Bryan C. Mills' via golang-nuts]

If a C thread calls into Go, the goroutine processing that call (and only 
that goroutine) will run on the C thread. The Go runtime will initialize 
any of its own per-thread state for that thread as needed. If a goroutine 
calls into C, and that C thread calls back into Go, I believe that the 
C-to-Go call will be on the same goroutine as the Go-to-C call.

So if you have a chain of cross-language calls (say, Go to C to Go to C to 
Go), that should still only consume one OS thread in total.

On the other hand, if you have a goroutine without any C call frames, and 
that goroutine may have thread-local data for the Python interpreter, you 
should explicitly call LockOSThread on that goroutine — and never unlock 
it! — to avoid that data migrating unexpectedly to another goroutine. 
See https://go.dev/issue/20395.

On Monday, October 23, 2023 at 8:34:28 AM UTC-4 Domenico Andreoli wrote:

> Hi,
>
>   I'm writing a Go library to embed/extend CPython (Pygolo 
> ). CPython uses thread local storage and 
> therefore I need to care about pinning goroutines to OS threads accordingly.
>
> It's pretty much clear that when the Python interpreter is embedded it can 
> be accessed only from one goroutine and that such goroutine must be locked 
> to an OS thread all the time.
>
> It's possible to create Python thread states and allow multiple threads to 
> access the interpreter, therefore other goroutines - if locked to an OS 
> thread - can access the interpreter (modulo Python GIL).
>
> It's also clear that if the Python interpreter calls back a Go function it 
> will happen from one of the locked threads of above and therefore nothing 
> special needs to be done on the Go side.
>
> Or not? If a thread is locked no other goroutines are allowed to run as 
> per documentation of LockOSThread, so on which goroutine the Go callback 
> will actually run?
>
> Different story is when Go is used to extend the Python interpreter, when 
> Go calls are happening in a thread not owned by the Go runtime. Here I have 
> some doubts, I think I read something about cgo locking threads of Go 
> callbacks but I can't find it any more. 
>
> I guess that at the first Go call cgo initializes the Go runtime and 
> suddenly a few threads and goroutines spawn to life but how the goroutine 
> scheduling works when C is in control of the calling threads?
>
> Is it possible that multiple goroutines get scheduled on the caller C 
> thread?
>
> Are Go callbacks goroutines reused? If not and I leave a Go callback 
> goroutine locked to a thread, is the calling C thread going to be killed, 
> as per documentation of LockOSThread?
>
> Thanks in advance for any clarification.
>
> Regards,
> Dom
>
> -- 
> rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
> ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/68c9fc17-cca7-4f54-b4af-69df8bf3300dn%40googlegroups.com.