[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Tamás Gulácsi]

Neither I see a convenient way.
BUT if you add a .go file into the directories where your precompiled 
libraries live,
then "go get" will download them too (and only those dirs that have .go 
files in it).

So your next mission is to prepare the right #cgo CFLAGS LDFLAGS 
incantations to use those libraries.

Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):

> Thanks Tamás, I may not be understanding correctly, but after taking a 
> look at github.com/godror/godror, and the odpi subdirectory,
> I see it is including all the `.c` files on the fly 
> .
>
> A couple of reasons immediately come to mind, that make this not a 
> generically valid option:
>
> * ODPI library is all C code (see src 
> ) so it works 
> including in Go: my dependencies are C++/Rust code, for which I write a 
> small C wrapper (or for Rust just `extern "C"`). Also architecture 
> dependent compilation is different in C++/C/Rust code ...
> * The C++ libraries I'm including have sub-dependencies of themselves (one 
> of which is llvm). It uses Bazel to organize it, and to manually move all 
> the required C++ files to a directory would be years of work :) Plus would 
> require me to slave to maintain things in sync. 
> * The C++ libraries take hours to compile ... I don't want to impose this 
> to users of my libraries.
>
> I think the only way to work this out is distributing the pre-compiled 
> C++/Rust libraries, so the Go simply refer to them (and we get the fast 
> compilation times from Go). But then, how to best distribute them in an 
> automatic fashion, so that users don't need to one by one figure out how to 
> install them (and clean up them later if they are no longer using...) ?
>
> Or maybe there is another convenient way I'm not seeing ?
>
>
> On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:
>
>> Can't you build (make go build for you) those libraries?
>> For example, see github.com/godror/godror just includes the sources of 
>> that third party library in an "odpi" subdir, and with
>> ```
>> /*
>> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>>
>> #include "dpi.c"
>>
>> */
>> import "C"
>> ```
>> it is compiled automatically.
>>
>>
>> Caveat: for "go get" to download a directory, it must include a sth.go 
>> file ("require.go" in the odpi/* subdirs).
>> But it may work that your precompiled libs in a subdir, with a mock .go 
>> file gets downloaded.
>> But how will it found by your lib/app ?
>>
>> Tamás
>>
>>
>> Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):
>>
>>> Thanks Richard. Indeed, as you pointed out the downside is the bloating 
>>> of the git repo, but it makes sense.
>>>
>>> But does the user have to manually clone the repository and move the 
>>> `.a` file to, let's say, /usr/local/lib, or does a simple `go get` 
>>> magically does everything ?
>>>
>>>
>>> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:
>>>
 It isn't a great solution, but I currently include the built library 
 files and necessary headers in the git repo alongside the Go code. You can 
 see an example here 
 https://github.com/richardwilkes/unison/tree/main/internal/skia where 
 I include the skia library I built for use in my UI framework, unison.

 The main downside of this is bloating the git repo with the binary .a 
 and .dll files... but I've not found a better way to handle it. glfw, 
 which 
 unison also depends upon, does something very similar.

 - Rich

 On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:

> hi,
>
> I'm developing a couple of ML framework/libraries for Go that depend 
> on C/C++ code. Once C-libraries dependencies are installed, the CGO 
> integration work great.
>
> Now, for end-users that just want to use these Go libraries, having to 
> figure out how to manually build and install those C/C++/Rust 
> dependencies 
> is a hassle -- sadly each one with a different type of build system.
>
> I offer pre-built `.tgz` files (for a limited set of architectures) 
> with the required `.h` and `.a/.so` files along the releases, which 
> facilitates. But it's still a hassle to install -- and no auto-uninstall 
> if 
> someone is no longer using the Go module.
>
> I was wondering if others have figured out how to handle this in a 
> nicer way.  Is there a recommended way to distribute prebuilt CGO 
> dependencies ?
>
> I like how Python wheels (`.whl` files) solve the issue, by including 
> the pre-built libraries in a sub-directory of the python library 
> installation. I was hoping there would be a similar way (maybe with a 
> separate tool) to integrate with `go.mod`. 
>
> Any pointers, thoughts or suggestions are very appreciated.
>

Re: [go-nuts] Dynamic template data from yaml

['Dan Kortschak' via golang-nuts]

On Sat, 2023-10-14 at 13:15 -0700, Tong Sun wrote:
> Hmm... somehow I had the impression that only the exported fields can
> be used in template as variables.
> 
> Is that a wrong impression or things have changed?

You are indexing into a map so so the notion of fields is not relevant.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/bd1e225259450e2d0cfb10c132045fe7e32faa4c.camel%40kortschak.io.

Re: [go-nuts] Dynamic template data from yaml

[Tong Sun]

On Saturday, October 14, 2023 at 3:47:54 PM UTC-4 Dan Kortschak wrote:

On Sat, 2023-10-14 at 09:33 -0700, Tong Sun wrote: 
> Please take a look at  
> https://go.dev/play/p/dTDR50dtHB0 
> 
> I want to 
> 
> - define my template data dynamically from yaml 
> - and export the yaml data if they are unexported 
> 
> I.e., for the following code: 
> 
> t := template.New("") 
> t, err = t.Parse("It's {{.A}} {{.B.C}}!\n") 
> if err != nil { 
> log.Fatalf("error: %v", err) 
> } 
> t.Execute(os.Stdout, m) 
> 
> The input is `map[A:Easy! B:map[C:2 D:[3 4]]]`. 
> But why the template was not able to produce any output for the 
> dynamic fields? 

It is trying to match a string to a main.MyKey. If the only reason you 
are using that type is to change the case of the key strings, I'd just 
index into the lowercase, https://go.dev/play/p/wi9KICK1zmW.


 wow, thanks.

Hmm... somehow I had the impression that only the exported fields can be 
used in template as variables.

Is that a wrong impression or things have changed?


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8a2df85b-5d5a-4a9d-b0ae-8cc3d46b7aa3n%40googlegroups.com.

Re: [go-nuts] Dynamic template data from yaml

['Dan Kortschak' via golang-nuts]

On Sat, 2023-10-14 at 09:33 -0700, Tong Sun wrote:
> Please take a look at 
> https://go.dev/play/p/dTDR50dtHB0
> 
> I want to
> 
> - define my template data dynamically from yaml
> - and export the yaml data if they are unexported
> 
> I.e., for the following code:
> 
>   t := template.New("")
>   t, err = t.Parse("It's {{.A}} {{.B.C}}!\n")
>   if err != nil {
>   log.Fatalf("error: %v", err)
>   }
>   t.Execute(os.Stdout, m)
> 
> The input is `map[A:Easy! B:map[C:2 D:[3 4]]]`.
> But why the template was not able to produce any output for the
> dynamic fields?

It is trying to match a string to a main.MyKey. If the only reason you
are using that type is to change the case of the key strings, I'd just
index into the lowercase, https://go.dev/play/p/wi9KICK1zmW.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/fee9f5570d0c9370b4ecf3c667df98b4f6a6f0b7.camel%40kortschak.io.

[go-nuts] Dynamic template data from yaml

[Tong Sun]

Please take a look at 
https://go.dev/play/p/dTDR50dtHB0

I want to

- define my template data dynamically from yaml
- and export the yaml data if they are unexported

I.e., for the following code:

t := template.New("")
t, err = t.Parse("It's {{.A}} {{.B.C}}!\n")
if err != nil {
log.Fatalf("error: %v", err)
}
t.Execute(os.Stdout, m)

The input is `map[A:Easy! B:map[C:2 D:[3 4]]]`.
But why the template was not able to produce any output for the dynamic 
fields?

The error is `<.A>: can't evaluate field A in type map[main.MyKey]interface 
{}`
Why is that and how to fix it please?

Thanks

PS. Whole program included below:

-
package main

import (
"fmt"
"log"
"os"
"strings"
"text/template"

"gopkg.in/yaml.v3"
)

var data = `
a: Easy!
b:
  c: 2
  d: [3, 4]
`

type MyKey string

func (k *MyKey) UnmarshalYAML(n *yaml.Node) error {
var s string
if err := n.Decode(); err != nil {
return err
}
*k = MyKey(strings.ToUpper(s))
return nil
}

func main() {
m := make(map[MyKey]any)

err := yaml.Unmarshal([]byte(data), )
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- m:\n%v\n\n", m)

d, err := yaml.Marshal()
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Printf("--- m dump:\n%s\n\n", string(d))

t := template.New("")
t, err = t.Parse("It's {{.A}} {{.B.C}}!\n")
if err != nil {
log.Fatalf("error: %v", err)
}
err = t.Execute(os.Stdout, m)
if err != nil {
log.Fatalf("error: %v", err)
}
}
-

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3f7484d6-fbcd-4b88-aeeb-f581e9eb4952n%40googlegroups.com.

Re: [go-nuts] bufio.Scanner - possible bug or doc err?

['Mark' via golang-nuts]

Yes, that's a much better solution.

On Friday, October 13, 2023 at 8:40:45 PM UTC+1 Ian Lance Taylor wrote:

> On Thu, Oct 12, 2023 at 11:42 PM 'Mark' via golang-nuts
>  wrote:
> >
> > Yes, I can see now.
> >
> > Perhaps consider changing:
> >
> > Programs that need more control over error handling or large tokens, or 
> must run sequential scans on a reader, should use bufio.Reader instead.
> >
> > to:
> >
> > Programs that need more control over error handling or large tokens 
> (such as lines longer than MaxScanTokenSize), or must run sequential scans 
> on a reader, should use bufio.Reader instead.
>
> Thanks, instead of that I added a link to Scanner.Buffer
> (https://go.dev/cl/535216). I hope that will help guide people in the
> right direction.
>
> Ian
>
>
> > On Thursday, October 12, 2023 at 8:56:05 PM UTC+1 Ian Lance Taylor wrote:
> >>
> >> On Thu, Oct 12, 2023 at 10:21 AM 'Mark' via golang-nuts
> >>  wrote:
> >> >
> >> > The docs for bufio.Scanner do say
> >> > "Programs that need more control over error handling or large tokens, 
> or must run sequential scans on a reader, should use bufio.Reader instead"
> >> > Perhaps it would be more helpful to mention what the token length 
> limit is?
> >>
> >> It's MaxScanTokenSize: https://pkg.go.dev/bufio#pkg-constants . See
> >> also https://pkg.go.dev/bufio#Scanner.Buffer .
> >>
> >> Ian
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golang-nuts...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/0392f8b3-a006-4bc0-aa54-3759aa0d3b7en%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a74e8c1b-e11e-4f5d-b66c-43bef18e1715n%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks Tamás, I may not be understanding correctly, but after taking a look 
at github.com/godror/godror, and the odpi subdirectory,
I see it is including all the `.c` files on the fly 
.

A couple of reasons immediately come to mind, that make this not a 
generically valid option:

* ODPI library is all C code (see src 
) so it works 
including in Go: my dependencies are C++/Rust code, for which I write a 
small C wrapper (or for Rust just `extern "C"`). Also architecture 
dependent compilation is different in C++/C/Rust code ...
* The C++ libraries I'm including have sub-dependencies of themselves (one 
of which is llvm). It uses Bazel to organize it, and to manually move all 
the required C++ files to a directory would be years of work :) Plus would 
require me to slave to maintain things in sync. 
* The C++ libraries take hours to compile ... I don't want to impose this 
to users of my libraries.

I think the only way to work this out is distributing the pre-compiled 
C++/Rust libraries, so the Go simply refer to them (and we get the fast 
compilation times from Go). But then, how to best distribute them in an 
automatic fashion, so that users don't need to one by one figure out how to 
install them (and clean up them later if they are no longer using...) ?

Or maybe there is another convenient way I'm not seeing ?


On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:

> Can't you build (make go build for you) those libraries?
> For example, see github.com/godror/godror just includes the sources of 
> that third party library in an "odpi" subdir, and with
> ```
> /*
> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>
> #include "dpi.c"
>
> */
> import "C"
> ```
> it is compiled automatically.
>
>
> Caveat: for "go get" to download a directory, it must include a sth.go 
> file ("require.go" in the odpi/* subdirs).
> But it may work that your precompiled libs in a subdir, with a mock .go 
> file gets downloaded.
> But how will it found by your lib/app ?
>
> Tamás
>
>
> Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):
>
>> Thanks Richard. Indeed, as you pointed out the downside is the bloating 
>> of the git repo, but it makes sense.
>>
>> But does the user have to manually clone the repository and move the `.a` 
>> file to, let's say, /usr/local/lib, or does a simple `go get` magically 
>> does everything ?
>>
>>
>> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:
>>
>>> It isn't a great solution, but I currently include the built library 
>>> files and necessary headers in the git repo alongside the Go code. You can 
>>> see an example here 
>>> https://github.com/richardwilkes/unison/tree/main/internal/skia where I 
>>> include the skia library I built for use in my UI framework, unison.
>>>
>>> The main downside of this is bloating the git repo with the binary .a 
>>> and .dll files... but I've not found a better way to handle it. glfw, which 
>>> unison also depends upon, does something very similar.
>>>
>>> - Rich
>>>
>>> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>>>
 hi,

 I'm developing a couple of ML framework/libraries for Go that depend on 
 C/C++ code. Once C-libraries dependencies are installed, the CGO 
 integration work great.

 Now, for end-users that just want to use these Go libraries, having to 
 figure out how to manually build and install those C/C++/Rust dependencies 
 is a hassle -- sadly each one with a different type of build system.

 I offer pre-built `.tgz` files (for a limited set of architectures) 
 with the required `.h` and `.a/.so` files along the releases, which 
 facilitates. But it's still a hassle to install -- and no auto-uninstall 
 if 
 someone is no longer using the Go module.

 I was wondering if others have figured out how to handle this in a 
 nicer way.  Is there a recommended way to distribute prebuilt CGO 
 dependencies ?

 I like how Python wheels (`.whl` files) solve the issue, by including 
 the pre-built libraries in a sub-directory of the python library 
 installation. I was hoping there would be a similar way (maybe with a 
 separate tool) to integrate with `go.mod`. 

 Any pointers, thoughts or suggestions are very appreciated.

 Thanks!
 Jan

 ps: While searching I saw similar questions, but none that exactly 
 answered this aspect of distribution. Just in case, apologies if it's a 
 duplicate question.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit