Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Konstantin Ivaschenko]

@Jan You haven't thought of a good way to do it?

суббота, 4 ноября 2023 г. в 05:23:51 UTC+3, Luke Crook: 

> For Windows, could you package and distribute the prebuilt binaries using 
> an MSI file? (https://nsis.sourceforge.io/Main_Page)
>
>
>
> On Fri, Nov 3, 2023 at 12:10 PM Robert Engels  
> wrote:
>
>> Better to rewrite the native portions in Go. 
>>
>> If you’re not going to do that, just give instructions on how to build 
>> and let the app builder figure out how to package it. 
>>
>> On Nov 3, 2023, at 1:47 PM, Jason E. Aten  wrote:
>>
>> 
>> It is still rough around lots of edges, but you are wanting to write your 
>> own deployment system, you might find in https://nixos.org/ some 
>> inspiration.
>>
>> It generalizes the idea of cryptographic hashed based dependency 
>> management to, well, everything.
>>
>> Sadly not that easy to use; it may be harsh to inflict on end users.  But 
>> it tackles the reproducibility issue (it works on my machine! but not 
>> yours?) head on.
>> You might profitably write some glue between Go and Nix that provides a 
>> nice end user experience.
>>
>> On Friday, November 3, 2023 at 5:23:21 PM UTC Jan wrote:
>>
>>> That works if what I'm doing is a end product (a server or something). 
>>> But what I'm doing is in itself a library. Imagine if every library offers 
>>> a docker/VM image, how is the end-user supposed to merge those docker 
>>> images ? The docker is also not a viable/ergonomic solution in my case. I 
>>> mean ... actually I do offer a docker for one of my projects 
>>> , along with 
>>> Jupyter, a Tutorial and demo. But generically, it is a library, and I would 
>>> like it to be easily `go get`able. Or at most with one simple/standard step.
>>>
>>> Yes, I hear you with respect to unnecessary extra dependencies (X11) -- 
>>> at least make them optional, right ? But In my case the dependencies are 
>>> essential, and the docker doesn't really solve the problem...  But also I'm 
>>> running out of hope that there would be anything to solve it, I'm almost 
>>> thinking I should write something myself.
>>>
>>>
>>> On Thursday, November 2, 2023 at 11:54:01 PM UTC+1 Jason E. Aten wrote:
>>>
 What I would do for that case would be to deliver the users a either a 
 VM image (heavy), or a docker container (lighter) with batteries included 
 inside. 

 There is often little need to write a docker file unless you really 
 want. Just get everything working inside docker and "docker commit" it to 
 an image. I do this often for projects that need python/R + 100s of 
 libraries.  Its really the only sane way to deliver sprawling dependencies 
 that assume they can write all over the filesystem.  

 One hint: I recently did this for code that wanted to output graphics, 
 and that needed X11 (or Xvfb), and that wanted systemd, which is available 
 in docker but used to be discouraged so is not available in many base 
 images. So I would recommend starting an docker image that already 
 supports 
 systemd and X11, if graphics (or x11vnc, say) is ever going to be desired. 
 Its a pain to add after the fact. Podman claims to be useful for this, but 
 I found in immature and not really production ready when trying to run it 
 on blah standard Ubuntu 22.

 On Wednesday, November 1, 2023 at 9:08:51 PM UTC Jan wrote:

> Thanks @Jason, but the point was exactly not need to do anything 
> extra: no manual unzipping of a file, or manual Makefile/Magefile.
>
> Let's say project in Go links some 30 external modules, 5 of them have 
> their own specific steps that need to be read, understood and and run to 
> install them ... very quickly the user just gives up from 
> installing/compiling the thing, with good reason ... 
>
> The Go toolset solves that beautifully for Go only projects. But at 
> least in my line of work, that's not feasible, I need to interface with 
> libraries that in turn require other libraries in C++ (OpenXLA/llvm for 
> JIT), Rust (HuggingFace Tokenizer), C (Gtk), etc.
>
> cheers
>
>
>
>
> On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:
>
>> > including the `.a` (static libraries) in the Github just got to its 
>> limit (100Mb)
>>
>> I have used bzip2 to compress libraries that are too big for Github. 
>> If that gets them under the limit, then great. 
>> Just provide installation instructions or a Makefile target that 
>> decompresses them once checked out.
>>
>> (Or xz? Apparently it can compress more than bzip2; but I have no 
>> experience with it. https://en.wikipedia.org/wiki/XZ_Utils )
>>
> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it,

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

['Luke Crook' via golang-nuts]

For Windows, could you package and distribute the prebuilt binaries using
an MSI file? (https://nsis.sourceforge.io/Main_Page)



On Fri, Nov 3, 2023 at 12:10 PM Robert Engels  wrote:

> Better to rewrite the native portions in Go.
>
> If you’re not going to do that, just give instructions on how to build and
> let the app builder figure out how to package it.
>
> On Nov 3, 2023, at 1:47 PM, Jason E. Aten  wrote:
>
> 
> It is still rough around lots of edges, but you are wanting to write your
> own deployment system, you might find in https://nixos.org/ some
> inspiration.
>
> It generalizes the idea of cryptographic hashed based dependency
> management to, well, everything.
>
> Sadly not that easy to use; it may be harsh to inflict on end users.  But
> it tackles the reproducibility issue (it works on my machine! but not
> yours?) head on.
> You might profitably write some glue between Go and Nix that provides a
> nice end user experience.
>
> On Friday, November 3, 2023 at 5:23:21 PM UTC Jan wrote:
>
>> That works if what I'm doing is a end product (a server or something).
>> But what I'm doing is in itself a library. Imagine if every library offers
>> a docker/VM image, how is the end-user supposed to merge those docker
>> images ? The docker is also not a viable/ergonomic solution in my case. I
>> mean ... actually I do offer a docker for one of my projects
>> , along with
>> Jupyter, a Tutorial and demo. But generically, it is a library, and I would
>> like it to be easily `go get`able. Or at most with one simple/standard step.
>>
>> Yes, I hear you with respect to unnecessary extra dependencies (X11) --
>> at least make them optional, right ? But In my case the dependencies are
>> essential, and the docker doesn't really solve the problem...  But also I'm
>> running out of hope that there would be anything to solve it, I'm almost
>> thinking I should write something myself.
>>
>>
>> On Thursday, November 2, 2023 at 11:54:01 PM UTC+1 Jason E. Aten wrote:
>>
>>> What I would do for that case would be to deliver the users a either a
>>> VM image (heavy), or a docker container (lighter) with batteries included
>>> inside.
>>>
>>> There is often little need to write a docker file unless you really
>>> want. Just get everything working inside docker and "docker commit" it to
>>> an image. I do this often for projects that need python/R + 100s of
>>> libraries.  Its really the only sane way to deliver sprawling dependencies
>>> that assume they can write all over the filesystem.
>>>
>>> One hint: I recently did this for code that wanted to output graphics,
>>> and that needed X11 (or Xvfb), and that wanted systemd, which is available
>>> in docker but used to be discouraged so is not available in many base
>>> images. So I would recommend starting an docker image that already supports
>>> systemd and X11, if graphics (or x11vnc, say) is ever going to be desired.
>>> Its a pain to add after the fact. Podman claims to be useful for this, but
>>> I found in immature and not really production ready when trying to run it
>>> on blah standard Ubuntu 22.
>>>
>>> On Wednesday, November 1, 2023 at 9:08:51 PM UTC Jan wrote:
>>>
 Thanks @Jason, but the point was exactly not need to do anything extra:
 no manual unzipping of a file, or manual Makefile/Magefile.

 Let's say project in Go links some 30 external modules, 5 of them have
 their own specific steps that need to be read, understood and and run to
 install them ... very quickly the user just gives up from
 installing/compiling the thing, with good reason ...

 The Go toolset solves that beautifully for Go only projects. But at
 least in my line of work, that's not feasible, I need to interface with
 libraries that in turn require other libraries in C++ (OpenXLA/llvm for
 JIT), Rust (HuggingFace Tokenizer), C (Gtk), etc.

 cheers




 On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:

> > including the `.a` (static libraries) in the Github just got to its
> limit (100Mb)
>
> I have used bzip2 to compress libraries that are too big for Github.
> If that gets them under the limit, then great.
> Just provide installation instructions or a Makefile target that
> decompresses them once checked out.
>
> (Or xz? Apparently it can compress more than bzip2; but I have no
> experience with it. https://en.wikipedia.org/wiki/XZ_Utils )
>
 --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/149f28a5-417e-4642-9c04-13f0254e2dcbn%40googlegroups.com
> 

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Robert Engels]

Better to rewrite the native portions in Go. 

If you’re not going to do that, just give instructions on how to build and let 
the app builder figure out how to package it. 

> On Nov 3, 2023, at 1:47 PM, Jason E. Aten  wrote:
> 
> 
> It is still rough around lots of edges, but you are wanting to write your own 
> deployment system, you might find in https://nixos.org/ some inspiration.
> 
> It generalizes the idea of cryptographic hashed based dependency management 
> to, well, everything.
> 
> Sadly not that easy to use; it may be harsh to inflict on end users.  But it 
> tackles the reproducibility issue (it works on my machine! but not yours?) 
> head on.
> You might profitably write some glue between Go and Nix that provides a nice 
> end user experience.
> 
>> On Friday, November 3, 2023 at 5:23:21 PM UTC Jan wrote:
>> That works if what I'm doing is a end product (a server or something). But 
>> what I'm doing is in itself a library. Imagine if every library offers a 
>> docker/VM image, how is the end-user supposed to merge those docker images ? 
>> The docker is also not a viable/ergonomic solution in my case. I mean ... 
>> actually I do offer a docker for one of my projects, along with Jupyter, a 
>> Tutorial and demo. But generically, it is a library, and I would like it to 
>> be easily `go get`able. Or at most with one simple/standard step.
>> 
>> Yes, I hear you with respect to unnecessary extra dependencies (X11) -- at 
>> least make them optional, right ? But In my case the dependencies are 
>> essential, and the docker doesn't really solve the problem...  But also I'm 
>> running out of hope that there would be anything to solve it, I'm almost 
>> thinking I should write something myself.
>> 
>> 
>>> On Thursday, November 2, 2023 at 11:54:01 PM UTC+1 Jason E. Aten wrote:
>>> What I would do for that case would be to deliver the users a either a VM 
>>> image (heavy), or a docker container (lighter) with batteries included 
>>> inside. 
>>> 
>>> There is often little need to write a docker file unless you really want. 
>>> Just get everything working inside docker and "docker commit" it to an 
>>> image. I do this often for projects that need python/R + 100s of libraries. 
>>>  Its really the only sane way to deliver sprawling dependencies that assume 
>>> they can write all over the filesystem.  
>>> 
>>> One hint: I recently did this for code that wanted to output graphics, and 
>>> that needed X11 (or Xvfb), and that wanted systemd, which is available in 
>>> docker but used to be discouraged so is not available in many base images. 
>>> So I would recommend starting an docker image that already supports systemd 
>>> and X11, if graphics (or x11vnc, say) is ever going to be desired. Its a 
>>> pain to add after the fact. Podman claims to be useful for this, but I 
>>> found in immature and not really production ready when trying to run it on 
>>> blah standard Ubuntu 22.
>>> 
 On Wednesday, November 1, 2023 at 9:08:51 PM UTC Jan wrote:
 Thanks @Jason, but the point was exactly not need to do anything extra: no 
 manual unzipping of a file, or manual Makefile/Magefile.
 
 Let's say project in Go links some 30 external modules, 5 of them have 
 their own specific steps that need to be read, understood and and run to 
 install them ... very quickly the user just gives up from 
 installing/compiling the thing, with good reason ... 
 
 The Go toolset solves that beautifully for Go only projects. But at least 
 in my line of work, that's not feasible, I need to interface with 
 libraries that in turn require other libraries in C++ (OpenXLA/llvm for 
 JIT), Rust (HuggingFace Tokenizer), C (Gtk), etc.
 
 cheers
 
 
 
 
> On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:
> > including the `.a` (static libraries) in the Github just got to its 
> > limit (100Mb)
> 
> I have used bzip2 to compress libraries that are too big for Github. If 
> that gets them under the limit, then great. 
> Just provide installation instructions or a Makefile target that 
> decompresses them once checked out.
> 
> (Or xz? Apparently it can compress more than bzip2; but I have no 
> experience with it. https://en.wikipedia.org/wiki/XZ_Utils )
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/149f28a5-417e-4642-9c04-13f0254e2dcbn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web 

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jason E. Aten]

It is still rough around lots of edges, but you are wanting to write your 
own deployment system, you might find in https://nixos.org/ some 
inspiration.

It generalizes the idea of cryptographic hashed based dependency management 
to, well, everything.

Sadly not that easy to use; it may be harsh to inflict on end users.  But 
it tackles the reproducibility issue (it works on my machine! but not 
yours?) head on.
You might profitably write some glue between Go and Nix that provides a 
nice end user experience.

On Friday, November 3, 2023 at 5:23:21 PM UTC Jan wrote:

> That works if what I'm doing is a end product (a server or something). But 
> what I'm doing is in itself a library. Imagine if every library offers a 
> docker/VM image, how is the end-user supposed to merge those docker images 
> ? The docker is also not a viable/ergonomic solution in my case. I mean ... 
> actually I do offer a docker for one of my projects 
> , along with 
> Jupyter, a Tutorial and demo. But generically, it is a library, and I would 
> like it to be easily `go get`able. Or at most with one simple/standard step.
>
> Yes, I hear you with respect to unnecessary extra dependencies (X11) -- at 
> least make them optional, right ? But In my case the dependencies are 
> essential, and the docker doesn't really solve the problem...  But also I'm 
> running out of hope that there would be anything to solve it, I'm almost 
> thinking I should write something myself.
>
>
> On Thursday, November 2, 2023 at 11:54:01 PM UTC+1 Jason E. Aten wrote:
>
>> What I would do for that case would be to deliver the users a either a VM 
>> image (heavy), or a docker container (lighter) with batteries included 
>> inside. 
>>
>> There is often little need to write a docker file unless you really want. 
>> Just get everything working inside docker and "docker commit" it to an 
>> image. I do this often for projects that need python/R + 100s of 
>> libraries.  Its really the only sane way to deliver sprawling dependencies 
>> that assume they can write all over the filesystem.  
>>
>> One hint: I recently did this for code that wanted to output graphics, 
>> and that needed X11 (or Xvfb), and that wanted systemd, which is available 
>> in docker but used to be discouraged so is not available in many base 
>> images. So I would recommend starting an docker image that already supports 
>> systemd and X11, if graphics (or x11vnc, say) is ever going to be desired. 
>> Its a pain to add after the fact. Podman claims to be useful for this, but 
>> I found in immature and not really production ready when trying to run it 
>> on blah standard Ubuntu 22.
>>
>> On Wednesday, November 1, 2023 at 9:08:51 PM UTC Jan wrote:
>>
>>> Thanks @Jason, but the point was exactly not need to do anything extra: 
>>> no manual unzipping of a file, or manual Makefile/Magefile.
>>>
>>> Let's say project in Go links some 30 external modules, 5 of them have 
>>> their own specific steps that need to be read, understood and and run to 
>>> install them ... very quickly the user just gives up from 
>>> installing/compiling the thing, with good reason ... 
>>>
>>> The Go toolset solves that beautifully for Go only projects. But at 
>>> least in my line of work, that's not feasible, I need to interface with 
>>> libraries that in turn require other libraries in C++ (OpenXLA/llvm for 
>>> JIT), Rust (HuggingFace Tokenizer), C (Gtk), etc.
>>>
>>> cheers
>>>
>>>
>>>
>>>
>>> On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:
>>>
 > including the `.a` (static libraries) in the Github just got to its 
 limit (100Mb)

 I have used bzip2 to compress libraries that are too big for Github. If 
 that gets them under the limit, then great. 
 Just provide installation instructions or a Makefile target that 
 decompresses them once checked out.

 (Or xz? Apparently it can compress more than bzip2; but I have no 
 experience with it. https://en.wikipedia.org/wiki/XZ_Utils )

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/149f28a5-417e-4642-9c04-13f0254e2dcbn%40googlegroups.com.

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

That works if what I'm doing is a end product (a server or something). But 
what I'm doing is in itself a library. Imagine if every library offers a 
docker/VM image, how is the end-user supposed to merge those docker images 
? The docker is also not a viable/ergonomic solution in my case. I mean ... 
actually I do offer a docker for one of my projects 
, along with Jupyter, 
a Tutorial and demo. But generically, it is a library, and I would like it 
to be easily `go get`able. Or at most with one simple/standard step.

Yes, I hear you with respect to unnecessary extra dependencies (X11) -- at 
least make them optional, right ? But In my case the dependencies are 
essential, and the docker doesn't really solve the problem...  But also I'm 
running out of hope that there would be anything to solve it, I'm almost 
thinking I should write something myself.


On Thursday, November 2, 2023 at 11:54:01 PM UTC+1 Jason E. Aten wrote:

> What I would do for that case would be to deliver the users a either a VM 
> image (heavy), or a docker container (lighter) with batteries included 
> inside. 
>
> There is often little need to write a docker file unless you really want. 
> Just get everything working inside docker and "docker commit" it to an 
> image. I do this often for projects that need python/R + 100s of 
> libraries.  Its really the only sane way to deliver sprawling dependencies 
> that assume they can write all over the filesystem.  
>
> One hint: I recently did this for code that wanted to output graphics, and 
> that needed X11 (or Xvfb), and that wanted systemd, which is available in 
> docker but used to be discouraged so is not available in many base images. 
> So I would recommend starting an docker image that already supports systemd 
> and X11, if graphics (or x11vnc, say) is ever going to be desired. Its a 
> pain to add after the fact. Podman claims to be useful for this, but I 
> found in immature and not really production ready when trying to run it on 
> blah standard Ubuntu 22.
>
> On Wednesday, November 1, 2023 at 9:08:51 PM UTC Jan wrote:
>
>> Thanks @Jason, but the point was exactly not need to do anything extra: 
>> no manual unzipping of a file, or manual Makefile/Magefile.
>>
>> Let's say project in Go links some 30 external modules, 5 of them have 
>> their own specific steps that need to be read, understood and and run to 
>> install them ... very quickly the user just gives up from 
>> installing/compiling the thing, with good reason ... 
>>
>> The Go toolset solves that beautifully for Go only projects. But at least 
>> in my line of work, that's not feasible, I need to interface with libraries 
>> that in turn require other libraries in C++ (OpenXLA/llvm for JIT), Rust 
>> (HuggingFace Tokenizer), C (Gtk), etc.
>>
>> cheers
>>
>>
>>
>>
>> On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:
>>
>>> > including the `.a` (static libraries) in the Github just got to its 
>>> limit (100Mb)
>>>
>>> I have used bzip2 to compress libraries that are too big for Github. If 
>>> that gets them under the limit, then great. 
>>> Just provide installation instructions or a Makefile target that 
>>> decompresses them once checked out.
>>>
>>> (Or xz? Apparently it can compress more than bzip2; but I have no 
>>> experience with it. https://en.wikipedia.org/wiki/XZ_Utils )
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/58a7156a-94e8-4084-b82a-a7269d785089n%40googlegroups.com.

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jason E. Aten]

What I would do for that case would be to deliver the users a either a VM 
image (heavy), or a docker container (lighter) with batteries included 
inside. 

There is often little need to write a docker file unless you really want. 
Just get everything working inside docker and "docker commit" it to an 
image. I do this often for projects that need python/R + 100s of 
libraries.  Its really the only sane way to deliver sprawling dependencies 
that assume they can write all over the filesystem.  

One hint: I recently did this for code that wanted to output graphics, and 
that needed X11 (or Xvfb), and that wanted systemd, which is available in 
docker but used to be discouraged so is not available in many base images. 
So I would recommend starting an docker image that already supports systemd 
and X11, if graphics (or x11vnc, say) is ever going to be desired. Its a 
pain to add after the fact. Podman claims to be useful for this, but I 
found in immature and not really production ready when trying to run it on 
blah standard Ubuntu 22.

On Wednesday, November 1, 2023 at 9:08:51 PM UTC Jan wrote:

> Thanks @Jason, but the point was exactly not need to do anything extra: no 
> manual unzipping of a file, or manual Makefile/Magefile.
>
> Let's say project in Go links some 30 external modules, 5 of them have 
> their own specific steps that need to be read, understood and and run to 
> install them ... very quickly the user just gives up from 
> installing/compiling the thing, with good reason ... 
>
> The Go toolset solves that beautifully for Go only projects. But at least 
> in my line of work, that's not feasible, I need to interface with libraries 
> that in turn require other libraries in C++ (OpenXLA/llvm for JIT), Rust 
> (HuggingFace Tokenizer), C (Gtk), etc.
>
> cheers
>
>
>
>
> On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:
>
>> > including the `.a` (static libraries) in the Github just got to its 
>> limit (100Mb)
>>
>> I have used bzip2 to compress libraries that are too big for Github. If 
>> that gets them under the limit, then great. 
>> Just provide installation instructions or a Makefile target that 
>> decompresses them once checked out.
>>
>> (Or xz? Apparently it can compress more than bzip2; but I have no 
>> experience with it. https://en.wikipedia.org/wiki/XZ_Utils )
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/56491297-80a0-492f-a794-46eec374871fn%40googlegroups.com.

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks @Jason, but the point was exactly not need to do anything extra: no 
manual unzipping of a file, or manual Makefile/Magefile.

Let's say project in Go links some 30 external modules, 5 of them have 
their own specific steps that need to be read, understood and and run to 
install them ... very quickly the user just gives up from 
installing/compiling the thing, with good reason ... 

The Go toolset solves that beautifully for Go only projects. But at least 
in my line of work, that's not feasible, I need to interface with libraries 
that in turn require other libraries in C++ (OpenXLA/llvm for JIT), Rust 
(HuggingFace Tokenizer), C (Gtk), etc.

cheers




On Monday, October 30, 2023 at 7:42:19 PM UTC+1 Jason E. Aten wrote:

> > including the `.a` (static libraries) in the Github just got to its 
> limit (100Mb)
>
> I have used bzip2 to compress libraries that are too big for Github. If 
> that gets them under the limit, then great. 
> Just provide installation instructions or a Makefile target that 
> decompresses them once checked out.
>
> (Or xz? Apparently it can compress more than bzip2; but I have no 
> experience with it. https://en.wikipedia.org/wiki/XZ_Utils )
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/096e5389-d183-4b2a-bb6b-b136398dc5acn%40googlegroups.com.

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks for all the replies. 

Unfortunately, as I found out today, including the `.a` (static libraries) 
in the Github just got to its limit (100Mb), so I don't think this is an 
option anymore :( I'm going to investigate the Gihtub LFS (Large File 
Storage), not sure how well it will integrate with `go get` though -- I've 
never used it before.

 Interesting to know that an empty Go package file is no longer needed to 
fetch a directory with binary dependencies (handy also for go:embed in 
cases where the binary files are on a separate directory).

Still for my Go library with a `.so` dependency I'm not sure yet what is 
the answer. I'm thinking about creating another tool that goes over all the 
dependencies of module, and automatically downloads release files to a 
configured directory ... so at least all one needs to do is execute one 
command and get all dependencies of all libraries installed. Any thoughts ?

@Mike Schinkel: the go:embed is used in a different context (when embedding 
blobs of binary). We need prebuilt binary libraries that will be 
linked with the Go code during compilation. Preferably something that will 
transparently work with `go get` / `go build` / `go install`.

cheers

On Tuesday, October 17, 2023 at 8:46:01 PM UTC+2 Justin Israel wrote:

> On Wednesday, October 18, 2023 at 3:12:29 AM UTC+13 Marc-Antoine Ruel 
> wrote:
>
> Sorry I wasn't clear. The static libraries are in a subdirectory because 
> the user should not care and these libraries are effectively third party 
> code.
>
>
> Actually, you were totally clear. Sorry if my comment didn't make sense. I 
> get that the user doesn't need to worry about the third_party directory 
> with the header and pre-built static libraries. And I saw how you arrange 
> to pick up the right ones in the code. My point was that from previous 
> experience, "go get" would prune away directories that do not contain go 
> source code, after it performs the clone from the remote, into the module 
> cache. So I would have expected that when someone uses your library as a 
> dependency, it would have omitted the needed third_party directory, for not 
> having any Go source files in it. But when I tested it just now, it does 
> seem to retain the subdirectory and link correctly. Maybe this was improved 
> as of recent Go releases. 
> In earlier versions of the Go tool, I would have either included a dummy 
> Go source file in the subdirectory, or just kept the non-go files in the 
> root with the rest of the Go source.
> Maybe someone else can clarify if this workflow has been improved?
>
>
> This declares the generic code available everywhere:
> https://github.com/periph/d2xx/blob/main/d2xx.go
>
> One set of files declare the import that is OS and CPU arch appropriate, 
> e.g. 
> https://github.com/periph/d2xx/blob/main/d2xx_linux_amd64.go
>
> Then another set of two files defines the OS specific code. In practice, 
> POSIX and windows;
> https://github.com/periph/d2xx/blob/main/d2xx_posix.go
> https://github.com/periph/d2xx/blob/main/d2xx_windows.go
>
> A CGO_ENABLED=0 build gets instead redirected to
> https://github.com/periph/d2xx/blob/main/d2xx_posix_no_cgo.go
> but not on windows because this package uses dynamic linking on windows. 
> Adapt as appropriate in your use case.
>
> I hope this helps.
>
> M-A
>
>
> Le lun. 16 oct. 2023, 21 h 13, Justin Israel  a 
> écrit :
>
>
>
> On Tuesday, October 17, 2023 at 10:40:33 AM UTC+13 Marc-Antoine Ruel wrote:
>
> I second Richard's suggestion. I used the same trick for 
> https://github.com/periph/d2xx. This git repository contains a minimalist 
> shim for the .a libraries and nothing else. It's designed to compile on any 
> platform. It falls back to a scaffolding if the corresponding .a library is 
> not present for the OS-arch combination or if CGO is disabled. It makes 
> testing easier.
>
> Then a proper package under https://github.com/periph/host/tree/main/ftdi 
> exposes a higher level abstraction for the user.
>
>
> With only headers and static libs in the thirdparty directory, is this 
> package "go-gettable"? 
> Will go make the subdirectory available in that case? It usually ignores 
> if there is no Go source files. 
>  
>
> M-A
>
> Le lun. 16 oct. 2023, à 14 h 48, Mike Schinkel  a 
> écrit :
>
> Hi Jan,
>
> I'm going to go out on a limb here and suggest looking at using Go's 
> `embed` feature?
>
> https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/
>
> I have not tried it with C libraries so there may be numerous reasons why 
> it may not work for your needs. For example, I do not know what is required 
> to *"install"* the libraries so that might be your sticking point that 
> embed would not address. But if building on the user's machine is the real 
> problem and you can distribute pre-build binaries then this might work for 
> you.  
>
> Basically Go reads a `//go:embed` comment, converts your files into Go 
> source code, compiles that code, and then provi

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Justin Israel]

On Wednesday, October 18, 2023 at 3:12:29 AM UTC+13 Marc-Antoine Ruel wrote:

Sorry I wasn't clear. The static libraries are in a subdirectory because 
the user should not care and these libraries are effectively third party 
code.


Actually, you were totally clear. Sorry if my comment didn't make sense. I 
get that the user doesn't need to worry about the third_party directory 
with the header and pre-built static libraries. And I saw how you arrange 
to pick up the right ones in the code. My point was that from previous 
experience, "go get" would prune away directories that do not contain go 
source code, after it performs the clone from the remote, into the module 
cache. So I would have expected that when someone uses your library as a 
dependency, it would have omitted the needed third_party directory, for not 
having any Go source files in it. But when I tested it just now, it does 
seem to retain the subdirectory and link correctly. Maybe this was improved 
as of recent Go releases. 
In earlier versions of the Go tool, I would have either included a dummy Go 
source file in the subdirectory, or just kept the non-go files in the root 
with the rest of the Go source.
Maybe someone else can clarify if this workflow has been improved?


This declares the generic code available everywhere:
https://github.com/periph/d2xx/blob/main/d2xx.go

One set of files declare the import that is OS and CPU arch appropriate, 
e.g. 
https://github.com/periph/d2xx/blob/main/d2xx_linux_amd64.go

Then another set of two files defines the OS specific code. In practice, 
POSIX and windows;
https://github.com/periph/d2xx/blob/main/d2xx_posix.go
https://github.com/periph/d2xx/blob/main/d2xx_windows.go

A CGO_ENABLED=0 build gets instead redirected to
https://github.com/periph/d2xx/blob/main/d2xx_posix_no_cgo.go
but not on windows because this package uses dynamic linking on windows. 
Adapt as appropriate in your use case.

I hope this helps.

M-A


Le lun. 16 oct. 2023, 21 h 13, Justin Israel  a écrit :



On Tuesday, October 17, 2023 at 10:40:33 AM UTC+13 Marc-Antoine Ruel wrote:

I second Richard's suggestion. I used the same trick for 
https://github.com/periph/d2xx. This git repository contains a minimalist 
shim for the .a libraries and nothing else. It's designed to compile on any 
platform. It falls back to a scaffolding if the corresponding .a library is 
not present for the OS-arch combination or if CGO is disabled. It makes 
testing easier.

Then a proper package under https://github.com/periph/host/tree/main/ftdi 
exposes a higher level abstraction for the user.


With only headers and static libs in the thirdparty directory, is this 
package "go-gettable"? 
Will go make the subdirectory available in that case? It usually ignores if 
there is no Go source files. 
 

M-A

Le lun. 16 oct. 2023, à 14 h 48, Mike Schinkel  a 
écrit :

Hi Jan,

I'm going to go out on a limb here and suggest looking at using Go's 
`embed` feature?

https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/

I have not tried it with C libraries so there may be numerous reasons why 
it may not work for your needs. For example, I do not know what is required 
to *"install"* the libraries so that might be your sticking point that 
embed would not address. But if building on the user's machine is the real 
problem and you can distribute pre-build binaries then this might work for 
you.  

Basically Go reads a `//go:embed` comment, converts your files into Go 
source code, compiles that code, and then provides a package that allows 
you to write those files to disk from an your Go app before you need to 
load the libraries.

Maybe this will work for you?  If yes, would love to hear back how it 
worked out.

-Mike
On Monday, October 16, 2023 at 8:40:54 AM UTC-4 Bruno Albuquerque wrote:

I guess you switched things here. Shared libraries (.so) need to be 
available at engine. Static libraries (.a) are bagged into the binary.

-Bruno


On Sun, Oct 15, 2023, 3:55 PM Jan  wrote:

Thanks Tamas, this is useful information. 

One of my libraries is using a `.a` library -- as opposed to `.so`, which 
is another level of complexity, since the library has to be available in 
runtime, not only in compile time -- and I'm going to follow your 
"incantation" suggestion.


On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote:

Neither I see a convenient way.
BUT if you add a .go file into the directories where your precompiled 
libraries live,
then "go get" will download them too (and only those dirs that have .go 
files in it).

So your next mission is to prepare the right #cgo CFLAGS LDFLAGS 
incantations to use those libraries.

Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):

Thanks Tamás, I may not be understanding correctly, but after taking a look 
at github.com/godror/godror, and the odpi subdirectory,
I see it is including all the `.c` files on the fly 

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Marc-Antoine Ruel]

Sorry I wasn't clear. The static libraries are in a subdirectory because
the user should not care and these libraries are effectively third party
code.

This declares the generic code available everywhere:
https://github.com/periph/d2xx/blob/main/d2xx.go

One set of files declare the import that is OS and CPU arch appropriate,
e.g.
https://github.com/periph/d2xx/blob/main/d2xx_linux_amd64.go

Then another set of two files defines the OS specific code. In practice,
POSIX and windows;
https://github.com/periph/d2xx/blob/main/d2xx_posix.go
https://github.com/periph/d2xx/blob/main/d2xx_windows.go

A CGO_ENABLED=0 build gets instead redirected to
https://github.com/periph/d2xx/blob/main/d2xx_posix_no_cgo.go
but not on windows because this package uses dynamic linking on windows.
Adapt as appropriate in your use case.

I hope this helps.

M-A


Le lun. 16 oct. 2023, 21 h 13, Justin Israel  a
écrit :

>
>
> On Tuesday, October 17, 2023 at 10:40:33 AM UTC+13 Marc-Antoine Ruel wrote:
>
> I second Richard's suggestion. I used the same trick for
> https://github.com/periph/d2xx. This git repository contains a minimalist
> shim for the .a libraries and nothing else. It's designed to compile on any
> platform. It falls back to a scaffolding if the corresponding .a library is
> not present for the OS-arch combination or if CGO is disabled. It makes
> testing easier.
>
> Then a proper package under https://github.com/periph/host/tree/main/ftdi
> exposes a higher level abstraction for the user.
>
>
> With only headers and static libs in the thirdparty directory, is this
> package "go-gettable"?
> Will go make the subdirectory available in that case? It usually ignores
> if there is no Go source files.
>
>
> M-A
>
> Le lun. 16 oct. 2023, à 14 h 48, Mike Schinkel  a
> écrit :
>
> Hi Jan,
>
> I'm going to go out on a limb here and suggest looking at using Go's
> `embed` feature?
>
> https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/
>
> I have not tried it with C libraries so there may be numerous reasons why
> it may not work for your needs. For example, I do not know what is required
> to *"install"* the libraries so that might be your sticking point that
> embed would not address. But if building on the user's machine is the real
> problem and you can distribute pre-build binaries then this might work for
> you.
>
> Basically Go reads a `//go:embed` comment, converts your files into Go
> source code, compiles that code, and then provides a package that allows
> you to write those files to disk from an your Go app before you need to
> load the libraries.
>
> Maybe this will work for you?  If yes, would love to hear back how it
> worked out.
>
> -Mike
> On Monday, October 16, 2023 at 8:40:54 AM UTC-4 Bruno Albuquerque wrote:
>
> I guess you switched things here. Shared libraries (.so) need to be
> available at engine. Static libraries (.a) are bagged into the binary.
>
> -Bruno
>
>
> On Sun, Oct 15, 2023, 3:55 PM Jan  wrote:
>
> Thanks Tamas, this is useful information.
>
> One of my libraries is using a `.a` library -- as opposed to `.so`, which
> is another level of complexity, since the library has to be available in
> runtime, not only in compile time -- and I'm going to follow your
> "incantation" suggestion.
>
>
> On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote:
>
> Neither I see a convenient way.
> BUT if you add a .go file into the directories where your precompiled
> libraries live,
> then "go get" will download them too (and only those dirs that have .go
> files in it).
>
> So your next mission is to prepare the right #cgo CFLAGS LDFLAGS
> incantations to use those libraries.
>
> Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):
>
> Thanks Tamás, I may not be understanding correctly, but after taking a
> look at github.com/godror/godror, and the odpi subdirectory,
> I see it is including all the `.c` files on the fly
> .
>
> A couple of reasons immediately come to mind, that make this not a
> generically valid option:
>
> * ODPI library is all C code (see src
> ) so it works
> including in Go: my dependencies are C++/Rust code, for which I write a
> small C wrapper (or for Rust just `extern "C"`). Also architecture
> dependent compilation is different in C++/C/Rust code ...
> * The C++ libraries I'm including have sub-dependencies of themselves (one
> of which is llvm). It uses Bazel to organize it, and to manually move all
> the required C++ files to a directory would be years of work :) Plus would
> require me to slave to maintain things in sync.
> * The C++ libraries take hours to compile ... I don't want to impose this
> to users of my libraries.
>
> I think the only way to work this out is distributing the pre-compiled
> C++/Rust libraries, so the Go simply refer to them (and we get the fast
> compilation times from Go). But then, how to b

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Justin Israel]

On Tuesday, October 17, 2023 at 10:40:33 AM UTC+13 Marc-Antoine Ruel wrote:

I second Richard's suggestion. I used the same trick for 
https://github.com/periph/d2xx. This git repository contains a minimalist 
shim for the .a libraries and nothing else. It's designed to compile on any 
platform. It falls back to a scaffolding if the corresponding .a library is 
not present for the OS-arch combination or if CGO is disabled. It makes 
testing easier.

Then a proper package under https://github.com/periph/host/tree/main/ftdi 
exposes a higher level abstraction for the user.


With only headers and static libs in the thirdparty directory, is this 
package "go-gettable"? 
Will go make the subdirectory available in that case? It usually ignores if 
there is no Go source files. 
 

M-A

Le lun. 16 oct. 2023, à 14 h 48, Mike Schinkel  a 
écrit :

Hi Jan,

I'm going to go out on a limb here and suggest looking at using Go's 
`embed` feature?

https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/

I have not tried it with C libraries so there may be numerous reasons why 
it may not work for your needs. For example, I do not know what is required 
to *"install"* the libraries so that might be your sticking point that 
embed would not address. But if building on the user's machine is the real 
problem and you can distribute pre-build binaries then this might work for 
you.  

Basically Go reads a `//go:embed` comment, converts your files into Go 
source code, compiles that code, and then provides a package that allows 
you to write those files to disk from an your Go app before you need to 
load the libraries.

Maybe this will work for you?  If yes, would love to hear back how it 
worked out.

-Mike
On Monday, October 16, 2023 at 8:40:54 AM UTC-4 Bruno Albuquerque wrote:

I guess you switched things here. Shared libraries (.so) need to be 
available at engine. Static libraries (.a) are bagged into the binary.

-Bruno


On Sun, Oct 15, 2023, 3:55 PM Jan  wrote:

Thanks Tamas, this is useful information. 

One of my libraries is using a `.a` library -- as opposed to `.so`, which 
is another level of complexity, since the library has to be available in 
runtime, not only in compile time -- and I'm going to follow your 
"incantation" suggestion.


On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote:

Neither I see a convenient way.
BUT if you add a .go file into the directories where your precompiled 
libraries live,
then "go get" will download them too (and only those dirs that have .go 
files in it).

So your next mission is to prepare the right #cgo CFLAGS LDFLAGS 
incantations to use those libraries.

Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):

Thanks Tamás, I may not be understanding correctly, but after taking a look 
at github.com/godror/godror, and the odpi subdirectory,
I see it is including all the `.c` files on the fly 
.

A couple of reasons immediately come to mind, that make this not a 
generically valid option:

* ODPI library is all C code (see src 
) so it works 
including in Go: my dependencies are C++/Rust code, for which I write a 
small C wrapper (or for Rust just `extern "C"`). Also architecture 
dependent compilation is different in C++/C/Rust code ...
* The C++ libraries I'm including have sub-dependencies of themselves (one 
of which is llvm). It uses Bazel to organize it, and to manually move all 
the required C++ files to a directory would be years of work :) Plus would 
require me to slave to maintain things in sync. 
* The C++ libraries take hours to compile ... I don't want to impose this 
to users of my libraries.

I think the only way to work this out is distributing the pre-compiled 
C++/Rust libraries, so the Go simply refer to them (and we get the fast 
compilation times from Go). But then, how to best distribute them in an 
automatic fashion, so that users don't need to one by one figure out how to 
install them (and clean up them later if they are no longer using...) ?

Or maybe there is another convenient way I'm not seeing ?


On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:

Can't you build (make go build for you) those libraries?
For example, see github.com/godror/godror just includes the sources of that 
third party library in an "odpi" subdir, and with
```
/*
#cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed

#include "dpi.c"

*/
import "C"
```
it is compiled automatically.


Caveat: for "go get" to download a directory, it must include a sth.go file 
("require.go" in the odpi/* subdirs).
But it may work that your precompiled libs in a subdir, with a mock .go 
file gets downloaded.
But how will it found by your lib/app ?

Tamás


Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):

Thanks Richard. Indeed, as you pointed out the downside is the bloating of 
the g

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Marc-Antoine Ruel]

I second Richard's suggestion. I used the same trick for
https://github.com/periph/d2xx. This git repository contains a minimalist
shim for the .a libraries and nothing else. It's designed to compile on any
platform. It falls back to a scaffolding if the corresponding .a library is
not present for the OS-arch combination or if CGO is disabled. It makes
testing easier.

Then a proper package under https://github.com/periph/host/tree/main/ftdi
exposes a higher level abstraction for the user.

M-A

Le lun. 16 oct. 2023, à 14 h 48, Mike Schinkel  a
écrit :

> Hi Jan,
>
> I'm going to go out on a limb here and suggest looking at using Go's
> `embed` feature?
>
> https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/
>
> I have not tried it with C libraries so there may be numerous reasons why
> it may not work for your needs. For example, I do not know what is required
> to *"install"* the libraries so that might be your sticking point that
> embed would not address. But if building on the user's machine is the real
> problem and you can distribute pre-build binaries then this might work for
> you.
>
> Basically Go reads a `//go:embed` comment, converts your files into Go
> source code, compiles that code, and then provides a package that allows
> you to write those files to disk from an your Go app before you need to
> load the libraries.
>
> Maybe this will work for you?  If yes, would love to hear back how it
> worked out.
>
> -Mike
> On Monday, October 16, 2023 at 8:40:54 AM UTC-4 Bruno Albuquerque wrote:
>
>> I guess you switched things here. Shared libraries (.so) need to be
>> available at engine. Static libraries (.a) are bagged into the binary.
>>
>> -Bruno
>>
>>
>> On Sun, Oct 15, 2023, 3:55 PM Jan  wrote:
>>
>>> Thanks Tamas, this is useful information.
>>>
>>> One of my libraries is using a `.a` library -- as opposed to `.so`,
>>> which is another level of complexity, since the library has to be available
>>> in runtime, not only in compile time -- and I'm going to follow your
>>> "incantation" suggestion.
>>>
>>>
>>> On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote:
>>>
 Neither I see a convenient way.
 BUT if you add a .go file into the directories where your precompiled
 libraries live,
 then "go get" will download them too (and only those dirs that have .go
 files in it).

 So your next mission is to prepare the right #cgo CFLAGS LDFLAGS
 incantations to use those libraries.

 Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):

> Thanks Tamás, I may not be understanding correctly, but after taking a
> look at github.com/godror/godror, and the odpi subdirectory,
> I see it is including all the `.c` files on the fly
> .
>
> A couple of reasons immediately come to mind, that make this not a
> generically valid option:
>
> * ODPI library is all C code (see src
> ) so it works
> including in Go: my dependencies are C++/Rust code, for which I write a
> small C wrapper (or for Rust just `extern "C"`). Also architecture
> dependent compilation is different in C++/C/Rust code ...
> * The C++ libraries I'm including have sub-dependencies of themselves
> (one of which is llvm). It uses Bazel to organize it, and to manually move
> all the required C++ files to a directory would be years of work :) Plus
> would require me to slave to maintain things in sync.
> * The C++ libraries take hours to compile ... I don't want to impose
> this to users of my libraries.
>
> I think the only way to work this out is distributing the pre-compiled
> C++/Rust libraries, so the Go simply refer to them (and we get the fast
> compilation times from Go). But then, how to best distribute them in an
> automatic fashion, so that users don't need to one by one figure out how 
> to
> install them (and clean up them later if they are no longer using...) ?
>
> Or maybe there is another convenient way I'm not seeing ?
>
>
> On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:
>
>> Can't you build (make go build for you) those libraries?
>> For example, see github.com/godror/godror just includes the sources
>> of that third party library in an "odpi" subdir, and with
>> ```
>> /*
>> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>>
>> #include "dpi.c"
>>
>> */
>> import "C"
>> ```
>> it is compiled automatically.
>>
>>
>> Caveat: for "go get" to download a directory, it must include a
>> sth.go file ("require.go" in the odpi/* subdirs).
>> But it may work that your precompiled libs in a subdir, with a mock
>> .go file gets downloaded.
>> But how will it found by your lib/app ?
>>
>> Tamás

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Mike Schinkel]

Hi Jan,

I'm going to go out on a limb here and suggest looking at using Go's 
`embed` feature?

https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/

I have not tried it with C libraries so there may be numerous reasons why 
it may not work for your needs. For example, I do not know what is required 
to *"install"* the libraries so that might be your sticking point that 
embed would not address. But if building on the user's machine is the real 
problem and you can distribute pre-build binaries then this might work for 
you.  

Basically Go reads a `//go:embed` comment, converts your files into Go 
source code, compiles that code, and then provides a package that allows 
you to write those files to disk from an your Go app before you need to 
load the libraries.

Maybe this will work for you?  If yes, would love to hear back how it 
worked out.

-Mike
On Monday, October 16, 2023 at 8:40:54 AM UTC-4 Bruno Albuquerque wrote:

> I guess you switched things here. Shared libraries (.so) need to be 
> available at engine. Static libraries (.a) are bagged into the binary.
>
> -Bruno
>
>
> On Sun, Oct 15, 2023, 3:55 PM Jan  wrote:
>
>> Thanks Tamas, this is useful information. 
>>
>> One of my libraries is using a `.a` library -- as opposed to `.so`, which 
>> is another level of complexity, since the library has to be available in 
>> runtime, not only in compile time -- and I'm going to follow your 
>> "incantation" suggestion.
>>
>>
>> On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote:
>>
>>> Neither I see a convenient way.
>>> BUT if you add a .go file into the directories where your precompiled 
>>> libraries live,
>>> then "go get" will download them too (and only those dirs that have .go 
>>> files in it).
>>>
>>> So your next mission is to prepare the right #cgo CFLAGS LDFLAGS 
>>> incantations to use those libraries.
>>>
>>> Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):
>>>
 Thanks Tamás, I may not be understanding correctly, but after taking a 
 look at github.com/godror/godror, and the odpi subdirectory,
 I see it is including all the `.c` files on the fly 
 .

 A couple of reasons immediately come to mind, that make this not a 
 generically valid option:

 * ODPI library is all C code (see src 
 ) so it works 
 including in Go: my dependencies are C++/Rust code, for which I write a 
 small C wrapper (or for Rust just `extern "C"`). Also architecture 
 dependent compilation is different in C++/C/Rust code ...
 * The C++ libraries I'm including have sub-dependencies of themselves 
 (one of which is llvm). It uses Bazel to organize it, and to manually move 
 all the required C++ files to a directory would be years of work :) Plus 
 would require me to slave to maintain things in sync. 
 * The C++ libraries take hours to compile ... I don't want to impose 
 this to users of my libraries.

 I think the only way to work this out is distributing the pre-compiled 
 C++/Rust libraries, so the Go simply refer to them (and we get the fast 
 compilation times from Go). But then, how to best distribute them in an 
 automatic fashion, so that users don't need to one by one figure out how 
 to 
 install them (and clean up them later if they are no longer using...) ?

 Or maybe there is another convenient way I'm not seeing ?


 On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:

> Can't you build (make go build for you) those libraries?
> For example, see github.com/godror/godror just includes the sources 
> of that third party library in an "odpi" subdir, and with
> ```
> /*
> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>
> #include "dpi.c"
>
> */
> import "C"
> ```
> it is compiled automatically.
>
>
> Caveat: for "go get" to download a directory, it must include a sth.go 
> file ("require.go" in the odpi/* subdirs).
> But it may work that your precompiled libs in a subdir, with a mock 
> .go file gets downloaded.
> But how will it found by your lib/app ?
>
> Tamás
>
>
> Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):
>
>> Thanks Richard. Indeed, as you pointed out the downside is the 
>> bloating of the git repo, but it makes sense.
>>
>> But does the user have to manually clone the repository and move the 
>> `.a` file to, let's say, /usr/local/lib, or does a simple `go get` 
>> magically does everything ?
>>
>>
>> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes 
>> wrote:
>>
>>> It isn't a great solution, but I currently include the built library 
>>> files and necessary headers in the git re

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Bruno Albuquerque]

I guess you switched things here. Shared libraries (.so) need to be
available at engine. Static libraries (.a) are bagged into the binary.

-Bruno


On Sun, Oct 15, 2023, 3:55 PM Jan  wrote:

> Thanks Tamas, this is useful information.
>
> One of my libraries is using a `.a` library -- as opposed to `.so`, which
> is another level of complexity, since the library has to be available in
> runtime, not only in compile time -- and I'm going to follow your
> "incantation" suggestion.
>
>
> On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote:
>
>> Neither I see a convenient way.
>> BUT if you add a .go file into the directories where your precompiled
>> libraries live,
>> then "go get" will download them too (and only those dirs that have .go
>> files in it).
>>
>> So your next mission is to prepare the right #cgo CFLAGS LDFLAGS
>> incantations to use those libraries.
>>
>> Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):
>>
>>> Thanks Tamás, I may not be understanding correctly, but after taking a
>>> look at github.com/godror/godror, and the odpi subdirectory,
>>> I see it is including all the `.c` files on the fly
>>> .
>>>
>>> A couple of reasons immediately come to mind, that make this not a
>>> generically valid option:
>>>
>>> * ODPI library is all C code (see src
>>> ) so it works
>>> including in Go: my dependencies are C++/Rust code, for which I write a
>>> small C wrapper (or for Rust just `extern "C"`). Also architecture
>>> dependent compilation is different in C++/C/Rust code ...
>>> * The C++ libraries I'm including have sub-dependencies of themselves
>>> (one of which is llvm). It uses Bazel to organize it, and to manually move
>>> all the required C++ files to a directory would be years of work :) Plus
>>> would require me to slave to maintain things in sync.
>>> * The C++ libraries take hours to compile ... I don't want to impose
>>> this to users of my libraries.
>>>
>>> I think the only way to work this out is distributing the pre-compiled
>>> C++/Rust libraries, so the Go simply refer to them (and we get the fast
>>> compilation times from Go). But then, how to best distribute them in an
>>> automatic fashion, so that users don't need to one by one figure out how to
>>> install them (and clean up them later if they are no longer using...) ?
>>>
>>> Or maybe there is another convenient way I'm not seeing ?
>>>
>>>
>>> On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:
>>>
 Can't you build (make go build for you) those libraries?
 For example, see github.com/godror/godror just includes the sources of
 that third party library in an "odpi" subdir, and with
 ```
 /*
 #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed

 #include "dpi.c"

 */
 import "C"
 ```
 it is compiled automatically.


 Caveat: for "go get" to download a directory, it must include a sth.go
 file ("require.go" in the odpi/* subdirs).
 But it may work that your precompiled libs in a subdir, with a mock .go
 file gets downloaded.
 But how will it found by your lib/app ?

 Tamás


 Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):

> Thanks Richard. Indeed, as you pointed out the downside is the
> bloating of the git repo, but it makes sense.
>
> But does the user have to manually clone the repository and move the
> `.a` file to, let's say, /usr/local/lib, or does a simple `go get`
> magically does everything ?
>
>
> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:
>
>> It isn't a great solution, but I currently include the built library
>> files and necessary headers in the git repo alongside the Go code. You 
>> can
>> see an example here
>> https://github.com/richardwilkes/unison/tree/main/internal/skia
>> where I include the skia library I built for use in my UI framework, 
>> unison.
>>
>> The main downside of this is bloating the git repo with the binary .a
>> and .dll files... but I've not found a better way to handle it. glfw, 
>> which
>> unison also depends upon, does something very similar.
>>
>> - Rich
>>
>> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>>
>>> hi,
>>>
>>> I'm developing a couple of ML framework/libraries for Go that depend
>>> on C/C++ code. Once C-libraries dependencies are installed, the CGO
>>> integration work great.
>>>
>>> Now, for end-users that just want to use these Go libraries, having
>>> to figure out how to manually build and install those C/C++/Rust
>>> dependencies is a hassle -- sadly each one with a different type of 
>>> build
>>> system.
>>>
>>> I offer pre-built `.tgz` files (for a limited se

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks Tamas, this is useful information. 

One of my libraries is using a `.a` library -- as opposed to `.so`, which 
is another level of complexity, since the library has to be available in 
runtime, not only in compile time -- and I'm going to follow your 
"incantation" suggestion.


On Sunday, October 15, 2023 at 7:55:55 AM UTC+2 Tamás Gulácsi wrote:

> Neither I see a convenient way.
> BUT if you add a .go file into the directories where your precompiled 
> libraries live,
> then "go get" will download them too (and only those dirs that have .go 
> files in it).
>
> So your next mission is to prepare the right #cgo CFLAGS LDFLAGS 
> incantations to use those libraries.
>
> Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):
>
>> Thanks Tamás, I may not be understanding correctly, but after taking a 
>> look at github.com/godror/godror, and the odpi subdirectory,
>> I see it is including all the `.c` files on the fly 
>> .
>>
>> A couple of reasons immediately come to mind, that make this not a 
>> generically valid option:
>>
>> * ODPI library is all C code (see src 
>> ) so it works 
>> including in Go: my dependencies are C++/Rust code, for which I write a 
>> small C wrapper (or for Rust just `extern "C"`). Also architecture 
>> dependent compilation is different in C++/C/Rust code ...
>> * The C++ libraries I'm including have sub-dependencies of themselves 
>> (one of which is llvm). It uses Bazel to organize it, and to manually move 
>> all the required C++ files to a directory would be years of work :) Plus 
>> would require me to slave to maintain things in sync. 
>> * The C++ libraries take hours to compile ... I don't want to impose this 
>> to users of my libraries.
>>
>> I think the only way to work this out is distributing the pre-compiled 
>> C++/Rust libraries, so the Go simply refer to them (and we get the fast 
>> compilation times from Go). But then, how to best distribute them in an 
>> automatic fashion, so that users don't need to one by one figure out how to 
>> install them (and clean up them later if they are no longer using...) ?
>>
>> Or maybe there is another convenient way I'm not seeing ?
>>
>>
>> On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:
>>
>>> Can't you build (make go build for you) those libraries?
>>> For example, see github.com/godror/godror just includes the sources of 
>>> that third party library in an "odpi" subdir, and with
>>> ```
>>> /*
>>> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>>>
>>> #include "dpi.c"
>>>
>>> */
>>> import "C"
>>> ```
>>> it is compiled automatically.
>>>
>>>
>>> Caveat: for "go get" to download a directory, it must include a sth.go 
>>> file ("require.go" in the odpi/* subdirs).
>>> But it may work that your precompiled libs in a subdir, with a mock .go 
>>> file gets downloaded.
>>> But how will it found by your lib/app ?
>>>
>>> Tamás
>>>
>>>
>>> Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):
>>>
 Thanks Richard. Indeed, as you pointed out the downside is the bloating 
 of the git repo, but it makes sense.

 But does the user have to manually clone the repository and move the 
 `.a` file to, let's say, /usr/local/lib, or does a simple `go get` 
 magically does everything ?


 On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:

> It isn't a great solution, but I currently include the built library 
> files and necessary headers in the git repo alongside the Go code. You 
> can 
> see an example here 
> https://github.com/richardwilkes/unison/tree/main/internal/skia where 
> I include the skia library I built for use in my UI framework, unison.
>
> The main downside of this is bloating the git repo with the binary .a 
> and .dll files... but I've not found a better way to handle it. glfw, 
> which 
> unison also depends upon, does something very similar.
>
> - Rich
>
> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>
>> hi,
>>
>> I'm developing a couple of ML framework/libraries for Go that depend 
>> on C/C++ code. Once C-libraries dependencies are installed, the CGO 
>> integration work great.
>>
>> Now, for end-users that just want to use these Go libraries, having 
>> to figure out how to manually build and install those C/C++/Rust 
>> dependencies is a hassle -- sadly each one with a different type of 
>> build 
>> system.
>>
>> I offer pre-built `.tgz` files (for a limited set of architectures) 
>> with the required `.h` and `.a/.so` files along the releases, which 
>> facilitates. But it's still a hassle to install -- and no auto-uninstall 
>> if 
>> someone is no longer using the Go module.
>>
>> I was wondering if others have 

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Tamás Gulácsi]

Neither I see a convenient way.
BUT if you add a .go file into the directories where your precompiled 
libraries live,
then "go get" will download them too (and only those dirs that have .go 
files in it).

So your next mission is to prepare the right #cgo CFLAGS LDFLAGS 
incantations to use those libraries.

Jan a következőt írta (2023. október 14., szombat, 8:37:48 UTC+2):

> Thanks Tamás, I may not be understanding correctly, but after taking a 
> look at github.com/godror/godror, and the odpi subdirectory,
> I see it is including all the `.c` files on the fly 
> .
>
> A couple of reasons immediately come to mind, that make this not a 
> generically valid option:
>
> * ODPI library is all C code (see src 
> ) so it works 
> including in Go: my dependencies are C++/Rust code, for which I write a 
> small C wrapper (or for Rust just `extern "C"`). Also architecture 
> dependent compilation is different in C++/C/Rust code ...
> * The C++ libraries I'm including have sub-dependencies of themselves (one 
> of which is llvm). It uses Bazel to organize it, and to manually move all 
> the required C++ files to a directory would be years of work :) Plus would 
> require me to slave to maintain things in sync. 
> * The C++ libraries take hours to compile ... I don't want to impose this 
> to users of my libraries.
>
> I think the only way to work this out is distributing the pre-compiled 
> C++/Rust libraries, so the Go simply refer to them (and we get the fast 
> compilation times from Go). But then, how to best distribute them in an 
> automatic fashion, so that users don't need to one by one figure out how to 
> install them (and clean up them later if they are no longer using...) ?
>
> Or maybe there is another convenient way I'm not seeing ?
>
>
> On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:
>
>> Can't you build (make go build for you) those libraries?
>> For example, see github.com/godror/godror just includes the sources of 
>> that third party library in an "odpi" subdir, and with
>> ```
>> /*
>> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>>
>> #include "dpi.c"
>>
>> */
>> import "C"
>> ```
>> it is compiled automatically.
>>
>>
>> Caveat: for "go get" to download a directory, it must include a sth.go 
>> file ("require.go" in the odpi/* subdirs).
>> But it may work that your precompiled libs in a subdir, with a mock .go 
>> file gets downloaded.
>> But how will it found by your lib/app ?
>>
>> Tamás
>>
>>
>> Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):
>>
>>> Thanks Richard. Indeed, as you pointed out the downside is the bloating 
>>> of the git repo, but it makes sense.
>>>
>>> But does the user have to manually clone the repository and move the 
>>> `.a` file to, let's say, /usr/local/lib, or does a simple `go get` 
>>> magically does everything ?
>>>
>>>
>>> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:
>>>
 It isn't a great solution, but I currently include the built library 
 files and necessary headers in the git repo alongside the Go code. You can 
 see an example here 
 https://github.com/richardwilkes/unison/tree/main/internal/skia where 
 I include the skia library I built for use in my UI framework, unison.

 The main downside of this is bloating the git repo with the binary .a 
 and .dll files... but I've not found a better way to handle it. glfw, 
 which 
 unison also depends upon, does something very similar.

 - Rich

 On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:

> hi,
>
> I'm developing a couple of ML framework/libraries for Go that depend 
> on C/C++ code. Once C-libraries dependencies are installed, the CGO 
> integration work great.
>
> Now, for end-users that just want to use these Go libraries, having to 
> figure out how to manually build and install those C/C++/Rust 
> dependencies 
> is a hassle -- sadly each one with a different type of build system.
>
> I offer pre-built `.tgz` files (for a limited set of architectures) 
> with the required `.h` and `.a/.so` files along the releases, which 
> facilitates. But it's still a hassle to install -- and no auto-uninstall 
> if 
> someone is no longer using the Go module.
>
> I was wondering if others have figured out how to handle this in a 
> nicer way.  Is there a recommended way to distribute prebuilt CGO 
> dependencies ?
>
> I like how Python wheels (`.whl` files) solve the issue, by including 
> the pre-built libraries in a sub-directory of the python library 
> installation. I was hoping there would be a similar way (maybe with a 
> separate tool) to integrate with `go.mod`. 
>
> Any pointers, thoughts or suggestions are very appreciated.
>
>

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks Tamás, I may not be understanding correctly, but after taking a look 
at github.com/godror/godror, and the odpi subdirectory,
I see it is including all the `.c` files on the fly 
.

A couple of reasons immediately come to mind, that make this not a 
generically valid option:

* ODPI library is all C code (see src 
) so it works 
including in Go: my dependencies are C++/Rust code, for which I write a 
small C wrapper (or for Rust just `extern "C"`). Also architecture 
dependent compilation is different in C++/C/Rust code ...
* The C++ libraries I'm including have sub-dependencies of themselves (one 
of which is llvm). It uses Bazel to organize it, and to manually move all 
the required C++ files to a directory would be years of work :) Plus would 
require me to slave to maintain things in sync. 
* The C++ libraries take hours to compile ... I don't want to impose this 
to users of my libraries.

I think the only way to work this out is distributing the pre-compiled 
C++/Rust libraries, so the Go simply refer to them (and we get the fast 
compilation times from Go). But then, how to best distribute them in an 
automatic fashion, so that users don't need to one by one figure out how to 
install them (and clean up them later if they are no longer using...) ?

Or maybe there is another convenient way I'm not seeing ?


On Thursday, October 12, 2023 at 6:39:34 PM UTC+2 Tamás Gulácsi wrote:

> Can't you build (make go build for you) those libraries?
> For example, see github.com/godror/godror just includes the sources of 
> that third party library in an "odpi" subdir, and with
> ```
> /*
> #cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed
>
> #include "dpi.c"
>
> */
> import "C"
> ```
> it is compiled automatically.
>
>
> Caveat: for "go get" to download a directory, it must include a sth.go 
> file ("require.go" in the odpi/* subdirs).
> But it may work that your precompiled libs in a subdir, with a mock .go 
> file gets downloaded.
> But how will it found by your lib/app ?
>
> Tamás
>
>
> Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):
>
>> Thanks Richard. Indeed, as you pointed out the downside is the bloating 
>> of the git repo, but it makes sense.
>>
>> But does the user have to manually clone the repository and move the `.a` 
>> file to, let's say, /usr/local/lib, or does a simple `go get` magically 
>> does everything ?
>>
>>
>> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:
>>
>>> It isn't a great solution, but I currently include the built library 
>>> files and necessary headers in the git repo alongside the Go code. You can 
>>> see an example here 
>>> https://github.com/richardwilkes/unison/tree/main/internal/skia where I 
>>> include the skia library I built for use in my UI framework, unison.
>>>
>>> The main downside of this is bloating the git repo with the binary .a 
>>> and .dll files... but I've not found a better way to handle it. glfw, which 
>>> unison also depends upon, does something very similar.
>>>
>>> - Rich
>>>
>>> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>>>
 hi,

 I'm developing a couple of ML framework/libraries for Go that depend on 
 C/C++ code. Once C-libraries dependencies are installed, the CGO 
 integration work great.

 Now, for end-users that just want to use these Go libraries, having to 
 figure out how to manually build and install those C/C++/Rust dependencies 
 is a hassle -- sadly each one with a different type of build system.

 I offer pre-built `.tgz` files (for a limited set of architectures) 
 with the required `.h` and `.a/.so` files along the releases, which 
 facilitates. But it's still a hassle to install -- and no auto-uninstall 
 if 
 someone is no longer using the Go module.

 I was wondering if others have figured out how to handle this in a 
 nicer way.  Is there a recommended way to distribute prebuilt CGO 
 dependencies ?

 I like how Python wheels (`.whl` files) solve the issue, by including 
 the pre-built libraries in a sub-directory of the python library 
 installation. I was hoping there would be a similar way (maybe with a 
 separate tool) to integrate with `go.mod`. 

 Any pointers, thoughts or suggestions are very appreciated.

 Thanks!
 Jan

 ps: While searching I saw similar questions, but none that exactly 
 answered this aspect of distribution. Just in case, apologies if it's a 
 duplicate question.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nut

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Tamás Gulácsi]

Can't you build (make go build for you) those libraries?
For example, see github.com/godror/godror just includes the sources of that 
third party library in an "odpi" subdir, and with
```
/*
#cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed

#include "dpi.c"

*/
import "C"
```
it is compiled automatically.


Caveat: for "go get" to download a directory, it must include a sth.go file 
("require.go" in the odpi/* subdirs).
But it may work that your precompiled libs in a subdir, with a mock .go 
file gets downloaded.
But how will it found by your lib/app ?

Tamás


Jan a következőt írta (2023. október 12., csütörtök, 8:14:41 UTC+2):

> Thanks Richard. Indeed, as you pointed out the downside is the bloating of 
> the git repo, but it makes sense.
>
> But does the user have to manually clone the repository and move the `.a` 
> file to, let's say, /usr/local/lib, or does a simple `go get` magically 
> does everything ?
>
>
> On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:
>
>> It isn't a great solution, but I currently include the built library 
>> files and necessary headers in the git repo alongside the Go code. You can 
>> see an example here 
>> https://github.com/richardwilkes/unison/tree/main/internal/skia where I 
>> include the skia library I built for use in my UI framework, unison.
>>
>> The main downside of this is bloating the git repo with the binary .a and 
>> .dll files... but I've not found a better way to handle it. glfw, which 
>> unison also depends upon, does something very similar.
>>
>> - Rich
>>
>> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>>
>>> hi,
>>>
>>> I'm developing a couple of ML framework/libraries for Go that depend on 
>>> C/C++ code. Once C-libraries dependencies are installed, the CGO 
>>> integration work great.
>>>
>>> Now, for end-users that just want to use these Go libraries, having to 
>>> figure out how to manually build and install those C/C++/Rust dependencies 
>>> is a hassle -- sadly each one with a different type of build system.
>>>
>>> I offer pre-built `.tgz` files (for a limited set of architectures) with 
>>> the required `.h` and `.a/.so` files along the releases, which facilitates. 
>>> But it's still a hassle to install -- and no auto-uninstall if someone is 
>>> no longer using the Go module.
>>>
>>> I was wondering if others have figured out how to handle this in a nicer 
>>> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>>>
>>> I like how Python wheels (`.whl` files) solve the issue, by including 
>>> the pre-built libraries in a sub-directory of the python library 
>>> installation. I was hoping there would be a similar way (maybe with a 
>>> separate tool) to integrate with `go.mod`. 
>>>
>>> Any pointers, thoughts or suggestions are very appreciated.
>>>
>>> Thanks!
>>> Jan
>>>
>>> ps: While searching I saw similar questions, but none that exactly 
>>> answered this aspect of distribution. Just in case, apologies if it's a 
>>> duplicate question.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1624fe5e-ec87-4022-91f5-d9990594c183n%40googlegroups.com.

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks Tim, but my goal is to make things transparent and easy to the users 
of my libraries. So I don't want to force them to run Makefiles (in my case 
large Bazel builds for one project, Makefile for the other), install 
C++/Rust build tools, deal with dockers, etc. Ideally, `go get` would just 
take care of everything. If that is not possible, I'm wondering what is the 
easiest next thing.

The reasoning is that a largish project often depends on dozens of 
libraries, if we need to do this kind of stuff for every library, its too 
much friction. Go tools are awesome ('go get' works so well for me) when 
there are no CGO dependencies ... alas, not everything is implemented in Go.

Indeed, pre-built binaries is platform dependent -- and requires separate 
files/tgz per architecture, as you point out. How to have these 
dependencies installed, and picked up by go compiler automatically (setting 
up of CGO's CFLAG, LDFLAG LD_LIBRARY_PATH, etc), and be removed if no 
longer used ? With minimum user headache/work...

cheers
Jan

On Thursday, October 12, 2023 at 3:16:58 AM UTC+2 Tim Casey wrote:

>
>
> I would put down Makefile includes for the supported targets and build 
> each target at a time, as cross compiled shared libraries.  This is easier 
> for linux, harder for windows.
> But, you can build/export from docker containers as part of the overall 
> build process to allow simulated build environments.   At some point, you 
> may come to a dependency in which you have to build the included C code for 
> it to run in platform dependent libraries.  At this point, you probably 
> will want to build on the environment as you go.  You can cross target *.o, 
> but build the *.so on the specific target.
>
> You could have separate repos and distribute the C code from a dependency 
> point of view.  This would be a tgz of 'shared/$ARCH/$COMPILED_LIB", or 
> some such.
>
> On Wed, Oct 11, 2023 at 5:29 PM Richard Wilkes  
> wrote:
>
>> It isn't a great solution, but I currently include the built library 
>> files and necessary headers in the git repo alongside the Go code. You can 
>> see an example here 
>> https://github.com/richardwilkes/unison/tree/main/internal/skia where I 
>> include the skia library I built for use in my UI framework, unison.
>>
>> The main downside of this is bloating the git repo with the binary .a and 
>> .dll files... but I've not found a better way to handle it. glfw, which 
>> unison also depends upon, does something very similar.
>>
>> - Rich
>>
>> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>>
>>> hi,
>>>
>>> I'm developing a couple of ML framework/libraries for Go that depend on 
>>> C/C++ code. Once C-libraries dependencies are installed, the CGO 
>>> integration work great.
>>>
>>> Now, for end-users that just want to use these Go libraries, having to 
>>> figure out how to manually build and install those C/C++/Rust dependencies 
>>> is a hassle -- sadly each one with a different type of build system.
>>>
>>> I offer pre-built `.tgz` files (for a limited set of architectures) with 
>>> the required `.h` and `.a/.so` files along the releases, which facilitates. 
>>> But it's still a hassle to install -- and no auto-uninstall if someone is 
>>> no longer using the Go module.
>>>
>>> I was wondering if others have figured out how to handle this in a nicer 
>>> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>>>
>>> I like how Python wheels (`.whl` files) solve the issue, by including 
>>> the pre-built libraries in a sub-directory of the python library 
>>> installation. I was hoping there would be a similar way (maybe with a 
>>> separate tool) to integrate with `go.mod`. 
>>>
>>> Any pointers, thoughts or suggestions are very appreciated.
>>>
>>> Thanks!
>>> Jan
>>>
>>> ps: While searching I saw similar questions, but none that exactly 
>>> answered this aspect of distribution. Just in case, apologies if it's a 
>>> duplicate question.
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/3b03b9e1-655e-452a-9b97-e9471b9b9dc1n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a5feedde-9a18-4b29-92a0-6f76d96f01c4n%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks Richard. Indeed, as you pointed out the downside is the bloating of 
the git repo, but it makes sense.

But does the user have to manually clone the repository and move the `.a` 
file to, let's say, /usr/local/lib, or does a simple `go get` magically 
does everything ?


On Thursday, October 12, 2023 at 2:29:21 AM UTC+2 Richard Wilkes wrote:

> It isn't a great solution, but I currently include the built library files 
> and necessary headers in the git repo alongside the Go code. You can see an 
> example here 
> https://github.com/richardwilkes/unison/tree/main/internal/skia where I 
> include the skia library I built for use in my UI framework, unison.
>
> The main downside of this is bloating the git repo with the binary .a and 
> .dll files... but I've not found a better way to handle it. glfw, which 
> unison also depends upon, does something very similar.
>
> - Rich
>
> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>
>> hi,
>>
>> I'm developing a couple of ML framework/libraries for Go that depend on 
>> C/C++ code. Once C-libraries dependencies are installed, the CGO 
>> integration work great.
>>
>> Now, for end-users that just want to use these Go libraries, having to 
>> figure out how to manually build and install those C/C++/Rust dependencies 
>> is a hassle -- sadly each one with a different type of build system.
>>
>> I offer pre-built `.tgz` files (for a limited set of architectures) with 
>> the required `.h` and `.a/.so` files along the releases, which facilitates. 
>> But it's still a hassle to install -- and no auto-uninstall if someone is 
>> no longer using the Go module.
>>
>> I was wondering if others have figured out how to handle this in a nicer 
>> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>>
>> I like how Python wheels (`.whl` files) solve the issue, by including the 
>> pre-built libraries in a sub-directory of the python library installation. 
>> I was hoping there would be a similar way (maybe with a separate tool) to 
>> integrate with `go.mod`. 
>>
>> Any pointers, thoughts or suggestions are very appreciated.
>>
>> Thanks!
>> Jan
>>
>> ps: While searching I saw similar questions, but none that exactly 
>> answered this aspect of distribution. Just in case, apologies if it's a 
>> duplicate question.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4e84cdbf-15c4-42bf-a6f3-67aec78a5dc1n%40googlegroups.com.

Re: [go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Tim Casey]

I would put down Makefile includes for the supported targets and build each
target at a time, as cross compiled shared libraries.  This is easier for
linux, harder for windows.
But, you can build/export from docker containers as part of the overall
build process to allow simulated build environments.   At some point, you
may come to a dependency in which you have to build the included C code for
it to run in platform dependent libraries.  At this point, you probably
will want to build on the environment as you go.  You can cross target *.o,
but build the *.so on the specific target.

You could have separate repos and distribute the C code from a dependency
point of view.  This would be a tgz of 'shared/$ARCH/$COMPILED_LIB", or
some such.

On Wed, Oct 11, 2023 at 5:29 PM Richard Wilkes 
wrote:

> It isn't a great solution, but I currently include the built library files
> and necessary headers in the git repo alongside the Go code. You can see an
> example here
> https://github.com/richardwilkes/unison/tree/main/internal/skia where I
> include the skia library I built for use in my UI framework, unison.
>
> The main downside of this is bloating the git repo with the binary .a and
> .dll files... but I've not found a better way to handle it. glfw, which
> unison also depends upon, does something very similar.
>
> - Rich
>
> On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:
>
>> hi,
>>
>> I'm developing a couple of ML framework/libraries for Go that depend on
>> C/C++ code. Once C-libraries dependencies are installed, the CGO
>> integration work great.
>>
>> Now, for end-users that just want to use these Go libraries, having to
>> figure out how to manually build and install those C/C++/Rust dependencies
>> is a hassle -- sadly each one with a different type of build system.
>>
>> I offer pre-built `.tgz` files (for a limited set of architectures) with
>> the required `.h` and `.a/.so` files along the releases, which facilitates.
>> But it's still a hassle to install -- and no auto-uninstall if someone is
>> no longer using the Go module.
>>
>> I was wondering if others have figured out how to handle this in a nicer
>> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>>
>> I like how Python wheels (`.whl` files) solve the issue, by including the
>> pre-built libraries in a sub-directory of the python library installation.
>> I was hoping there would be a similar way (maybe with a separate tool) to
>> integrate with `go.mod`.
>>
>> Any pointers, thoughts or suggestions are very appreciated.
>>
>> Thanks!
>> Jan
>>
>> ps: While searching I saw similar questions, but none that exactly
>> answered this aspect of distribution. Just in case, apologies if it's a
>> duplicate question.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/3b03b9e1-655e-452a-9b97-e9471b9b9dc1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMyFqkRai48nKKFvqPN1ALU0LipzTgJe0YEPFmqx1Dv0T9RkTA%40mail.gmail.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Richard Wilkes]

It isn't a great solution, but I currently include the built library files 
and necessary headers in the git repo alongside the Go code. You can see an 
example here 
https://github.com/richardwilkes/unison/tree/main/internal/skia where I 
include the skia library I built for use in my UI framework, unison.

The main downside of this is bloating the git repo with the binary .a and 
.dll files... but I've not found a better way to handle it. glfw, which 
unison also depends upon, does something very similar.

- Rich

On Wednesday, October 11, 2023 at 2:58:23 AM UTC-7 Jan wrote:

> hi,
>
> I'm developing a couple of ML framework/libraries for Go that depend on 
> C/C++ code. Once C-libraries dependencies are installed, the CGO 
> integration work great.
>
> Now, for end-users that just want to use these Go libraries, having to 
> figure out how to manually build and install those C/C++/Rust dependencies 
> is a hassle -- sadly each one with a different type of build system.
>
> I offer pre-built `.tgz` files (for a limited set of architectures) with 
> the required `.h` and `.a/.so` files along the releases, which facilitates. 
> But it's still a hassle to install -- and no auto-uninstall if someone is 
> no longer using the Go module.
>
> I was wondering if others have figured out how to handle this in a nicer 
> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>
> I like how Python wheels (`.whl` files) solve the issue, by including the 
> pre-built libraries in a sub-directory of the python library installation. 
> I was hoping there would be a similar way (maybe with a separate tool) to 
> integrate with `go.mod`. 
>
> Any pointers, thoughts or suggestions are very appreciated.
>
> Thanks!
> Jan
>
> ps: While searching I saw similar questions, but none that exactly 
> answered this aspect of distribution. Just in case, apologies if it's a 
> duplicate question.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3b03b9e1-655e-452a-9b97-e9471b9b9dc1n%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Yep, I want a solution on how to distribute C-dependencies of libraries 
that use CGO, so end users don't have to manually sort out dependencies of 
various libraries.

Thanks for trying though.

On Wednesday, October 11, 2023 at 1:10:12 PM UTC+2 Peter Galbavy wrote:

> Hang on - I missed your original statement that you are building 
> *libraries* yourself. I am building executables. Sorry for the confusion 
> and my too-fast scan reading!
>
> On Wednesday, 11 October 2023 at 12:08:53 UTC+1 Peter Galbavy wrote:
>
>> If you link statically, e.g. from my Dockerfile:
>>
>>   go build -tags netgo,osusergo --ldflags '-s -w -linkmode external 
>> -extldflags=-static'
>>
>> then the libraries are in the single binary. I also then use "upx" to 
>> compress the resulting binaries to save a little space. The build tags are 
>> there to not require GLIBC for user and dns lookups.
>>
>>
>>
>> On Wednesday, 11 October 2023 at 11:43:20 UTC+1 Jan wrote:
>>
>>> Edit: ...so folks using your library don't need to install them manually 
>>> ? What about uninstalling ?
>>> On Wednesday, October 11, 2023 at 12:41:40 PM UTC+2 Jan wrote:
>>>
 Thanks Peter, but I don't follow ... I also use docker to build the 
 pre-built binaries.

 But how do you distribute them, so folks using your library don't think 
 to install them ?

 On Wednesday, October 11, 2023 at 12:32:57 PM UTC+2 Peter Galbavy wrote:

> I use a docker container (golang:bullseye seems the most compatible) 
> and build static binaries. This is feasible as long as your dependencies 
> do 
> not require dynamic loading of dependencies.
>
> Poor (my Dockerfile skills) example here: 
> https://github.com/ITRS-Group/cordial/blob/main/Dockerfile
>
> Peter
>
> On Wednesday, 11 October 2023 at 10:58:23 UTC+1 Jan wrote:
>
>> hi,
>>
>> I'm developing a couple of ML framework/libraries for Go that depend 
>> on C/C++ code. Once C-libraries dependencies are installed, the CGO 
>> integration work great.
>>
>> Now, for end-users that just want to use these Go libraries, having 
>> to figure out how to manually build and install those C/C++/Rust 
>> dependencies is a hassle -- sadly each one with a different type of 
>> build 
>> system.
>>
>> I offer pre-built `.tgz` files (for a limited set of architectures) 
>> with the required `.h` and `.a/.so` files along the releases, which 
>> facilitates. But it's still a hassle to install -- and no auto-uninstall 
>> if 
>> someone is no longer using the Go module.
>>
>> I was wondering if others have figured out how to handle this in a 
>> nicer way.  Is there a recommended way to distribute prebuilt CGO 
>> dependencies ?
>>
>> I like how Python wheels (`.whl` files) solve the issue, by including 
>> the pre-built libraries in a sub-directory of the python library 
>> installation. I was hoping there would be a similar way (maybe with a 
>> separate tool) to integrate with `go.mod`. 
>>
>> Any pointers, thoughts or suggestions are very appreciated.
>>
>> Thanks!
>> Jan
>>
>> ps: While searching I saw similar questions, but none that exactly 
>> answered this aspect of distribution. Just in case, apologies if it's a 
>> duplicate question.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/52054323-ac00-4b15-92f7-7c95cb84b1a1n%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Peter Galbavy]

Hang on - I missed your original statement that you are building 
*libraries* yourself. I am building executables. Sorry for the confusion 
and my too-fast scan reading!

On Wednesday, 11 October 2023 at 12:08:53 UTC+1 Peter Galbavy wrote:

> If you link statically, e.g. from my Dockerfile:
>
>   go build -tags netgo,osusergo --ldflags '-s -w -linkmode external 
> -extldflags=-static'
>
> then the libraries are in the single binary. I also then use "upx" to 
> compress the resulting binaries to save a little space. The build tags are 
> there to not require GLIBC for user and dns lookups.
>
>
>
> On Wednesday, 11 October 2023 at 11:43:20 UTC+1 Jan wrote:
>
>> Edit: ...so folks using your library don't need to install them manually 
>> ? What about uninstalling ?
>> On Wednesday, October 11, 2023 at 12:41:40 PM UTC+2 Jan wrote:
>>
>>> Thanks Peter, but I don't follow ... I also use docker to build the 
>>> pre-built binaries.
>>>
>>> But how do you distribute them, so folks using your library don't think 
>>> to install them ?
>>>
>>> On Wednesday, October 11, 2023 at 12:32:57 PM UTC+2 Peter Galbavy wrote:
>>>
 I use a docker container (golang:bullseye seems the most compatible) 
 and build static binaries. This is feasible as long as your dependencies 
 do 
 not require dynamic loading of dependencies.

 Poor (my Dockerfile skills) example here: 
 https://github.com/ITRS-Group/cordial/blob/main/Dockerfile

 Peter

 On Wednesday, 11 October 2023 at 10:58:23 UTC+1 Jan wrote:

> hi,
>
> I'm developing a couple of ML framework/libraries for Go that depend 
> on C/C++ code. Once C-libraries dependencies are installed, the CGO 
> integration work great.
>
> Now, for end-users that just want to use these Go libraries, having to 
> figure out how to manually build and install those C/C++/Rust 
> dependencies 
> is a hassle -- sadly each one with a different type of build system.
>
> I offer pre-built `.tgz` files (for a limited set of architectures) 
> with the required `.h` and `.a/.so` files along the releases, which 
> facilitates. But it's still a hassle to install -- and no auto-uninstall 
> if 
> someone is no longer using the Go module.
>
> I was wondering if others have figured out how to handle this in a 
> nicer way.  Is there a recommended way to distribute prebuilt CGO 
> dependencies ?
>
> I like how Python wheels (`.whl` files) solve the issue, by including 
> the pre-built libraries in a sub-directory of the python library 
> installation. I was hoping there would be a similar way (maybe with a 
> separate tool) to integrate with `go.mod`. 
>
> Any pointers, thoughts or suggestions are very appreciated.
>
> Thanks!
> Jan
>
> ps: While searching I saw similar questions, but none that exactly 
> answered this aspect of distribution. Just in case, apologies if it's a 
> duplicate question.
>


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c31db183-c69a-43e4-bee3-8f113d5e37c4n%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Peter Galbavy]

If you link statically, e.g. from my Dockerfile:

  go build -tags netgo,osusergo --ldflags '-s -w -linkmode external 
-extldflags=-static'

then the libraries are in the single binary. I also then use "upx" to 
compress the resulting binaries to save a little space. The build tags are 
there to not require GLIBC for user and dns lookups.



On Wednesday, 11 October 2023 at 11:43:20 UTC+1 Jan wrote:

> Edit: ...so folks using your library don't need to install them manually ? 
> What about uninstalling ?
> On Wednesday, October 11, 2023 at 12:41:40 PM UTC+2 Jan wrote:
>
>> Thanks Peter, but I don't follow ... I also use docker to build the 
>> pre-built binaries.
>>
>> But how do you distribute them, so folks using your library don't think 
>> to install them ?
>>
>> On Wednesday, October 11, 2023 at 12:32:57 PM UTC+2 Peter Galbavy wrote:
>>
>>> I use a docker container (golang:bullseye seems the most compatible) and 
>>> build static binaries. This is feasible as long as your dependencies do not 
>>> require dynamic loading of dependencies.
>>>
>>> Poor (my Dockerfile skills) example here: 
>>> https://github.com/ITRS-Group/cordial/blob/main/Dockerfile
>>>
>>> Peter
>>>
>>> On Wednesday, 11 October 2023 at 10:58:23 UTC+1 Jan wrote:
>>>
 hi,

 I'm developing a couple of ML framework/libraries for Go that depend on 
 C/C++ code. Once C-libraries dependencies are installed, the CGO 
 integration work great.

 Now, for end-users that just want to use these Go libraries, having to 
 figure out how to manually build and install those C/C++/Rust dependencies 
 is a hassle -- sadly each one with a different type of build system.

 I offer pre-built `.tgz` files (for a limited set of architectures) 
 with the required `.h` and `.a/.so` files along the releases, which 
 facilitates. But it's still a hassle to install -- and no auto-uninstall 
 if 
 someone is no longer using the Go module.

 I was wondering if others have figured out how to handle this in a 
 nicer way.  Is there a recommended way to distribute prebuilt CGO 
 dependencies ?

 I like how Python wheels (`.whl` files) solve the issue, by including 
 the pre-built libraries in a sub-directory of the python library 
 installation. I was hoping there would be a similar way (maybe with a 
 separate tool) to integrate with `go.mod`. 

 Any pointers, thoughts or suggestions are very appreciated.

 Thanks!
 Jan

 ps: While searching I saw similar questions, but none that exactly 
 answered this aspect of distribution. Just in case, apologies if it's a 
 duplicate question.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/dd567a9b-21a5-4621-a85d-06b33e70ad31n%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Edit: ...so folks using your library don't need to install them manually ? 
What about uninstalling ?
On Wednesday, October 11, 2023 at 12:41:40 PM UTC+2 Jan wrote:

> Thanks Peter, but I don't follow ... I also use docker to build the 
> pre-built binaries.
>
> But how do you distribute them, so folks using your library don't think to 
> install them ?
>
> On Wednesday, October 11, 2023 at 12:32:57 PM UTC+2 Peter Galbavy wrote:
>
>> I use a docker container (golang:bullseye seems the most compatible) and 
>> build static binaries. This is feasible as long as your dependencies do not 
>> require dynamic loading of dependencies.
>>
>> Poor (my Dockerfile skills) example here: 
>> https://github.com/ITRS-Group/cordial/blob/main/Dockerfile
>>
>> Peter
>>
>> On Wednesday, 11 October 2023 at 10:58:23 UTC+1 Jan wrote:
>>
>>> hi,
>>>
>>> I'm developing a couple of ML framework/libraries for Go that depend on 
>>> C/C++ code. Once C-libraries dependencies are installed, the CGO 
>>> integration work great.
>>>
>>> Now, for end-users that just want to use these Go libraries, having to 
>>> figure out how to manually build and install those C/C++/Rust dependencies 
>>> is a hassle -- sadly each one with a different type of build system.
>>>
>>> I offer pre-built `.tgz` files (for a limited set of architectures) with 
>>> the required `.h` and `.a/.so` files along the releases, which facilitates. 
>>> But it's still a hassle to install -- and no auto-uninstall if someone is 
>>> no longer using the Go module.
>>>
>>> I was wondering if others have figured out how to handle this in a nicer 
>>> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>>>
>>> I like how Python wheels (`.whl` files) solve the issue, by including 
>>> the pre-built libraries in a sub-directory of the python library 
>>> installation. I was hoping there would be a similar way (maybe with a 
>>> separate tool) to integrate with `go.mod`. 
>>>
>>> Any pointers, thoughts or suggestions are very appreciated.
>>>
>>> Thanks!
>>> Jan
>>>
>>> ps: While searching I saw similar questions, but none that exactly 
>>> answered this aspect of distribution. Just in case, apologies if it's a 
>>> duplicate question.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/715a15ae-119f-457c-b581-525e72d2518cn%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

Thanks Peter, but I don't follow ... I also use docker to build the 
pre-built binaries.

But how do you distribute them, so folks using your library don't think to 
install them ?

On Wednesday, October 11, 2023 at 12:32:57 PM UTC+2 Peter Galbavy wrote:

> I use a docker container (golang:bullseye seems the most compatible) and 
> build static binaries. This is feasible as long as your dependencies do not 
> require dynamic loading of dependencies.
>
> Poor (my Dockerfile skills) example here: 
> https://github.com/ITRS-Group/cordial/blob/main/Dockerfile
>
> Peter
>
> On Wednesday, 11 October 2023 at 10:58:23 UTC+1 Jan wrote:
>
>> hi,
>>
>> I'm developing a couple of ML framework/libraries for Go that depend on 
>> C/C++ code. Once C-libraries dependencies are installed, the CGO 
>> integration work great.
>>
>> Now, for end-users that just want to use these Go libraries, having to 
>> figure out how to manually build and install those C/C++/Rust dependencies 
>> is a hassle -- sadly each one with a different type of build system.
>>
>> I offer pre-built `.tgz` files (for a limited set of architectures) with 
>> the required `.h` and `.a/.so` files along the releases, which facilitates. 
>> But it's still a hassle to install -- and no auto-uninstall if someone is 
>> no longer using the Go module.
>>
>> I was wondering if others have figured out how to handle this in a nicer 
>> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>>
>> I like how Python wheels (`.whl` files) solve the issue, by including the 
>> pre-built libraries in a sub-directory of the python library installation. 
>> I was hoping there would be a similar way (maybe with a separate tool) to 
>> integrate with `go.mod`. 
>>
>> Any pointers, thoughts or suggestions are very appreciated.
>>
>> Thanks!
>> Jan
>>
>> ps: While searching I saw similar questions, but none that exactly 
>> answered this aspect of distribution. Just in case, apologies if it's a 
>> duplicate question.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/53d67347-9184-4f21-b66a-2f07a178dd09n%40googlegroups.com.

[go-nuts] Re: Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Peter Galbavy]

I use a docker container (golang:bullseye seems the most compatible) and 
build static binaries. This is feasible as long as your dependencies do not 
require dynamic loading of dependencies.

Poor (my Dockerfile skills) example 
here: https://github.com/ITRS-Group/cordial/blob/main/Dockerfile

Peter

On Wednesday, 11 October 2023 at 10:58:23 UTC+1 Jan wrote:

> hi,
>
> I'm developing a couple of ML framework/libraries for Go that depend on 
> C/C++ code. Once C-libraries dependencies are installed, the CGO 
> integration work great.
>
> Now, for end-users that just want to use these Go libraries, having to 
> figure out how to manually build and install those C/C++/Rust dependencies 
> is a hassle -- sadly each one with a different type of build system.
>
> I offer pre-built `.tgz` files (for a limited set of architectures) with 
> the required `.h` and `.a/.so` files along the releases, which facilitates. 
> But it's still a hassle to install -- and no auto-uninstall if someone is 
> no longer using the Go module.
>
> I was wondering if others have figured out how to handle this in a nicer 
> way.  Is there a recommended way to distribute prebuilt CGO dependencies ?
>
> I like how Python wheels (`.whl` files) solve the issue, by including the 
> pre-built libraries in a sub-directory of the python library installation. 
> I was hoping there would be a similar way (maybe with a separate tool) to 
> integrate with `go.mod`. 
>
> Any pointers, thoughts or suggestions are very appreciated.
>
> Thanks!
> Jan
>
> ps: While searching I saw similar questions, but none that exactly 
> answered this aspect of distribution. Just in case, apologies if it's a 
> duplicate question.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f70dd0e8-6d21-4c36-a755-fdc52f9aff06n%40googlegroups.com.