Re: autotools not suited to proprietary development?

[Ryan McDougall]

On Wed, 2006-10-04 at 21:33 -0700, Andre Stechert wrote:
> On Oct 4, 2006, at 8:55 PM, Ryan McDougall wrote:
> 
> > However the problem remains that Im at a bit of a loss how to ship a
> > shared .SO library easily. If I build on my machine (or a set of
> > supported build machines) then the build will link to my prefix (lets
> > say /usr/lib) and the end user has no choice where he wants to  
> > place his
> > library?
> 
> This can be hard to do portably but not because of the autotools.  I  
> think it
> boils down to two kinds of problems:
> 
> 1) If your build is libtoolized, then you may end up with -rpath  
> references
> in your .la files (Google for "debian rpath" to get a sense of the  
> drama that's
> followed this issue over the years).  If you're just distributing  
> the .so's, then
> this isn't so much of a problem, as you can fix it by having your  
> users install
> the .so wherever they want, but they have to reference by augmenting  
> their
> LD_LIBRARY_PATH appropriately. 


> In summary, if you're careful about it, you can do exactly what you  
> propose.
> 
> Cheers,
> Andre

Apologies if this is starting to wander off course, but Im really trying
to learn all this stuff so I can port more programs to linux, so I hope
you dont mind being the ones to give me a clue but ...

should I understand that (for example) when redhat/debian build lets say
libc for packaging as a binary, they download a tarball and do a
complicated form 
'./configure --prefix=/usr && make && make install' on a bare machine
without any libc, then tar up the result for an RPM or DEB?

I have looked briefly how to make RPMs, but Ive got some missing pieces
here...

Im well aware this is getting offtopic, so if possible could you spare a
clue and a link to a place where I can research the problem more myself?

Cheers,

Re: autotools not suited to proprietary development?

[Ryan McDougall]

On Wed, 2006-10-04 at 20:11 -0700, Andre Stechert wrote:
> Hi,
> 
> I think you're confusing the idea of a build system for portable  
> software
> (something the autotools suite can help with) and an installer (or  
> package
> if you're installing onto a system that has a package manager).  From a
> Windows perspective, it's the same as the difference between Visual  
> Studio
> and something like InstallShield or NSIS.

I think is is quite true, years of free software having skewed my
experiences and spoiled me to easy source-based portability. :)

However the problem remains that Im at a bit of a loss how to ship a
shared .SO library easily. If I build on my machine (or a set of
supported build machines) then the build will link to my prefix (lets
say /usr/lib) and the end user has no choice where he wants to place his
library?

This is the kind of detail I would very much hope was solved by some
software somewhere...

Cheers,
> 
> One trick that may help is automake's "DESTDIR" variable, which  
> allows you
> to create a "degenerate" package (i.e., something that doesn't allow  
> you to
> do fancy system setup -- not unlike installing a Windows app from  
> a .zip file).
> When you're doing make install, try instead:
> 
>   make install DESTDIR=/tmp/staging
> 
> Automake will build everything as if it's going to install to your -- 
> prefix (including
> embedding references to your prefix in the appropriate places) and then
> at the last minute, it will take the files it was going to install  
> and "destdir" them
> at /tmp/staging.  Then you can simply tar/gz the files at /tmp/ 
> staging and ship
> those to your customers.  Or zip them.
> 
> Cheers,
> Andre
> 
> On Oct 4, 2006, at 7:41 PM, Ryan McDougall wrote:
> 
> > Hello,
> >
> >  First of all, Im only looking for serious answers, and I dont want to
> > start any flamewars, so Ill start out by saying that quite likely the
> > only problem here is my lack of knowledge of autotools, and the other
> > the issues involved. Please correct me at any point you disagree with.
> >
> >  After some cajoling of new company from myself -- a linux and free
> > software buff, and a lonely professor who wants to use one of our SDKs
> > on linux, it was finally decided to make a linux port of our
> > SmartCollision SDK.
> >  The C++ code is clean and portable, so "porting" really just consists
> > of building the binaries, and putting them in a friendly  
> > distributable.
> >
> >  In windows the deliverable is a zip file with .DLLs, .LIBs, and  
> > header
> > files that should work on any modern widows. The linux deliverable
> > should be the same, however, given the large amount of variability in
> > the linux world, I would like to have autoconf do as much work as
> > possible, which I think means getting the code to compile on my build
> > machine, and link on the users environment.
> >
> >  With my inexperience with doing proprietary development, I chose  
> > to use
> > autotools since I was the only tool I was familiar with from open  
> > source
> > projects. The problem is the major assumption in autotools is that the
> > source code is being shipped to the end user, and thus the  
> > distributable
> > package is the same as the source tarball. I was unable, in the time
> > allowed, to come up with a way to easily (ie clean, without many  
> > hacks)
> > give me the sort of solution I was looking for, which would look
> > something like:
> > './configure && make && make bindist' on the build machine
> > and
> > './configure && make && make install' on the end users machine
> >
> > As it is we just
> > './configure --prefix=/package/dir/ && make && make install'
> > then manually package the static libA.a libraries and headers  
> > there. We
> > couldnt do shared libraries since libtool wanted to link the .so to  
> > the
> > build machine's prefix (via -rpath switch).
> >
> > Has this problem been solved before with autotools? Or is this just  
> > not
> > a problem autotools wants to solve? If so, does anyone have  
> > alternative
> > recommendations?
> >
> > I ask because I have hope we can port our remaining products to linux,
> > and that I can learn how to do this sort of thing properly next time.
> > But if it gets too expensive for my boss, he wont be likely to approve
> > it.
> >
> > Cheers,
> >
> >
> >
> >
> 

autotools not suited to proprietary development?

[Ryan McDougall]

Hello,

 First of all, Im only looking for serious answers, and I dont want to
start any flamewars, so Ill start out by saying that quite likely the
only problem here is my lack of knowledge of autotools, and the other
the issues involved. Please correct me at any point you disagree with.

 After some cajoling of new company from myself -- a linux and free
software buff, and a lonely professor who wants to use one of our SDKs
on linux, it was finally decided to make a linux port of our
SmartCollision SDK.
 The C++ code is clean and portable, so "porting" really just consists
of building the binaries, and putting them in a friendly distributable.

 In windows the deliverable is a zip file with .DLLs, .LIBs, and header
files that should work on any modern widows. The linux deliverable
should be the same, however, given the large amount of variability in
the linux world, I would like to have autoconf do as much work as
possible, which I think means getting the code to compile on my build
machine, and link on the users environment.

 With my inexperience with doing proprietary development, I chose to use
autotools since I was the only tool I was familiar with from open source
projects. The problem is the major assumption in autotools is that the
source code is being shipped to the end user, and thus the distributable
package is the same as the source tarball. I was unable, in the time
allowed, to come up with a way to easily (ie clean, without many hacks)
give me the sort of solution I was looking for, which would look
something like:
'./configure && make && make bindist' on the build machine
and
'./configure && make && make install' on the end users machine

As it is we just 
'./configure --prefix=/package/dir/ && make && make install' 
then manually package the static libA.a libraries and headers there. We
couldnt do shared libraries since libtool wanted to link the .so to the
build machine's prefix (via -rpath switch).

Has this problem been solved before with autotools? Or is this just not
a problem autotools wants to solve? If so, does anyone have alternative
recommendations?

I ask because I have hope we can port our remaining products to linux,
and that I can learn how to do this sort of thing properly next time.
But if it gets too expensive for my boss, he wont be likely to approve
it.

Cheers,

support for binary dist?

[Ryan McDougall]

Hello,
 I am porting a proprietary (yes I know, but it pays the bills) windows
application to linux using autotools. So far I have gotten automake to
do what I want up until the install/build stage.

 I wish to provide our end-users with a tarball that includes a
dynamically linkable library, and the needed header files for our SDK.

 As far as I can tell, I have two options:
1. make install into a package directory, then tar that up by hand
2. make dist while somehow excluding the original .CPP files.

 Which immediately gives me two problems:
1. make install links my dynamic SO to my package directory instead of
the user's prefix directory
2. make dist complains about the missing .CPP files, even if I include
all the .O and .LO files

 Does automake have any make bindist support? Can anyone recommend a
method to me?

 I dont have much experience with making proprietary (non-GNU)
applications so I apologize if I missed something.

Cheers,

Re: The automake/libtool way to make a dynamic lib from multiple sources

[Ryan McDougall]

On Thu, 2006-09-28 at 13:06 +0200, Ralf Wildenhues wrote:
> Hello Ryan,
> 
> * Ryan McDougall wrote on Thu, Sep 28, 2006 at 11:30:32AM CEST:
> > On Thu, 2006-09-28 at 18:23 +0900, Ryan McDougall wrote:
> > > 
> > > How can I either
> > > 1. Express to automake the correct dependancy
> 
> By writing
>   libA_la_LIBADD = libfoo.la
> 
> instead of
>   libA_la_LIBADD = -L. -lfoo
>  
> > I just discovered that I can trick automake with this:
> > 
> > zardir=/dev/null
> > zar_LTLIBRARIES=libfoo.la
> > lib_LTLIBRARIES=libA.la
> > 
> > and this will allow libfoo to be built before libA, however this is sort
> > of trickery is bound to bite me in the ass somewhere, right?
> 
> That is wrong, as it causes libfoo to not be a convenience archive, but
> a real, installable library.  It's doomed to failure, as the installed
> libA may depend on it being present.
> 
> Cheers,
> Ralf
> 

Cheers, everything works as I need it to to get the basic build up!

Thanks for all your help Ralf!

For the sake of those that google after me, here is my solution:
In the directory containing the target library directories, and the
dependent library directories I have the following Makefile.am

AUTOMAKE_OPTIONS = subdir-objects

noinst_LTLIBRARIES =
lib_LTLIBRARIES =

include libfoo/Makefile.am
include libbar/Makefile.am
include libA/Makefile.am

within the dependent library directories I have the following
Makefile.am

noinst_LTLIBRARIES += libfoo.la
libfoo_la_CPPFLAGS = -Ilibfoo/include
libfoo_la_SOURCES = libfoo/src/libfoo.cpp

within the target library directories I have the following Makefile.am

lib_LTLIBRARIES += libA.la
libA_la_LIBADD = libfoo.la
libA_la_CPPFLAGS = -DIMPORTANT_COMPILE_TIME_SWITCHES \
 -IlibA/include \
 -Ilibfoo/include
libA_la_SOURCES = libA/src/libA.cpp

Re: The automake/libtool way to make a dynamic lib from multiple sources

[Ryan McDougall]

On Thu, 2006-09-28 at 18:23 +0900, Ryan McDougall wrote:
> On Fri, 2006-09-22 at 04:34 +0200, Ralf Wildenhues wrote:
> > * Ryan McDougall wrote on Fri, Sep 22, 2006 at 02:54:21AM CEST:
> > > On Thu, 2006-09-21 at 13:32 +0200, Ralf Wildenhues wrote:
> > > > * Ryan McDougall wrote on Thu, Sep 21, 2006 at 11:40:37AM CEST:

> 
> How can I either
> 1. Express to automake the correct dependancy
> 2. Trick automake into building lib_LTLIBRARIES last (after
> noinst_LTLIBRARIES)
> 

I just discovered that I can trick automake with this:

zardir=/dev/null
zar_LTLIBRARIES=libfoo.la
lib_LTLIBRARIES=libA.la

and this will allow libfoo to be built before libA, however this is sort
of trickery is bound to bite me in the ass somewhere, right?

Cheers,

Re: The automake/libtool way to make a dynamic lib from multiple sources

[Ryan McDougall]

On Fri, 2006-09-22 at 04:34 +0200, Ralf Wildenhues wrote:
> * Ryan McDougall wrote on Fri, Sep 22, 2006 at 02:54:21AM CEST:
> > On Thu, 2006-09-21 at 13:32 +0200, Ralf Wildenhues wrote:
> > > * Ryan McDougall wrote on Thu, Sep 21, 2006 at 11:40:37AM CEST:

> > > 2) If you prefer nonrecursive makefiles (you may include makefile
> > > snippets from one into another), then you can just
> > > 
> > >   AUTOMAKE_OPTIONS = subdir-objects
> > >   module_LTLIBRARIES = libA.la libB.la libC.la
> > >   libA_la_SOURCES = foo/foo1.c bar/bar2.c A/a1.c ...
> > >   libB_la_SOURCES = baz/baz1/c bar/bar2.c B/b1.c ..
> > 
> > I thought of this, but in this case libA_la_SOURCES could be on the
> > order of 100-50 files long, which Id like to avoid if possible.
> 
> You don't have to set this in one, and you can use variables (and
> Automake conditionals) to factor.  For example:
> 
> -- Makefile.am --
> AUTOMAKE_OPTIONS = subdir-objects
> module_LTLIBRARIES = libA.la libB.la libC.la
> libA_la_SOURCES =
> libB_la_SOURCES =
> common_sources =
> include foo/snippet.am
> include bar/snippet.am
> if CONDITION_BAZ
> libB_la_SOURCES += baz/baz1.c
> endif
> libA_la_SOURCES += $(common_sources)
> libB_la_SOURCES += $(common_sources)
> 
> -- foo/snippet.am --
> ## Note the file name including the subdir here: we are being included
> ## from a level up (Automake includes just do literal replacement):
> libA_la_SOURCES += foo/foo1.c
> 
> -- bar/snippet.am
> common_sources += \
>   bar/bar2.c \
>   ...
> 

Thanks for all your help Ralf, I _really_ appreciate you taking your
time to help me out.

I am currently attempting to use a version of this (non-recursive)
method. I am avoiding exactly what you wrote since I need to pass
specific -D options to different packages without using libA_CPPFLAGS =
-DFORLIBFOO -DFORLIBBAR -DETC.

What I have now is I have replaced the file names in the recursive
Makefile.am's to be relative to the root directory like so:
libfoo_SOURCES=src/foo.cpp -> libfoo_SOURCES=libfoo/src/foo.cpp
and included libfoo/Makefile.am in the root Makefile.am.

This works well, and would nearly be finished, except in my root
Makefile.am I get this:
noinst_LTLIBRARIES = libfoo.la ...
lib_LTLIBRARIES = libA.la

which Automake dutifully turns into a rule which always builds libA.la
before libfoo.la, despite the fact that libA depends on libfoo (as
expressed by the variable libA_la_LIBADD = -lfoo).

How can I either
1. Express to automake the correct dependancy
2. Trick automake into building lib_LTLIBRARIES last (after
noinst_LTLIBRARIES)

ps. This would have been a lot easier if Automake would have
automatically provided targets for each one of the SUBDIRS. That way I
could have simply wrote one rule, libA: foo bar baz (where foo bar baz
are SUBDIRs that need to be build before the SUBDIR libA).

Should I open a bug/feature request?

Thanks a tonne for your unheralded work!!! Id hate to have to give up on
autotools.

Cheers,

Re: The automake/libtool way to make a dynamic lib from multiple sources

[Ryan McDougall]

On Fri, 2006-09-22 at 04:34 +0200, Ralf Wildenhues wrote:

> > > 
> > > You should keep in mind that the convenience archives will end up in
> > > total (i.e., all their symbols, not just needed ones) in the resulting
> > > module.
> > 
> > This is a good point, thank you. Do you have any more information
> > somewhere that elaborates on this point?
> 
> The Libtool manual has more details about this (but it's arguably a bit
> cryptic about convenience archives).  I'm not sure what you need to
> know.

Since Im leaning to this method, Id like to investigate just what sort
of inefficiencies Im going to incur this way. Basically I don't know a
whole lot about the details of dynamic linking, so Id like to learn
whats happening, if possible/reasonable.

> 
> > What is the "module" target?
> 
> Oh, I forgot.  If you install the modules in $(libdir), just use
> lib_LTLIBRARIES; for the above to work you need something like
>   moduledir = $(libdir)/foo
> 
> or similar (e.g., pkglibdir; or a setting in configure.ac).
> 
> You should not forget the libtool link flag -module (libA_la_LDFLAGS).

This sounds interesting, is there a place I can read about this modules
bit?

> 
> Cheers,
> Ralf
> 

Cheers,

Re: The automake/libtool way to make a dynamic lib from multiple sources

[Ryan McDougall]

On Thu, 2006-09-21 at 13:32 +0200, Ralf Wildenhues wrote:

> 
> The second approach will enable you to
>   make libA.la
> 
> in the toplevel directory.  Otherwise, there is nothing that prevents
> you from adding your own targets a la
>   A:
> cd src/A && $(MAKE) $(AM_MAKEFLAGS) libA.la
>   .PHONY: A
> 
> just as in ordinary makefiles.
> 
> Cheers,
> Ralf
> 

The problem with this is that with this method that I have to manually
specify the dependencies, right?

A: foo
cd src/A && $(MAKE) $(AM_MAKEFLAGS) libA.la
.PHONY: A

foo:
cd src/foo && $(MAKE) $(AM_MAKEFLAGS) libfoo.la
.PHONY: foo

etc ...

Cheers,

Re: The automake/libtool way to make a dynamic lib from multiple sources

[Ryan McDougall]

On Thu, 2006-09-21 at 13:32 +0200, Ralf Wildenhues wrote:
> Hello Ryan,
> 
> * Ryan McDougall wrote on Thu, Sep 21, 2006 at 11:40:37AM CEST:
> > 
> > Im porting a large code-base from windows to linux, and attempting to
> > use autotools to do so. The code in SVN is essentially one big tree of
> > somewhat related modules -- lets call them foo, bar, etc (Projects under
> > MS VS). I need to produce 3 dynamic libraries that can be built from
> > those modules -- lets call them A.so, B.so, and C.so (Solutions under
> > VS). They depend on a non-overlapping, non-disjoint set of the source
> > code modules in the directory.
> 
> non-overlapping, non-disjoint sounds a bit like a typo; is it?
> Or do you mean they just partially overlap?

Yes, that was me being tired and not having my brain entirely in
English. :)
> 
> > Directories for A, B, and C also
> > contain .cpp files and appear in the same parent directory as foo, bar,
> > etc.
> > 
> > Currently I have all directories building respective "foo.la"s --
> > including A.la, B.la, and C.la, and have them installing correctly.
> > 
> > Problems: 
> > - I dont need the foo,bar,etc constituent libraries, I just need A,B,C
> > libraries, which means I have to manually delete them from the install
> > location afterwords
> 
> 2 possibilities:
> 
> 1) If you prefer recursive makefiles then you can have each directory
> create one (or a number of) libtool convenience archives
> (noinst_LTLIBRARIES), which you can merge into the real modules
> (A_la_LIBADD = ../foo/libfoo.la)
> 
> You should keep in mind that the convenience archives will end up in
> total (i.e., all their symbols, not just needed ones) in the resulting
> module.

This is a good point, thank you. Do you have any more information
somewhere that elaborates on this point?
> 
> 2) If you prefer nonrecursive makefiles (you may include makefile
> snippets from one into another), then you can just
> 
>   AUTOMAKE_OPTIONS = subdir-objects
>   module_LTLIBRARIES = libA.la libB.la libC.la
>   libA_la_SOURCES = foo/foo1.c bar/bar2.c A/a1.c ...
>   libB_la_SOURCES = baz/baz1/c bar/bar2.c B/b1.c ..

I thought of this, but in this case libA_la_SOURCES could be on the
order of 100-50 files long, which Id like to avoid if possible.

AUTOMAKE_OPTIONS=subdir-objects tells automake to build temporary
objects ("foo.lo") in the sub directories to improve build times?

What is the "module" target?

> > - I would like to be able to type "Make A" from the parent directory,
> > and have the build system figure out which among foo,bar,etc libraries
> > are needed by A and build them in the manner you would expect
> 
> The second approach will enable you to
>   make libA.la
> 
> in the toplevel directory.  Otherwise, there is nothing that prevents
> you from adding your own targets a la
>   A:
> cd src/A && $(MAKE) $(AM_MAKEFLAGS) libA.la
>   .PHONY: A
> 
> just as in ordinary makefiles.

I already have this done, but I was hoping the was a more magic way to
do this.
> 
> Cheers,
> Ralf
> 

Thanks for all your help Ralf!

Cheers,

The automake/libtool way to make a dynamic lib from multiple sources

[Ryan McDougall]

Hello,
 I apologize if this has been answered somewhere. Ive read the book and
searched the archives.

 My situation is this: 

Im porting a large code-base from windows to linux, and attempting to
use autotools to do so. The code in SVN is essentially one big tree of
somewhat related modules -- lets call them foo, bar, etc (Projects under
MS VS). I need to produce 3 dynamic libraries that can be built from
those modules -- lets call them A.so, B.so, and C.so (Solutions under
VS). They depend on a non-overlapping, non-disjoint set of the source
code modules in the directory. Directories for A, B, and C also
contain .cpp files and appear in the same parent directory as foo, bar,
etc.

Currently I have all directories building respective "foo.la"s --
including A.la, B.la, and C.la, and have them installing correctly.

Problems: 
- I dont need the foo,bar,etc constituent libraries, I just need A,B,C
libraries, which means I have to manually delete them from the install
location afterwords
- I would like to be able to type "Make A" from the parent directory,
and have the build system figure out which among foo,bar,etc libraries
are needed by A and build them in the manner you would expect
- I would like to accomplish this with a minimum of messing with the
file hierarchy. Since Visual Studio can manage this feat, Im hoping its
not above autotools!

What do you recommend great wizards of autotoolry?

Cheers,