[gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Michał Górny]

Hello,

As suggested by j...@g.o in bug #321165, I would like to raise a proposal
on extending toolchain-funcs.eclass with helper functions to compile
applications directly.

There are many simple applications which come without neither a sophisticated
build system or even a tiny Makefile. In some cases, such applications aren't
even packages as tarball -- a single, compressed source file is published
instead.

In case of these applications, ebuilds inherit toolchain-funcs.eclass to
retrieve compiler information (tc-getCC/tc-getCXX) and call compiler manually,
passing appropriate flags.

This way, we get certain amount of code being repeated in many ebuilds,
slightly differing in the way of calling the compiler and order of arguments
being passed to it (see examples below). Although I haven't heard of any
problems related to that, it is certainly not fool-proof and hard to maintain.

As jer suggested, I'm providing below a listing of example ebuilds calling
the compiler directly and notes on differences between the way they call it.
Similar list could be obtained through calling:
$ grep tc-getCXX.*CXXFLAGS.*-o /usr/portage/*/*/*.ebuild
$ grep tc-getCC.*CFLAGS.*-o /usr/portage/*/*/*.ebuild

C++ examples include:
* app-misc/clockywock - single C++ file, ${LDFLAGS} before ${CXXFLAGS},
* app-pda/iripdb - two C++ files, with echoing the commands, ${LDFLAGS}
ignored,
* app-text/duconv - single C++ file with separate compilation and linking,
${LDFLAGS} after ${CXXFLAGS},
* dev-cpp/tree - single C++ file, ${LDFLAGS} ignored,
* games-arcade/gunocide2ex - multiple C++ files, ${LDFLAGS} after ${CXXFLAGS},
* mail-filter/normalizemime,
* media-sound/hearnet/hearnet,
* net-misc/l7-protocols,
* x11-apps/amlc.

C examples include:
* app-admin/hwreport - single C file, ${LDFLAGS} after ${CFLAGS},
* app-admin/sysrqd - single C file, ${LDFLAGS} ignored,
* app-arch/unlzx - single C file, ${LDFLAGS} after ${CFLAGS},
* app-arch/unmakeself - single C file, '-o' after libraries,
* app-cdr/bchunk - ${LDFLAGS} ignored,
* app-cdr/{cddetect,nrg2iso,uif2iso}
* app-crypt/scsign - '-pthread' at the beginning, '-o' after sources,
* app-emacs/mpg123-el - ${LDFLAGS} ignored,
* app-forensics/{cmospwd,galleta,pasco,rifiuti} - ${LDFLAGS} ignored,
'-o' after sources,
* app-misc/hilite - ${LDFLAGS} before ${CFLAGS},
* app-misc/irtrans-irclient - ${LDFLAGS} ignored,
* app-misc/sl - '-o' after libs, command echoed,
* app-text/unpaper - ${LDFLAGS} before ${CFLAGS}, '-o' after sources,
* dev-games/paklib - ${LDFLAGS} ignored, '-o' after sources,
* dev-java/rxtx - ${LDFLAGS} before ${CFLAGS}, '-o' after sources,
* dev-lang/bff,
* dev-libs/libhash - ${LDFLAGS} ignored,
* dev-libs/libmemcache - ${LDFLAGS} ignored,
* dev-libs/nss - (pseudo-configure check), seems ugly anyway,
* dev-tcltk/tclperl - ${LDFLAGS} ignored, '-o' in the middle of flags,
* dev-tex/tex4ht - ${CPPFLAGS} used (not many ebuilds care about them),
* dev-tinyos/{channelgen,listen,serial-forwarder} - ${LDFLAGS} ignored,
* dev-util/bsdiff,
* dev-util/lockrun - ${LDFLAGS} ignored, '-o' after sources,
* games-puzzle/trimines - creating pseudo-Makefile, ${LDFLAGS} ignored,
'sdl-config --cflags' after '-o', '-o' after sources,
* games-server/tetrix - ${LDFLAGS} ignored,
* games-util/biounzip,
* mail-filter/qmail-scanner - ${LDFLAGS} ignored,
* media-gfx/pngrewrite - ${LDFLAGS} before ${CFLAGS}, '-o' after libs,
* media-sound/cd-discid - echoing commands,
* media-sound/modplugplay,
* media-sound/mserv - '-o' after sources,
* media-sound/mup,
* media-sound/wavegain - ${LDFLAGS} before ${CFLAGS}, '-o' after sources,
defines after '-o',
* media-tv/mythtv - ${LDFLAGS} after sources,
* media-video/cpvts - echoing commands,
* net-analyzer/bigeye - ${LDFLAGS} ignored,
* net-analyzer/nagios-nrpe - ${LDFLAGS} ignored,
* net-analyzer/nipper - ${LDFLAGS} ignored, '-o' after sources,
* net-analyzer/ttcp - ${LDFLAGS} ignored,
* net-dns/bind-dns-keygen - ${LDFLAGS} ignored,
* net-dns/tinystats - ${LDFLAGS} after '-o',
* net-fs/mount-cifs - ${LDFLAGS} ignored,
* net-fs/samba-client,
* net-im/bitlbee - ${LDFLAGS} ignored,
* net-irc/ircd-hybrid - ${LDFLAGS} ignored,
* net-mail/qmailanalog - ${LDFLAGS} ignored, '-o' after sources,
* net-misc/connect,
* net-misc/{datapipe,getdate} - ${LDFLAGS} ignored,
* net-misc/ifenslave,
* net-print/cups-pdf - ${LDFLAGS} ignored,
* net-print/poster - ${LDFLAGS} ignored, '-o' after libs,
* sci-biology/consed - ${LDFLAGS} ignored, '-o' after sources,
* sci-chemistry/imosflm - separate compiling and linking, einfo printing,
* sci-libs/ufconfig - separate compiling and linking,
* sys-apps/netboot-base - ${LDFLAGS} ignored,
* sys-apps/usb_modeswitch,
* sys-apps/util-linux - (configure-like check),
* sys-block/lsiutil,
* sys-devel/gcc-config,
* x11-libs/gl2ps - separate compiling and linking,
* x11-misc/blockdpy,
* x11-misc/xidle - echoing, using eval.

And that's all for gentoo-x86 -- that's over 80 packages, with over 30
ignoring ${LDFLAG

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Mike Frysinger]

On Monday, May 31, 2010 15:12:46 Michał Górny wrote:
> There are many simple applications which come without neither a
> sophisticated build system or even a tiny Makefile. In some cases, such
> applications aren't even packages as tarball -- a single, compressed
> source file is published instead.
> 
> In case of these applications, ebuilds inherit toolchain-funcs.eclass to
> retrieve compiler information (tc-getCC/tc-getCXX) and call compiler
> manually, passing appropriate flags.

use emake then and leverage make's implicit rules.  sys-apps/unscd is a pretty 
straight forward example.
-mike


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Michał Górny]

On Wed, 2 Jun 2010 03:16:12 -0400
Mike Frysinger  wrote:

> On Monday, May 31, 2010 15:12:46 Michał Górny wrote:
> > There are many simple applications which come without neither a
> > sophisticated build system or even a tiny Makefile. In some cases,
> > such applications aren't even packages as tarball -- a single,
> > compressed source file is published instead.
> > 
> > In case of these applications, ebuilds inherit
> > toolchain-funcs.eclass to retrieve compiler information
> > (tc-getCC/tc-getCXX) and call compiler manually, passing
> > appropriate flags.
> 
> use emake then and leverage make's implicit rules.

The implicit make rules are less universal and -- in the fact -- pretty
poor. They're strictly make-dependant, which reduces the amount of
control over their behavior. In my opinion, if we should ever use such
behavior, it should be rather technical implementation of the functions
I'm proposing instead of inline use.

POSIX (man 1p make) defines only simple rule for C files:
$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $<

I wasn't able to find a rule for C++ files. GNU Make seems a little
better but it's documentation is blurry. Although it supports both C
and C++, and pretty wide set of variables, it's still less than
solution proposed by me. And it's only 'de-facto standard', which isn't
guaranteed to be kept unchanged in the future.

Namely, advantages of my solution over directly calling emake is:
1) direct control over how compiler will be called == better
portability,
2) possibility of using any output filename (with emake we'd have to
rename),
3) possibility of clearly using multiple input files,
4) possibility of using any input file suffix,
5) clear separation between user-specified and ebuild-specified flags
(yes, I'm aware of '+=' GNU make extension).

And after all, starting a make jobserver for a single compiler process
doesn't seem really useful.

> sys-apps/unscd is a pretty straight forward example.

In my opinion it's rather a poor example. It's the simplest case
possible -- single file, no additional flags, no libs.

-- 
Best regards,
Michał Górny





signature.asc
Description: PGP signature

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Mike Frysinger]

On Wednesday, June 02, 2010 09:17:00 Michał Górny wrote:
> On Wed, 2 Jun 2010 03:16:12 -0400 Mike Frysinger wrote:
> > On Monday, May 31, 2010 15:12:46 Michał Górny wrote:
> > > There are many simple applications which come without neither a
> > > sophisticated build system or even a tiny Makefile. In some cases,
> > > such applications aren't even packages as tarball -- a single,
> > > compressed source file is published instead.
> > > 
> > > In case of these applications, ebuilds inherit
> > > toolchain-funcs.eclass to retrieve compiler information
> > > (tc-getCC/tc-getCXX) and call compiler manually, passing
> > > appropriate flags.
> > 
> > use emake then and leverage make's implicit rules.
> 
> The implicit make rules are less universal and -- in the fact -- pretty
> poor. They're strictly make-dependant, which reduces the amount of
> control over their behavior. In my opinion, if we should ever use such
> behavior, it should be rather technical implementation of the functions
> I'm proposing instead of inline use.

ebuilds requires GNU make which means the implicit rules are "universal".  not 
that i really know what you're talking about.  the rules are also clearly 
documented and support all the proper flags we currently support.

> POSIX (man 1p make) defines only simple rule for C files:
>   $(CC) $(CFLAGS) $(LDFLAGS) -o $@ $<

POSIX rules are irrelevant.  read the GNU make documentation.

> Namely, advantages of my solution over directly calling emake is:
> 1) direct control over how compiler will be called == better
> portability,

irrelevant.  this is like saying we should use POSIX shell code instead of 
bash because it's "better portability".  we use the tools that are required by 
the ebuild system and that means GNU make.

> 2) possibility of using any output filename (with emake we'd have to
> rename),

the -o location is irrelevant in the compile line, and most packages name the 
source file the same as the output, so not that big of a deal

> 3) possibility of clearly using multiple input files,

yep

> 4) possibility of using any input file suffix,

perhaps, but very few packages are stupid enough to name a file moocow.txt 
when it's really a C file

> 5) clear separation between user-specified and ebuild-specified flags
> (yes, I'm aware of '+=' GNU make extension).

hard to sat it's irrelevant

> And after all, starting a make jobserver for a single compiler process
> doesn't seem really useful.

and serializing multiple files is worse

> > sys-apps/unscd is a pretty straight forward example.
> 
> In my opinion it's rather a poor example. It's the simplest case
> possible -- single file, no additional flags, no libs.

and there are many like it

btw, toolchain-funcs is not really the best place to add this.  it's an eclass 
for querying, not compiling.  better to start a new one.
-mike


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Michał Górny]

On Wed, 2 Jun 2010 17:56:38 -0400
Mike Frysinger  wrote:

> On Wednesday, June 02, 2010 09:17:00 Michał Górny wrote:
> > On Wed, 2 Jun 2010 03:16:12 -0400 Mike Frysinger wrote:
> > > use emake then and leverage make's implicit rules.
> > 
> > The implicit make rules are less universal and -- in the fact --
> > pretty poor. They're strictly make-dependant, which reduces the
> > amount of control over their behavior. In my opinion, if we should
> > ever use such behavior, it should be rather technical
> > implementation of the functions I'm proposing instead of inline use.
> 
> ebuilds requires GNU make which means the implicit rules are
> "universal".  not that i really know what you're talking about.  the
> rules are also clearly documented and support all the proper flags we
> currently support.

Ok, if we can surely assume 'emake' will always call GNU Make, then I
agree with your solution and withdraw my ideas.

-- 
Best regards,
Michał Górny





signature.asc
Description: PGP signature

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Mike Frysinger]

On Saturday, June 05, 2010 09:16:29 Michał Górny wrote:
> On Wed, 2 Jun 2010 17:56:38 -0400 Mike Frysinger wrote:
> > On Wednesday, June 02, 2010 09:17:00 Michał Górny wrote:
> > > On Wed, 2 Jun 2010 03:16:12 -0400 Mike Frysinger wrote:
> > > > use emake then and leverage make's implicit rules.
> > > 
> > > The implicit make rules are less universal and -- in the fact --
> > > pretty poor. They're strictly make-dependant, which reduces the
> > > amount of control over their behavior. In my opinion, if we should
> > > ever use such behavior, it should be rather technical
> > > implementation of the functions I'm proposing instead of inline use.
> > 
> > ebuilds requires GNU make which means the implicit rules are
> > "universal".  not that i really know what you're talking about.  the
> > rules are also clearly documented and support all the proper flags we
> > currently support.
> 
> Ok, if we can surely assume 'emake' will always call GNU Make, then I
> agree with your solution and withdraw my ideas.

i'm not saying `emake` covers all solutions you proposed, just that the base 
usage isnt nearly as bleak.  there are some cases where having a wrapper 
around emake would be useful (such as compiling multiple files or linking 
things in).  so if you wanted to slap something together, we can look at it 
further.
-mike


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Michał Górny]

On Sun, 6 Jun 2010 00:17:27 -0400
Mike Frysinger  wrote:

> i'm not saying `emake` covers all solutions you proposed, just that
> the base usage isnt nearly as bleak.  there are some cases where
> having a wrapper around emake would be useful (such as compiling
> multiple files or linking things in).  so if you wanted to slap
> something together, we can look at it further.

Well, I've already covered one corner case myself, which is the zpaq
ebuild. I would approve a comment on my idea here:

http://overlays.gentoo.org/proj/sunrise/changeset/10641/sunrise/app-arch/zpaq/zpaq-1.10.ebuild

The hard part was that zpaq.cpp had to be compiled to both the execu-
table and the stub object file (with additional -DOPT flag).

I've decided that the simplest and most fail-proof idea was to create
an additional source file, defining 'OPT' and including the real code,
and compiling all of that within a single 'emake' call.

-- 
Best regards,
Michał Górny





signature.asc
Description: PGP signature

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Mike Frysinger]

On Sunday, June 06, 2010 03:45:51 Michał Górny wrote:
> On Sun, 6 Jun 2010 00:17:27 -0400 Mike Frysinger wrote:
> > i'm not saying `emake` covers all solutions you proposed, just that
> > the base usage isnt nearly as bleak.  there are some cases where
> > having a wrapper around emake would be useful (such as compiling
> > multiple files or linking things in).  so if you wanted to slap
> > something together, we can look at it further.
> 
> Well, I've already covered one corner case myself, which is the zpaq
> ebuild. I would approve a comment on my idea here:
> 
> http://overlays.gentoo.org/proj/sunrise/changeset/10641/sunrise/app-arch/zp
> aq/zpaq-1.10.ebuild
> 
> The hard part was that zpaq.cpp had to be compiled to both the execu-
> table and the stub object file (with additional -DOPT flag).
> 
> I've decided that the simplest and most fail-proof idea was to create
> an additional source file, defining 'OPT' and including the real code,
> and compiling all of that within a single 'emake' call.

C/C++ files should always have a newline at the end ... that stub file is 
currently missing one.

i dont know anything about zpaq, but it looks like you're installing the bare 
.o files ?  that's a bit odd.  usually things like this are installed as 
static archives for people to link in ...

wrt the emake stuff, that looks fine
-mike


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Michał Górny]

On Tue, 8 Jun 2010 18:44:55 -0400
Mike Frysinger  wrote:

> On Sunday, June 06, 2010 03:45:51 Michał Górny wrote:
> > Well, I've already covered one corner case myself, which is the zpaq
> > ebuild. I would approve a comment on my idea here:
> > 
> > http://overlays.gentoo.org/proj/sunrise/changeset/10641/sunrise/app-arch/zp
> > aq/zpaq-1.10.ebuild
> > 
> > The hard part was that zpaq.cpp had to be compiled to both the
> > execu- table and the stub object file (with additional -DOPT flag).
> > 
> > I've decided that the simplest and most fail-proof idea was to
> > create an additional source file, defining 'OPT' and including the
> > real code, and compiling all of that within a single 'emake' call.
> 
> C/C++ files should always have a newline at the end ... that stub
> file is currently missing one.

Noted.

> i dont know anything about zpaq, but it looks like you're installing
> the bare .o files ?  that's a bit odd.  usually things like this are
> installed as static archives for people to link in ...

A single .o file exactly. ZPAQ in runtime can 'optimize' itself through
creating compression scheme sources and calling 'zpaqmake' to compile
it.

The default 'zpaqmake.bat' included with it compiled these schemes
along with 'zpaq.cpp' for the executable stub. I've decided that it'd
be better to provide already compiled stub.

Honestly, I didn't think long about the format. I guess .a could be
better, or maybe even a shared library -- but GNU make doesn't seem to
have implicit rules for these formats.

-- 
Best regards,
Michał Górny





signature.asc
Description: PGP signature

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Mike Frysinger]

On Wednesday, June 09, 2010 01:34:27 Michał Górny wrote:
> On Tue, 8 Jun 2010 18:44:55 -0400 Mike Frysinger wrote:
> > On Sunday, June 06, 2010 03:45:51 Michał Górny wrote:
> > i dont know anything about zpaq, but it looks like you're installing
> > the bare .o files ?  that's a bit odd.  usually things like this are
> > installed as static archives for people to link in ...
> 
> A single .o file exactly. ZPAQ in runtime can 'optimize' itself through
> creating compression scheme sources and calling 'zpaqmake' to compile
> it.
> 
> The default 'zpaqmake.bat' included with it compiled these schemes
> along with 'zpaq.cpp' for the executable stub. I've decided that it'd
> be better to provide already compiled stub.

if it's all internal to zpaq itself (i.e. packages arent externally pulling in 
the files w/out help), then the .o's are fine.

> Honestly, I didn't think long about the format. I guess .a could be
> better, or maybe even a shared library -- but GNU make doesn't seem to
> have implicit rules for these formats.

there are implicit rules for generating archives, but iirc, they dont work in 
parallel.  shared libs obviously dont exist because this is highly target 
specific -- this is why libtool exists in the first place.
-mike


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass: functions to call compiler

[Jeroen Roovers]

On Wed, 9 Jun 2010 01:41:00 -0400
Mike Frysinger  wrote:

> there are implicit rules for generating archives, but iirc, they dont
> work in parallel.

The buggy[1] implicit make rule generates parallel calls to ar(1) for
each file to add, instead of doing the obvious and calling ar once with
multiple file arguments. Funny because the fix is probably as easy to
write as the wording in that section of the make manual.


 jer


[1] http://www.gnu.org/software/make/manual/make.html#Archive-Pitfalls