A better way to get runtime coverage with libprofile_rt (from compiler_rt) / gcov?

[Garrett Cooper]

Hi all,
In order to adapt an existing process to clang, I went and tried to 
figure out how to get coverage generation to work on FreeBSD with shared 
binaries (not static binaries), which is the default for clang/gcc: 
https://github.com/yaneurabeya/scratch/blob/master/coverage-with-shared-libs-freebsd
 . In doing this I had to commit a few cardinal sins:
1. I had to specify -lgcov, even though the gcc docs said --coverage 
implied this.
2. I had to link in static libraries directly using LDADD: 
https://github.com/yaneurabeya/scratch/blob/master/coverage-with-shared-libs-freebsd/Makefile.inc#L9
 . If I didn’t, things would blow when it tried to link lib/libfoo.so into 
bin/foo , so there might be a difference in behavior between clang and gcc in 
that regard (I tried using LDFLAGS + LDADD with just -lclang_rt.profile-x86_64, 
but that didn’t seem to work either):

/usr/bin/ld: foo: hidden symbol `llvm_gcda_emit_arcs' in 
/usr/bin/../lib/clang/3.6.0/lib/freebsd/libclang_rt.profile-x86_64.a(GCDAProfiling.o)
 is referenced by DSO
/usr/bin/ld: final link failed: Nonrepresentable section on output

Does anyone know of a better way to solve this problem, i.e. avoid 
having to hardcode the full path to libclang_rt.profile-x86_64.a (or perhaps 
make clang do the right thing when trying to link in the library)?
Thanks!
-NGie

$ make -s obj depend all
===> lib (obj)
===> bin (obj)
===> lib (depend)
===> bin (depend)
===> lib (all)
===> bin (all)
$ make get-coverage
LD_LIBRARY_PATH=/usr/obj/usr/src.svn/tests/foo/lib 
/usr/obj/usr/src.svn/tests/foo/bin/foo
This is foo
lcov --capture --directory /usr/obj/usr/src.svn/tests/foo --output-file 
/usr/obj/usr/src.svn/tests/foo/coverage.info
Capturing coverage data from /usr/obj/usr/src.svn/tests/foo
Found gcov version: 4.2.1
Scanning /usr/obj/usr/src.svn/tests/foo for .gcda files ...
Found 2 data files in /usr/obj/usr/src.svn/tests/foo
Processing lib/foo.gcda
/usr/obj/usr/src.svn/tests/foo/lib/foo.gcno:version '402*', prefer '402p'
/usr/obj/usr/src.svn/tests/foo/lib/foo.gcda:version '402*', prefer version 
'402p'
/usr/obj/usr/src.svn/tests/foo/lib/foo.gcno:'foo' has arcs from exit block
Processing bin/foo.gcda
/usr/obj/usr/src.svn/tests/foo/bin/foo.gcno:version '402*', prefer '402p'
/usr/obj/usr/src.svn/tests/foo/bin/foo.gcda:version '402*', prefer version 
'402p'
Finished .info-file creation
genhtml /usr/obj/usr/src.svn/tests/foo/coverage.info --output-directory output
Reading data file /usr/obj/usr/src.svn/tests/foo/coverage.info
Found 2 entries.
Found common filename prefix "/usr/src.svn/tests/foo"
Writing .css and .png files.
Generating output.
Processing file bin/foo.c
Processing file lib/foo.c
Writing directory view page.
Overall coverage rate:
  lines..: 100.0% (4 of 4 lines)
  functions..: 100.0% (2 of 2 functions)



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Failed to build rescue with gcc 4.9

[Garrett Cooper]

On Apr 3, 2015, at 14:10, Warner Losh  wrote:

>> 
>> On Apr 3, 2015, at 2:39 PM, Ed Maste  wrote:
>> 
>> On 3 April 2015 at 13:02, Warner Losh  wrote:
>>> That shows that something in the list is needed. Likely only crunchhide.
>>> 
>>> It doesn’t tell us why we need it, or when we started needing it, or what
>>> other conditions we might need it. This information is critical to document
>>> so we know when we can stop doing it in the future. I’m extremely reluctant
>>> to commit this until we know these details.
>> 
>> Yes, it's crunchide. It was broken prior to r277259:
>> 
>> |crunchide: Correct 64-bit section header offset
>> |
>> |For 64-bit binaries the Elf_Ehdr e_shoff is at offset 40, not 44.
>> |Instead of using an incorrect hardcoded offset, let the compiler
>> |figure it out for us with offsetof().
>> |
>> |Differential Revision: https://reviews.freebsd.org/D1543
>> 
>> It's not completely clear to me why we did not encounter this before;
>> a comment before the erroneous write states:
>> 
>> /*
>> * update the offset of section header table in elf
>> * header if needed.
>> */
>> 
>> so I presume something about the object file created by gcc 4.9 causes
>> this code to be executed, while builds using the in-tree compiler did
>> not.
> 
> Ah Yes! I remember now! We should find the FreeBSD version at that date
> and either build it when we’re cross compiling, or rebuild it when we’re
> bootstrapping.
> 
> Thanks for finding this Ed.

These numbers need to be bumped by 1, but here they are (this also wasn’t 
backported to stable/8 or stable/9 AFAICT).
Cheers!

head:

$ svn cat -r r277259 ^/head/sys/sys/param.h  | grep 
'#define.*__FreeBSD_version' | awk '{ print $3 }’
1100054

stable/10:

$ svn cat -r 277557 ^/stable/10/sys/sys/param.h | grep 
'#define.*__FreeBSD_version' | awk '{ print $3 }'
1001506



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Fails to build sys/i386/boot2 with gcc 4.9

[Garrett Cooper]

> On Mar 29, 2015, at 15:56, Warner Losh  wrote:
> 
>> On Mar 29, 2015, at 2:29 PM, Craig Rodrigues  wrote:
>> 
>> On Sun, Mar 29, 2015 at 11:04 AM, Warner Losh  wrote:
>> 
>> If we built a UFS1-only boot2, that would fit in the 7.5k we have left
>> to play with. We could then build a UFS2-only boot2 that would easily
>> fit in the like 32k limit that UFS2 has.
>> 
>> The only reason we went to supporting both was to have something
>> universal. Since it requires a reformat to go from UFS1 -> UFS2 we
>> wanted the transition to be as smooth as possible so you didn’t have
>> to add boot blocks into the mix.
>> 
>> Now the only people that use UFS1 are people with really old systems
>> that are never going to upgrade, or people building new systems with
>> UFS1 because they are space constrained (for whatever reasons that
>> we’re not going to debate here: they are still real).
>> 
>> In the past 5 years, I have worked on some embedded systems where UFS1 was 
>> chosen because of very low memory and disk space requirements.
>> So those systems are real and out there.
>> 
>> Just out of curiousity, what is it about newer compilers that cause
>> the size of boot2 to increase so much?
>> 
>> Could we do some silly things like removing/reducing the use of printf()
>> to save some more bytes, in order to buy us more time, before having
>> to rewrite everything? :)
> 
> Removing printf isn’t going to save us. It usually compiles to 80-120 bytes.
> 
> I think the only sane way forward is boot2.ufs1 an boot2.ufs2 plus maybe
> some safety belts in the boot block splatter programs to prevent
> brickification.

Since the proposal to split up the code by filesystems is on the table, would 
it make sense to do something similar for zfs?
Thanks!
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: Jenkins build is still unstable: FreeBSD_HEAD-tests2 #867

[Garrett Cooper]

On Mar 22, 2015, at 15:09, Dimitry Andric  wrote:

> On 22 Mar 2015, at 23:04, Garrett Cooper  wrote:
>> 
>> On Mar 22, 2015, at 15:01, Craig Rodrigues  wrote:
>> 
>> ...
>> 
>>> OK, converting expr.y to use unsigned integers would require a bit of work.
>>> 
>>> Can you commit your patch to the Makefile?  It fixes the problem for now.
>> 
>> +1
>> 
>> I’d still like to know why clang 3.5 doesn’t have this behavior though — 
>> there might be other potential issues lurking around that need to be solved 
>> (either here, in ports, or both).
> 
> Because this version optimizes better around undefined behavior.  There
> are most likely many issues lurking around, and most certainly in ports.
> 
> I would recommend using UBSan to tackle this kind of thing.

I hope this got a ports tinderbox run first… Adding UBSan to tinderbox runs for 
toolchain upgrades [in the future] might be a good idea.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Jenkins build is still unstable: FreeBSD_HEAD-tests2 #867

[Garrett Cooper]

On Mar 22, 2015, at 15:01, Craig Rodrigues  wrote:

...

> OK, converting expr.y to use unsigned integers would require a bit of work.
> 
> Can you commit your patch to the Makefile?  It fixes the problem for now.

+1

I’d still like to know why clang 3.5 doesn’t have this behavior though — there 
might be other potential issues lurking around that need to be solved (either 
here, in ports, or both).

Thanks!


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Jenkins build is still unstable: FreeBSD_HEAD-tests2 #867

[Garrett Cooper]

On Mar 22, 2015, at 14:36, Dimitry Andric  wrote:

> On 22 Mar 2015, at 22:32, Craig Rodrigues  wrote:
>> 
>> On Sun, Mar 22, 2015 at 2:29 PM, Dimitry Andric  wrote:
>> 
>> Ah right, that was on i386, on amd64 it does result in -2^63.  It is indeed 
>> caused by reliance on signed integer wrapping.
>> 
>> This diff should fix it, without rewriting the utility:
>> 
>> Index: bin/expr/Makefile
>> ===
>> --- bin/expr/Makefile   (revision 280156)
>> +++ bin/expr/Makefile   (working copy)
>> @@ -6,6 +6,9 @@ PROG=   expr
>> SRCS=  expr.y
>> YFLAGS=
>> 
>> +# expr relies on signed integer wrapping
>> +CFLAGS+= -fwrapv
>> +
>> NO_WMISSING_VARIABLE_DECLARATIONS=
>> 
>> .if ${MK_TESTS} != "no"
>> 
>> 
>> Well, another alternative is to patch expr.y:
>> 
>> Index: expr.y
>> ===
>> --- expr.y  (revision 280353)
>> +++ expr.y  (working copy)
>> @@ -393,7 +393,7 @@
>> }
>> 
>> void
>> -assert_plus(intmax_t a, intmax_t b, intmax_t r)
>> +assert_plus(intmax_t a, intmax_t b, volatile intmax_t r)
>> {
>>/*
>> * sum of two positive numbers must be positive,
>> @@ -420,7 +420,7 @@
>> }
>> 
>> void
>> -assert_minus(intmax_t a, intmax_t b, intmax_t r)
>> +assert_minus(intmax_t a, intmax_t b, volatile intmax_t r)
>> {
>>/* special case subtraction of INTMAX_MIN */
>>if (b == INTMAX_MIN && a < 0)
>> 
>> 
>> There were already some patches previously done to this
>> file to add "volatile", so maybe this would be OK to do.
>> 
>> What do you think?
> 
> Volatile is not the solution, it is completely orthogonal.  The correct
> way would be to use unsigned integers, for which wrapping is defined,
> then convert those back and forth when presenting the results to the
> user.

Before doing that — what changed in the past week that changed the behavior of 
expr?
Thanks!


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Jenkins build is still unstable: FreeBSD_HEAD-tests2 #867

[Garrett Cooper]

On Mar 22, 2015, at 14:02, Craig Rodrigues  wrote:

> On Sun, Mar 22, 2015 at 11:26 AM,  wrote:
> 
>> See 
>> 
> 
> Can someone with toolchain expertise look at this?
> After the clang 3.6.1 import, /bin/expr behaves differently.
> 
> With clang 3.5.0:
> 
> # expr 4611686018427387904 + 4611686018427387904
> expr: overflow
> 
> With clang 3.6.1:
> 
> # expr 4611686018427387904 + 4611686018427387904
> -9223372036854775808

I suspect -fwrapv is at fault, but I need to do some digging first..
Thanks!


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Failed to build with external toolchain

[Garrett Cooper]

On Mar 7, 2015, at 12:12, Craig Rodrigues  wrote:

> Hi,
> 
> I ran the build again and this time I am getting errors about undefined
> symbol utimensat():
> 
> https://jenkins.freebsd.org/job/FreeBSD_HEAD_external_toolchain_gcc/14/console
> 
> Any ideas?

You might need to specify more options to gcc to get it to compile in base. 
AFAIK we still have all of the “bootstrap” (-sysroot, etc) customizations 
checked into base with clang/gcc, so using the tools from ports might not work 
in all cases :/.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Build failed in Jenkins: Build-UFS-image #599

[Garrett Cooper]

On Dec 5, 2014, at 16:52, jenkins-ad...@freebsd.org wrote:

> See 

I’m not entirely sure why the "could not determine COMPILER_TYPE" error 
popped up, but I have a couple of questions/concerns related to the makefile 
snippet.
1. Does it make sense to check CC when running make install?
2. Why isn’t this value determined once in Makefile.inc1 (per build 
phase), then passed down from there (I’ve already considered the scenario where 
someone explicitly sets CC in a non-toplevel Makefile, which is a problem, but 
an outlier rather than the norm)? AFAICT, it gets recomputed for every 
recursive make, which contributes to useless forking for something that 
honestly doesn’t change all that often/at all.
At EMC/Isilon at least, we set CC/CXX=false when running make 
distribute*/installkernel/installworld to catch logic errors with rebuilding 
code. Should this be in FreeBSD?
Thanks!

PS I filed https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195732 to track 
this issue and resolve the problem.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Unable to build world w/o clang on 11

[Garrett Cooper]

> On Nov 11, 2014, at 07:51, Chris H  wrote:
> 
> On Mon, 10 Nov 2014 19:52:48 -0800 "Chris H"  wrote
> 
>> On Mon, 10 Nov 2014 19:22:31 -0800 Steve Kargl
>>  wrote
>> 
 On Mon, Nov 10, 2014 at 06:37:35PM -0800, Chris H wrote:
 ===> lib/libdpv (depend)
 rm -f .depend
 CC='cc  ' mkdep -f .depend -a-I/usr/src/lib/libdpv -std=gnu99  
 /usr/src/lib
 /libdpv/dialog_util.c /usr/src/lib/libdpv/dialogrc.c
 /usr/src/lib/libdpv/dprompt
 c /usr/src/lib/libdpv/dpv.c /usr/src/lib/libdpv/status.c
 /usr/src/lib/libdpv/ut
 il.c
 In file included from /usr/src/lib/libdpv/dialog_util.c:43:
 In file included from /usr/src/lib/libdpv/dialog_util.h:34:
 /usr/src/lib/libdpv/dialogrc.h:34:10: fatal error: 'figpar.h' file not
 found #include 
 ^
 1 error generated.
 /usr/src/lib/libdpv/dialogrc.c:34:10: fatal error: 'figpar.h' file not
 found
>>> 
>>> Until such time that you can get the tree to build without -j, don't
>>> use it.
>>> 
>>> Now, the obvious question:  do you have figpar.h in /usr/src?
>> Ugh... it's been a hectic day. Apparently it wasn't included in
>> Revision: 274349
>> Last Changed Date: 2014-11-10 10:20:46 -0800 (Mon, 10 Nov 2014)
>> 
>> :P
>> 
>> I'll blow away src, and ports, and checkout what's ever available
>> now. Hoping it's *now* included. ;)
>> 
>> Thank you *very* much rubbing my face in the obvious, Steve. :)
> Checked out the current version of src from head, that was available (r274365)
> and gave it another go. While it *did* have the missing file. The results
> were less than hoped for:

This issue was fixed just a few hours ago by des@

Cheers!
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: Building clang in buildworld as part of the bootstrap process -- is it really necessary?

[Garrett Cooper]

> tl;dr: NO.

Makes sense. I'll do some poking around and see if things could potentially be 
optimized with the clang build. On beefy machines it's not a big deal, but as 
we know on machines without a ton of memory or SSDs, it can become painful, as 
expected.

Thanks :)!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Building clang in buildworld as part of the bootstrap process -- is it really necessary?

[Garrett Cooper]

Hi all,
One of the questions that came up from a co-worker is "why do I
need to build clang in buildworld if I already installed it from
ports"? I could see some valid reasons for doing this (one needs a
cross-compiler, one might need specific options that might not be set
in the ports version), but for native builds I would tend to agree
with this logic. With gcc it was wasteful building the compiler each
buildworld, but with clang it seems annoying continuing on this path
because the compile takes a long time to complete.
Alternatively, would anyone be opposed to adding some logic to
automatically bypass the bootstrap compiler, i.e. add some logic to
Makefile.inc1 that would skip compiling clang/gcc if and only if the
target triplet and version met some required values?
Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Bootstrap-related build failure with gcc due to change in sctp system headers; not with clang (was Re: svn commit: r269527 - in head: lib/libc/net sys/netinet)

[Garrett Cooper]

On Aug 4, 2014, at 1:07 PM, Michael Tuexen  wrote:

> Author: tuexen
> Date: Mon Aug  4 20:07:35 2014
> New Revision: 269527
> URL: http://svnweb.freebsd.org/changeset/base/269527
> 
> Log:
>  Add support for the SCTP_RECONFIG_SUPPORTED and the corresponding
>  sysctl controlling the negotiation of the RE-CONFIG extension.
> 
>  MFC after: 3 days

Hi FreeBSD toolchain,
I ran into a build bootstrap problem where it only seems to fail with 
gcc after this commit because it’s picking up installed system headers instead 
of the system headers in the obj tree. I’ve filed 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192394 to track this issue
 .
Thank you,
-Garrett


signature.asc
Description: Message signed with OpenPGP using GPGMail

Help with review/commit of conf/189156?

[Garrett Cooper]

Hello all,
I was wondering if someone could please look at committing this
enhancement for src. It would fix compiling ftp/curl (which gets
pulled in via devel/git and a number of other tools) when ${MK_GSSAPI}
=> no. As it stands, the headers always get installed regardless of
whether or not the libraries get installed in the base system, and the
knob in ports looks for /usr/include/gssapi* for keying whether or not
to automatically enable gssapi/kerberos support.
Thank you!!!
-Garrett

-- Forwarded message --
From:  
Date: Wed, Apr 30, 2014 at 5:00 PM
Subject: Re: conf/189156: include/Makefile does not honor MK_GSSAPI == no
To: Garrett Cooper 


Thank you very much for your problem report.
It has the internal identification `conf/189156'.
The individual assigned to look at your
report is: freebsd-bugs.

You can access the state of your problem report at any time
via this link:

http://www.freebsd.org/cgi/query-pr.cgi?pr=189156

>Category:   conf
>Responsible:freebsd-bugs
>Synopsis:   include/Makefile does not honor MK_GSSAPI == no
>Arrival-Date:   Thu May 01 00:00:00 UTC 2014
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: svn commit: r250719 - in head: . usr.bin/make

[Garrett Cooper]

On May 16, 2013, at 2:26 PM, Simon J. Gerraty wrote:

> Author: sjg
> Date: Thu May 16 21:26:46 2013
> New Revision: 250719
> URL: http://svnweb.freebsd.org/changeset/base/250719
> 
> Log:
>  Reverse the sense of the test wrt bmake, and guard against
>  MK_BMAKE not being defined.

I forgot to request this earlier, but could you please document bmake 
becoming the default system make in UPDATING and some of the associated 
caveats? It's particularly important to note that set -e is not being 
implicitly set on the command line (like before) and this is going to 
potentially confuse some people as cd ; ${MAKE} was the pattern that was done 
all over the place previously (like we discussed at YOW after BSDCan, stuff 
where people weren't using set -e and were using semicolons, needs to be fixed 
in Makefile, Makefile.inc1, etc in order for builds to not silently fail as 
they will today).
Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Fwd: conf/178421: [PATCH] compile_et needs to be built with bootstrap-tools for buildworld when WITH_KERBEROS is set

[Garrett Cooper]

FYI... I hit this pretty quickly when doing make tinderbox on my VM 
with a minimally tainted svn tree, so this probably should be fixed ASAP. If 
someone could please do this, I would be grateful (I've hit this before, but my 
memory is a bit short-term about some things and it got lost when I was 
migrating local modifications in the past).
Thanks!
-Garrett

Begin forwarded message:

> From: freebsd-gnats-sub...@freebsd.org
> Subject: Re: conf/178421: [PATCH] compile_et needs to be built with 
> bootstrap-tools for buildworld when WITH_KERBEROS is set
> Date: May 8, 2013 8:10:00 AM PDT
> To: Garrett Cooper 
> Reply-To: freebsd-gnats-sub...@freebsd.org, freebsd-b...@freebsd.org
> 
> Thank you very much for your problem report.
> It has the internal identification `conf/178421'.
> The individual assigned to look at your
> report is: freebsd-bugs. 
> 
> You can access the state of your problem report at any time
> via this link:
> 
> http://www.freebsd.org/cgi/query-pr.cgi?pr=178421
> 
>> Category:   conf
>> Responsible:freebsd-bugs
>> Synopsis:   [PATCH] compile_et needs to be built with bootstrap-tools 
>> for buildworld when WITH_KERBEROS is set
>> Arrival-Date:   Wed May 08 15:10:00 UTC 2013

___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding a variable to .mk and Makefile.inc1 to point to top of the FreeBSD source tree

[Garrett Cooper]

On May 7, 2013, at 10:09 PM, Warner Losh  wrote:

> 
> On May 7, 2013, at 9:42 PM, Garrett Cooper wrote:
> 
>> On Tue, May 7, 2013 at 2:26 PM, Garrett Cooper  wrote:
>> 
>> On Tue, May 7, 2013 at 2:00 PM, Warner Losh  wrote:
>> 
>> On May 7, 2013, at 2:46 PM, Garrett Cooper wrote:
>> 
>>> On May 7, 2013, at 1:39 PM, Brooks Davis wrote:
>>> 
>>>> On Tue, May 07, 2013 at 01:05:07PM -0700, Garrett Cooper wrote:
>>>>> Hi,
>>>>>  A common pattern that I've seen at Isilon and something else that I've
>>>>> wanted to have for a while is the ability to designate where the top of a
>>>>> source tree was. This is important and helpful when dealing with source
>>>>> files that build upon each other or depend on sources located in other
>>>>> sections of the tree; contrib stuff needs to set .PATH appropriately to
>>>>> point to sources at the top of the tree, sys stuff is riddled with S= in
>>>>> order to point to where /sys, etc lives, we build upon FreeBSD within an
>>>>> expected directory structure as well.
>>>>>  I haven't come up with a name, but was wondering if this was a good
>>>>> idea, and if so does anyone have any outstanding patches for this that can
>>>>> be pushed into FreeBSD?
>>>> 
>>>> I'd like to see this.  There's a variable for this in NetBSD and I've
>>>> wanted to do this because it makes code easier to relocate within the
>>>> tree.
>>> 
>>>  This is another good reason. It would make porting code to/from NetBSD 
>>> a LOT easier… especially because I plan on pulling a lot of test/test 
>>> infrastructure code from NetBSD and I really don't want to commit too many 
>>> local changes to the Makefiles. Less divergence -> better cross-pollination 
>>> -> less work for all -> win for the BSDs.
>>>  Thanks for the reminder.. I'll base it off what NetBSD did :).
>> 
>> SRCDIR
>> 
>> EVARINUSE?
>> 
>> share/mk/bsd.doc.mk:# SRCDIRDirectory where source files live.  
>> [${.CURDIR}]
>> share/mk/bsd.doc.mk:TRFLAGS+=   -I${SRCDIR}
>> share/mk/bsd.doc.mk:.PATH: ${.CURDIR} ${SRCDIR}
>> share/mk/bsd.doc.mk:cd ${SRCDIR}; \
>> share/mk/bsd.doc.mk:SRCDIR?=${.CURDIR}
>> share/mk/bsd.doc.mk:cd ${SRCDIR}; ${UNROFF} ${MACROS} ${UNROFFFLAGS} \
>> share/mk/bsd.doc.mk:cd ${SRCDIR}; ${UNROFF} -ms ${UNROFFFLAGS} \
>> share/mk/bsd.info.mk:SRCDIR?=   ${.CURDIR}
>> ...
>> share/doc/llvm/Makefile:SRCDIR= ${.CURDIR}/../../../contrib/llvm
>> share/doc/llvm/Makefile:.PATH: ${SRCDIR} ${SRCDIR}/lib/Support
>> share/doc/llvm/clang/Makefile:SRCDIR=   
>> ${.CURDIR}/../../../../contrib/llvm/tools/clang
>> share/doc/llvm/clang/Makefile:.PATH: ${SRCDIR}
>> 
>> Once upon a time, this *HAD* to be set, and wasn't inferred from the current 
>> top of the tree. Please, for the love of god, make sure that we don't lose 
>> the infer from top of tree ability, or I will hurt you. Often. Through all 
>> the minions that owe me minor favors.
>> 
>> I don't want to break that ever; it's a fantastic feature. If you could 
>> point me to where that magic awesomeness lives (make?), I'll be more than 
>> happy to address it in my branch where I'm going to do this.
>> 
>> I really don't like how NetBSD turned their top-level build command into a 
>> shell script [in part because it needs to bootstrap a bunch of tools]. It 
>> makes things painful when doing iterative builds..
>> 
>> So all in all, I completely and wholeheartedly agree with your concerns.
>> 
>> Oh sweet.. this is something to keep in mind too (from bsd.port.mk)...
>> 
>> # SRC_BASE  - The root of the src tree.  (Some ports require this to get
>> # kernel sources).  Default: /usr/src
>> 
>> All else fails, ports has done it first -_-...
>> 
>> Not the first var I've seen this done with...
> 
> I thought ports specifically named things differently to avoid conflicts.

Perhaps. It matches autoconf too. It's just a pain when there isn't 
consistency, but oh well...
Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding a variable to .mk and Makefile.inc1 to point to top of the FreeBSD source tree

[Garrett Cooper]

On Tue, May 7, 2013 at 2:26 PM, Garrett Cooper  wrote:

>
> On Tue, May 7, 2013 at 2:00 PM, Warner Losh  wrote:
>
>>
>> On May 7, 2013, at 2:46 PM, Garrett Cooper wrote:
>>
>> > On May 7, 2013, at 1:39 PM, Brooks Davis wrote:
>> >
>> >> On Tue, May 07, 2013 at 01:05:07PM -0700, Garrett Cooper wrote:
>> >>> Hi,
>> >>>   A common pattern that I've seen at Isilon and something else that
>> I've
>> >>> wanted to have for a while is the ability to designate where the top
>> of a
>> >>> source tree was. This is important and helpful when dealing with
>> source
>> >>> files that build upon each other or depend on sources located in other
>> >>> sections of the tree; contrib stuff needs to set .PATH appropriately
>> to
>> >>> point to sources at the top of the tree, sys stuff is riddled with S=
>> in
>> >>> order to point to where /sys, etc lives, we build upon FreeBSD within
>> an
>> >>> expected directory structure as well.
>> >>>   I haven't come up with a name, but was wondering if this was a good
>> >>> idea, and if so does anyone have any outstanding patches for this
>> that can
>> >>> be pushed into FreeBSD?
>> >>
>> >> I'd like to see this.  There's a variable for this in NetBSD and I've
>> >> wanted to do this because it makes code easier to relocate within the
>> >> tree.
>> >
>> >   This is another good reason. It would make porting code to/from
>> NetBSD a LOT easier… especially because I plan on pulling a lot of
>> test/test infrastructure code from NetBSD and I really don't want to commit
>> too many local changes to the Makefiles. Less divergence -> better
>> cross-pollination -> less work for all -> win for the BSDs.
>> >   Thanks for the reminder.. I'll base it off what NetBSD did :).
>>
>> SRCDIR
>>
>
> EVARINUSE?
>
> share/mk/bsd.doc.mk:# SRCDIRDirectory where source files live.
> [${.CURDIR}]
> share/mk/bsd.doc.mk:TRFLAGS+=   -I${SRCDIR}
> share/mk/bsd.doc.mk:.PATH: ${.CURDIR} ${SRCDIR}
> share/mk/bsd.doc.mk:cd ${SRCDIR}; \
> share/mk/bsd.doc.mk:SRCDIR?=${.CURDIR}
> share/mk/bsd.doc.mk:cd ${SRCDIR}; ${UNROFF} ${MACROS} ${UNROFFFLAGS} \
> share/mk/bsd.doc.mk:cd ${SRCDIR}; ${UNROFF} -ms ${UNROFFFLAGS} \
> share/mk/bsd.info.mk:SRCDIR?=   ${.CURDIR}
> ...
> share/doc/llvm/Makefile:SRCDIR= ${.CURDIR}/../../../contrib/llvm
> share/doc/llvm/Makefile:.PATH: ${SRCDIR} ${SRCDIR}/lib/Support
> share/doc/llvm/clang/Makefile:SRCDIR=
> ${.CURDIR}/../../../../contrib/llvm/tools/clang
> share/doc/llvm/clang/Makefile:.PATH: ${SRCDIR}
>
>
>> Once upon a time, this *HAD* to be set, and wasn't inferred from the
>> current top of the tree. Please, for the love of god, make sure that we
>> don't lose the infer from top of tree ability, or I will hurt you. Often.
>> Through all the minions that owe me minor favors.
>>
>
> I don't want to break that ever; it's a fantastic feature. If you could
> point me to where that magic awesomeness lives (make?), I'll be more than
> happy to address it in my branch where I'm going to do this.
>
> I really don't like how NetBSD turned their top-level build command into a
> shell script [in part because it needs to bootstrap a bunch of tools]. It
> makes things painful when doing iterative builds..
>
> So all in all, I completely and wholeheartedly agree with your concerns.
>

Oh sweet.. this is something to keep in mind too (from bsd.port.mk)...

# SRC_BASE  - The root of the src tree.  (Some ports require this to get
# kernel sources).  Default: /usr/src

All else fails, ports has done it first -_-...

Not the first var I've seen this done with...

Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding a variable to .mk and Makefile.inc1 to point to top of the FreeBSD source tree

[Garrett Cooper]

On Tue, May 7, 2013 at 2:00 PM, Warner Losh  wrote:

>
> On May 7, 2013, at 2:46 PM, Garrett Cooper wrote:
>
> > On May 7, 2013, at 1:39 PM, Brooks Davis wrote:
> >
> >> On Tue, May 07, 2013 at 01:05:07PM -0700, Garrett Cooper wrote:
> >>> Hi,
> >>>   A common pattern that I've seen at Isilon and something else that
> I've
> >>> wanted to have for a while is the ability to designate where the top
> of a
> >>> source tree was. This is important and helpful when dealing with source
> >>> files that build upon each other or depend on sources located in other
> >>> sections of the tree; contrib stuff needs to set .PATH appropriately to
> >>> point to sources at the top of the tree, sys stuff is riddled with S=
> in
> >>> order to point to where /sys, etc lives, we build upon FreeBSD within
> an
> >>> expected directory structure as well.
> >>>   I haven't come up with a name, but was wondering if this was a good
> >>> idea, and if so does anyone have any outstanding patches for this that
> can
> >>> be pushed into FreeBSD?
> >>
> >> I'd like to see this.  There's a variable for this in NetBSD and I've
> >> wanted to do this because it makes code easier to relocate within the
> >> tree.
> >
> >   This is another good reason. It would make porting code to/from
> NetBSD a LOT easier… especially because I plan on pulling a lot of
> test/test infrastructure code from NetBSD and I really don't want to commit
> too many local changes to the Makefiles. Less divergence -> better
> cross-pollination -> less work for all -> win for the BSDs.
> >   Thanks for the reminder.. I'll base it off what NetBSD did :).
>
> SRCDIR
>

EVARINUSE?

share/mk/bsd.doc.mk:# SRCDIRDirectory where source files live.
[${.CURDIR}]
share/mk/bsd.doc.mk:TRFLAGS+=   -I${SRCDIR}
share/mk/bsd.doc.mk:.PATH: ${.CURDIR} ${SRCDIR}
share/mk/bsd.doc.mk:cd ${SRCDIR}; \
share/mk/bsd.doc.mk:SRCDIR?=${.CURDIR}
share/mk/bsd.doc.mk:cd ${SRCDIR}; ${UNROFF} ${MACROS} ${UNROFFFLAGS} \
share/mk/bsd.doc.mk:cd ${SRCDIR}; ${UNROFF} -ms ${UNROFFFLAGS} \
share/mk/bsd.info.mk:SRCDIR?=   ${.CURDIR}
...
share/doc/llvm/Makefile:SRCDIR= ${.CURDIR}/../../../contrib/llvm
share/doc/llvm/Makefile:.PATH: ${SRCDIR} ${SRCDIR}/lib/Support
share/doc/llvm/clang/Makefile:SRCDIR=
${.CURDIR}/../../../../contrib/llvm/tools/clang
share/doc/llvm/clang/Makefile:.PATH: ${SRCDIR}


> Once upon a time, this *HAD* to be set, and wasn't inferred from the
> current top of the tree. Please, for the love of god, make sure that we
> don't lose the infer from top of tree ability, or I will hurt you. Often.
> Through all the minions that owe me minor favors.
>

I don't want to break that ever; it's a fantastic feature. If you could
point me to where that magic awesomeness lives (make?), I'll be more than
happy to address it in my branch where I'm going to do this.

I really don't like how NetBSD turned their top-level build command into a
shell script [in part because it needs to bootstrap a bunch of tools]. It
makes things painful when doing iterative builds..

So all in all, I completely and wholeheartedly agree with your concerns.

Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: clang doesn't make temporary files in all instances, causes build races by not using mk*temp(3) in /tmp

[Garrett Cooper]

On Tue, May 7, 2013 at 1:58 PM, Garrett Cooper  wrote:
> Hi Dimitriy,
> I ran into the following error when trying to execute make
> tinderbox with FreeBSD svn head with ATF changes I'm going to push to
> benno@:
>
> cc  -O -pipe  -DHAVE_CONFIG_H
> -I/scratch/freebsd/head-svn/gnu/lib/libgomp -I.
> -I/scratch/freebsd/head-svn/gnu/lib/libgomp/../../../contrib/gcclibs/libgomp
> -I/scratch/freebsd/head-svn/gnu/lib/libgomp/../../../contrib/gcclibs/libgomp/config/posix
> -std=gnu99 -Qunused-arguments  -c
> /scratch/freebsd/head-svn/gnu/lib/libgomp/../../../contrib/gcclibs/libgomp/config/posix/bar.c
> -o bar.o
> cc: error: unable to make temporary file: /tmp/bar: can't make unique
> filename: Permission denied
> *** [bar.o] Error code 1
>
> Did some poking around in the clang source and it looks like it's
> doing some less than intelligent things when generating "temporary"
> paths (from contrib/llvm/tools/clang/lib/Driver/Driver.cpp ):
>
> 1598 std::string Driver::GetTemporaryPath(StringRef Prefix, const char 
> *Suffix)
> 1599   const {
> 1600   // FIXME: This is lame; sys::Path should provide this function
> (in particular,
> 1601   // it should know how to find the temporary files dir).
> 1602   std::string Error;
> 1603   const char *TmpDir = ::getenv("TMPDIR");
> 1604   if (!TmpDir)
> 1605 TmpDir = ::getenv("TEMP");
> 1606   if (!TmpDir)
> 1607 TmpDir = ::getenv("TMP");
> 1608   if (!TmpDir)
> 1609 TmpDir = "/tmp";
> 1610   llvm::sys::Path P(TmpDir);
> 1611   P.appendComponent(Prefix);
> 1612   if (P.makeUnique(false, &Error)) {
> 1613 Diag(clang::diag::err_unable_to_make_temp) << Error;
> 1614 return "";
> 1615   }
> 1616
> 1617   // FIXME: Grumble, makeUnique sometimes leaves the file around!?  
> PR3837.
> 1618   P.eraseFromDisk(false, 0);
> 1619
> 1620   if (Suffix)
> 1621 P.appendSuffix(Suffix);
> 1622   return P.str();
>
> This logic (line 1612) is racy and incorrect.
> This _needs_ to be fixed in clang to properly prefix and rename to
> the target path in the filesystem where the compilation is being done
> in order to avoid races with partial compilations, etc.

And I was pointed to these awesome bug reports by someone internal to
Isilon (thanks Anton!):

- http://lists.cs.uiuc.edu/pipermail/llvmbugs/2012-August/024524.html
- http://trac.macports.org/ticket/37834

It looks like the clang folks haven't fixed this issue yet. Arg..

Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

clang doesn't make temporary files in all instances, causes build races by not using mk*temp(3) in /tmp

[Garrett Cooper]

Hi Dimitriy,
I ran into the following error when trying to execute make
tinderbox with FreeBSD svn head with ATF changes I'm going to push to
benno@:

cc  -O -pipe  -DHAVE_CONFIG_H
-I/scratch/freebsd/head-svn/gnu/lib/libgomp -I.
-I/scratch/freebsd/head-svn/gnu/lib/libgomp/../../../contrib/gcclibs/libgomp
-I/scratch/freebsd/head-svn/gnu/lib/libgomp/../../../contrib/gcclibs/libgomp/config/posix
-std=gnu99 -Qunused-arguments  -c
/scratch/freebsd/head-svn/gnu/lib/libgomp/../../../contrib/gcclibs/libgomp/config/posix/bar.c
-o bar.o
cc: error: unable to make temporary file: /tmp/bar: can't make unique
filename: Permission denied
*** [bar.o] Error code 1

Did some poking around in the clang source and it looks like it's
doing some less than intelligent things when generating "temporary"
paths (from contrib/llvm/tools/clang/lib/Driver/Driver.cpp ):

1598 std::string Driver::GetTemporaryPath(StringRef Prefix, const char *Suffix)
1599   const {
1600   // FIXME: This is lame; sys::Path should provide this function
(in particular,
1601   // it should know how to find the temporary files dir).
1602   std::string Error;
1603   const char *TmpDir = ::getenv("TMPDIR");
1604   if (!TmpDir)
1605 TmpDir = ::getenv("TEMP");
1606   if (!TmpDir)
1607 TmpDir = ::getenv("TMP");
1608   if (!TmpDir)
1609 TmpDir = "/tmp";
1610   llvm::sys::Path P(TmpDir);
1611   P.appendComponent(Prefix);
1612   if (P.makeUnique(false, &Error)) {
1613 Diag(clang::diag::err_unable_to_make_temp) << Error;
1614 return "";
1615   }
1616
1617   // FIXME: Grumble, makeUnique sometimes leaves the file around!?  PR3837.
1618   P.eraseFromDisk(false, 0);
1619
1620   if (Suffix)
1621 P.appendSuffix(Suffix);
1622   return P.str();

This logic (line 1612) is racy and incorrect.
This _needs_ to be fixed in clang to properly prefix and rename to
the target path in the filesystem where the compilation is being done
in order to avoid races with partial compilations, etc.
Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding a variable to .mk and Makefile.inc1 to point to top of the FreeBSD source tree

[Garrett Cooper]

On May 7, 2013, at 1:39 PM, Brooks Davis wrote:

> On Tue, May 07, 2013 at 01:05:07PM -0700, Garrett Cooper wrote:
>> Hi,
>>A common pattern that I've seen at Isilon and something else that I've
>> wanted to have for a while is the ability to designate where the top of a
>> source tree was. This is important and helpful when dealing with source
>> files that build upon each other or depend on sources located in other
>> sections of the tree; contrib stuff needs to set .PATH appropriately to
>> point to sources at the top of the tree, sys stuff is riddled with S= in
>> order to point to where /sys, etc lives, we build upon FreeBSD within an
>> expected directory structure as well.
>>I haven't come up with a name, but was wondering if this was a good
>> idea, and if so does anyone have any outstanding patches for this that can
>> be pushed into FreeBSD?
> 
> I'd like to see this.  There's a variable for this in NetBSD and I've
> wanted to do this because it makes code easier to relocate within the
> tree.

This is another good reason. It would make porting code to/from NetBSD 
a LOT easier… especially because I plan on pulling a lot of test/test 
infrastructure code from NetBSD and I really don't want to commit too many 
local changes to the Makefiles. Less divergence -> better cross-pollination -> 
less work for all -> win for the BSDs.
Thanks for the reminder.. I'll base it off what NetBSD did :).
Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding a variable to .mk and Makefile.inc1 to point to top of the FreeBSD source tree

[Garrett Cooper]

On Tue, May 7, 2013 at 1:12 PM, Xin Li  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> On 05/07/13 13:05, Garrett Cooper wrote:
> > Hi, A common pattern that I've seen at Isilon and something else
> > that I've wanted to have for a while is the ability to designate
> > where the top of a source tree was. This is important and helpful
> > when dealing with source files that build upon each other or depend
> > on sources located in other sections of the tree; contrib stuff
> > needs to set .PATH appropriately to point to sources at the top of
> > the tree, sys stuff is riddled with S= in order to point to where
> > /sys, etc lives, we build upon FreeBSD within an expected directory
> > structure as well. I haven't come up with a name, but was wondering
> > if this was a good idea, and if so does anyone have any outstanding
> > patches for this that can be pushed into FreeBSD?
>
> Is there anything wrong with the current '../../' approach?
>

Not in particular, other than our variable (ISI_TOP) is used in
referencing ${.CURDIR} and ${.OBJDIR}, and it's easy to make mistakes if
you goof up the dot-dots. With a properly defined directory like that it
makes things unambiguous in my mind and with a proper name it makes pathing
more intuitive than it currently is. Besides, it would make some other
things cleaner, like the dot-dot magic that config(8), etc does.
Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

[RFC] adding a variable to .mk and Makefile.inc1 to point to top of the FreeBSD source tree

[Garrett Cooper]

Hi,
A common pattern that I've seen at Isilon and something else that I've
wanted to have for a while is the ability to designate where the top of a
source tree was. This is important and helpful when dealing with source
files that build upon each other or depend on sources located in other
sections of the tree; contrib stuff needs to set .PATH appropriately to
point to sources at the top of the tree, sys stuff is riddled with S= in
order to point to where /sys, etc lives, we build upon FreeBSD within an
expected directory structure as well.
I haven't come up with a name, but was wondering if this was a good
idea, and if so does anyone have any outstanding patches for this that can
be pushed into FreeBSD?
Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: Fwd: docs/178286: [PATCH] document the LOCAL_* vars in build(7)

[Garrett Cooper]

On Wed, May 1, 2013 at 3:07 PM, Brooks Davis  wrote:
>
> On Wed, May 01, 2013 at 02:35:17PM -0700, Garrett Cooper wrote:
>> Hi Brooks and Joel,
>> I'd really appreciate if you guys could help out with this. I'm
>> having to fix the Isilon build system due to severe lack of
>> documentation in build(7). So many mistakes have been made because
>> people don't understand how to write FreeBSD Makefiles...
>
> One minor concern I have with this patch is documenting LOCAL_LIB_DIRS.
> I've not posted to the lists yet, but now that I understand buildworld
> better, I think I implemented it wrong.  It should have required that
> the directories be also be listed in LOCAL_DIRS.  The reason is that you
> could then have LOCAL_DIRS=foo and LOCAL_LIB_DIRS=foo/lib so you only
> added one directory to the FreeBSD tree but could still have libs.  I
> feared that someone might have started using this feature which would
> mean we can't easily change that.

That's assuming that someone hasn't hacked the base system (e.g.
bin/cp, etc) and introduced awesome build dependencies on external
libraries that don't cleanly fit into the current structure for
buildworld :). This is part of the reason why I opened up
http://www.freebsd.org/cgi/query-pr.cgi?pr=conf/178066 , but I realize
that this is starting to turn into a slippery slope and could be
articulated via meta-make and proper DPADD definitions _much_ better
than it is today... Makefile.inc1 is full of pixie dust and carefully
placed dependencies today, such that once you step outside that box
it's difficult to work within the system due to its lack of
flexibility and hooks.

It's nice that the __L ".PHONY" rules are automatically generated by
Makefile.inc1 because I really want buildworld-ish encapsulation; in
particular, the way things are done in the OneFS build system could
really be done better -- I'm not confident that some of my
predecessors working on the build system truly understood how the
FreeBSD build process works or were under extreme time crunches and
thus didn't look for a perfect/good solution and instead resorted to
some nasty build hacks that I'm now trying to unwind.

> Is Isilon using it?

Russell and I started using it recently. I'd like this support to
stick around until meta-make can be realized (even if it's going to
get ripped out eventually...) as it does make things cleaner when
people hack away at the base system.

Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Fwd: docs/178286: [PATCH] document the LOCAL_* vars in build(7)

[Garrett Cooper]

Hi Brooks and Joel,
I'd really appreciate if you guys could help out with this. I'm
having to fix the Isilon build system due to severe lack of
documentation in build(7). So many mistakes have been made because
people don't understand how to write FreeBSD Makefiles...
I'll shoot along more patches as I can.
Thanks,
-Garrett

-- Forwarded message --
From:  
Date: Wed, May 1, 2013 at 2:30 PM
Subject: Re: docs/178286: [PATCH] document the LOCAL_* vars in build(7)
To: Garrett Cooper 


Thank you very much for your problem report.
It has the internal identification `docs/178286'.
The individual assigned to look at your
report is: freebsd-doc.

You can access the state of your problem report at any time
via this link:

http://www.freebsd.org/cgi/query-pr.cgi?pr=178286

>Category:   docs
>Responsible:freebsd-doc
>Synopsis:   [PATCH] document the LOCAL_* vars in build(7)
>Arrival-Date:   Wed May 01 21:30:00 UTC 2013
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] [Optionally] build tests with buildworld

[Garrett Cooper]

On Apr 24, 2013, at 9:23 AM, Brooks Davis wrote:

>> …

>> Hi Brooks!
>>  Does this look ok? Most of the MK_TESTS logic has been shoved into 
>> bsd.own.mk and NO_TESTS is sprinkled around Makefile.inc1 as recommended.
>> Thanks!
>> -Garrett
> 
> The functional parts seem fine.  Some comments below.

Here's an updated version of the patch with your comments applied.
Thanks!
-Garrett



build-tests-with-world.patch
Description: Binary data
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] [Optionally] build tests with buildworld

[Garrett Cooper]

On Apr 23, 2013, at 10:28 AM, Garrett Cooper wrote:

> On Apr 23, 2013, at 9:59 AM, Brooks Davis wrote:
> 
>> On Fri, Apr 19, 2013 at 11:17:50PM -0700, Garrett Cooper wrote:
>>> Hi arch@ and toolchain@,
>>>   One of the items that I'm proposing be added to Makefile.inc1 in
>>> order to make building and installing tests on CURRENT (ATF and
>>> otherwise) is a build knob called TESTS_WITH_WORLD (the name can be
>>> modified), which allows me to build and install various tests on my
>>> git branch like the example ATF tests I produced, pjdfstest, some of
>>> the prove tests from tools/regression, etc (there are other
>>> outstanding changes, but this was the key one that I need feedback on
>>> just to be safe).
>> 
>> I don't understand way you don't use WITH_TESTS processed
>> through bsd.own.mk.  You'd presumably have to add it to the list of
>> supported NO_* options for the bootstrap case, but that's trivial.  Then
>> you use could use normal MK_* variables.  That would also let you use
>> WITH(OUT)_TESTS in individual directories and they would always work.
>> 
>> At a glance using WITHOUT_TESTS and NO_TEST internally would simplify
>> some of the special cases in your patch.
> 
> This is something that I considered, but I wasn't sure that it was the best 
> route to go about things because I thought we were doing away/had done away 
> with most of the NO_* knobs (and unfortunately one cannot mix and match 
> WITH_* and WITHOUT_* because they're considered contradictory according to 
> bsd.own.mk -- something that Simon has debated against having in the past).
> 
> I'm all for doing that though because that would simplify things greatly from 
> an end-user perspective.

Hi Brooks!
Does this look ok? Most of the MK_TESTS logic has been shoved into 
bsd.own.mk and NO_TESTS is sprinkled around Makefile.inc1 as recommended.
Thanks!
-Garrett


build-tests-with-world.patch
Description: Binary data
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] [Optionally] build tests with buildworld

[Garrett Cooper]

Hi Brooks!

On Apr 23, 2013, at 9:59 AM, Brooks Davis wrote:

> On Fri, Apr 19, 2013 at 11:17:50PM -0700, Garrett Cooper wrote:
>> Hi arch@ and toolchain@,
>>One of the items that I'm proposing be added to Makefile.inc1 in
>> order to make building and installing tests on CURRENT (ATF and
>> otherwise) is a build knob called TESTS_WITH_WORLD (the name can be
>> modified), which allows me to build and install various tests on my
>> git branch like the example ATF tests I produced, pjdfstest, some of
>> the prove tests from tools/regression, etc (there are other
>> outstanding changes, but this was the key one that I need feedback on
>> just to be safe).
> 
> I don't understand way you don't use WITH_TESTS processed
> through bsd.own.mk.  You'd presumably have to add it to the list of
> supported NO_* options for the bootstrap case, but that's trivial.  Then
> you use could use normal MK_* variables.  That would also let you use
> WITH(OUT)_TESTS in individual directories and they would always work.
> 
> At a glance using WITHOUT_TESTS and NO_TEST internally would simplify
> some of the special cases in your patch.

This is something that I considered, but I wasn't sure that it was the best 
route to go about things because I thought we were doing away/had done away 
with most of the NO_* knobs (and unfortunately one cannot mix and match WITH_* 
and WITHOUT_* because they're considered contradictory according to bsd.own.mk 
-- something that Simon has debated against having in the past).

I'm all for doing that though because that would simplify things greatly from 
an end-user perspective.

> On a side note, I like the HMAKE change independent of the rest of the
> patch.

Cool! I'll polish up the patch with that piece and send it over to benno@ :).

Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] [Optionally] build tests with buildworld

[Garrett Cooper]

On Apr 19, 2013, at 11:17 PM, Garrett Cooper  wrote:

> Hi arch@ and toolchain@,
>One of the items that I'm proposing be added to Makefile.inc1 in
> order to make building and installing tests on CURRENT (ATF and
> otherwise) is a build knob called TESTS_WITH_WORLD (the name can be
> modified), which allows me to build and install various tests on my
> git branch like the example ATF tests I produced, pjdfstest, some of
> the prove tests from tools/regression, etc (there are other
> outstanding changes, but this was the key one that I need feedback on
> just to be safe).
>The effective change is attached (Gmail will no doubt mangle it,
> so please let me know if you want another copy). I made the change to
> Makefile.inc1 in order to ensure that the change was self-contained
> and because it was the simplest, cleanest way to do things without
> introducing a lot of unwanted complexity.
>I'm asking for feedback on the following items:
>1. Does the change make functional sense? If not, why?
>2. Do the semantics (variable names, whether or not they're
> defined) need to be modified to match MK_* semantics or be made more
> consistent in any particular way? If so, why?
>3. Will anyone have serious heartburn (already have a similar
> change implemented, think it's done in a backwards manner) if this
> change is implemented? If so, why?
> Thanks!
> -Garrett
> 
> PS Please CC me on all replies as I'm not subscribed to the list.
> 

Ping? If no one replies in a week I'll consider it implicit agreement :)..
Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

[RFC] [Optionally] build tests with buildworld

[Garrett Cooper]

Hi arch@ and toolchain@,
One of the items that I'm proposing be added to Makefile.inc1 in
order to make building and installing tests on CURRENT (ATF and
otherwise) is a build knob called TESTS_WITH_WORLD (the name can be
modified), which allows me to build and install various tests on my
git branch like the example ATF tests I produced, pjdfstest, some of
the prove tests from tools/regression, etc (there are other
outstanding changes, but this was the key one that I need feedback on
just to be safe).
The effective change is attached (Gmail will no doubt mangle it,
so please let me know if you want another copy). I made the change to
Makefile.inc1 in order to ensure that the change was self-contained
and because it was the simplest, cleanest way to do things without
introducing a lot of unwanted complexity.
I'm asking for feedback on the following items:
1. Does the change make functional sense? If not, why?
2. Do the semantics (variable names, whether or not they're
defined) need to be modified to match MK_* semantics or be made more
consistent in any particular way? If so, why?
3. Will anyone have serious heartburn (already have a similar
change implemented, think it's done in a backwards manner) if this
change is implemented? If so, why?
Thanks!
-Garrett

PS Please CC me on all replies as I'm not subscribed to the list.


TESTS_WITH_WORLD.patch
Description: Binary data
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] teach config(8) about KERNCONFDIR

[Garrett Cooper]

On Wed, Dec 5, 2012 at 9:23 PM, Garrett Cooper  wrote:
> (Sorry in advance for GMail mangling).
>
> I recently moved all of my svn trees to git and in order to ensure
> that my trees are in as pristine a state as possible I've started
> setting KERNCONFDIR=/root in /etc/src.conf and I specify KERNCONF?=
>  -DEBUG. This works pretty well in a standard setup (when
> files are located in /sys//conf, are standalone, etc). In order
> to eliminate duplication, I wrote -DEBUG to include  and
> augment it a bit (enable WITNESS, INVARIANTS, etc). However, I
> discovered that if I do this, config fails to find the file because it
> only knows how to find 1) absolute paths, 2) relative paths to the obj
> dir, and 3) as a last ditch effort it tries to use ../../conf/
> (/sys/conf I believe for the generic KERNCONF directory). I added the
> patch below to teach config(8) about KERNCONFDIR for my own local use
> (and I think fixed a subtle bug with asprintf and free usage), but the
> support seems hacky. Another alternative (after doing a bit of
> research) that looks better is NetBSD's concept of prefix paths:

(for the archive's sake)
I ran buildkernel again and the patch doesn't work as-is if it's
not defined in the environment as it isn't one of the variables that's
passed through by default via make to buildkernel when running
config(8), so one needs to modify Makefile.inc1 as well in order for
things to function properly as described with my patch.
Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

[RFC] teach config(8) about KERNCONFDIR

[Garrett Cooper]

(Sorry in advance for GMail mangling).

I recently moved all of my svn trees to git and in order to ensure
that my trees are in as pristine a state as possible I've started
setting KERNCONFDIR=/root in /etc/src.conf and I specify KERNCONF?=
 -DEBUG. This works pretty well in a standard setup (when
files are located in /sys//conf, are standalone, etc). In order
to eliminate duplication, I wrote -DEBUG to include  and
augment it a bit (enable WITNESS, INVARIANTS, etc). However, I
discovered that if I do this, config fails to find the file because it
only knows how to find 1) absolute paths, 2) relative paths to the obj
dir, and 3) as a last ditch effort it tries to use ../../conf/
(/sys/conf I believe for the generic KERNCONF directory). I added the
patch below to teach config(8) about KERNCONFDIR for my own local use
(and I think fixed a subtle bug with asprintf and free usage), but the
support seems hacky. Another alternative (after doing a bit of
research) that looks better is NetBSD's concept of prefix paths:

 prefix [path]
 If path is given, it pushes a new prefix for include and cinclude.
 prefix statements act like a stack, and an empty path argument has the
 latest prefix popped out.  The path argument is either absolute or rela-
 tive to the current defined prefix, which defaults to the top of ther
 kernel source tree.

This sounds like a more generic way to resolve the issue when combined
with defparam as described in config(5) [1].

I was wondering if anyone would be interested in this support or not
(assuming that porting over the feature is trivial enough to do). If
so I'll look at just porting over the support for defparam and prefix
directives and will submit a PR for it if successful. If no one's
interested in it, I can just hack my src.conf to use /root or
autoprefix the path to the ${KERNCONF} and be done with it.

Thanks!
-Garrett

PS Please CC me when replying as I'm not currently subscribed to toolchain@.

1. http://netbsd.gw.com/cgi-bin/man-cgi?config+5+NetBSD-current

$ git diff usr.sbin/config/
diff --git a/usr.sbin/config/lang.l b/usr.sbin/config/lang.l
index 81f820f..2eba2b8 100644
--- a/usr.sbin/config/lang.l
+++ b/usr.sbin/config/lang.l
@@ -34,6 +34,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include "y.tab.h"
 #include "config.h"
@@ -255,22 +257,31 @@ include(const char *fname, int ateof)
 {
FILE *fp;
struct incl *in;
-   char *fnamebuf;
+   char *fnamebuf, *kernconfdir;

fnamebuf = NULL;
fp = fopen(fname, "r");
if (fp == NULL && fname[0] != '.' && fname[0] != '/') {
asprintf(&fnamebuf, "../../conf/%s", fname);
-   if (fnamebuf != NULL) {
+   if (fnamebuf != NULL)
fp = fopen(fnamebuf, "r");
-   free(fnamebuf);
+   }
+   /* If the first attempt failed, try again in ${KERNCONFDIR}. */
+   if (fp == NULL) {
+   kernconfdir = getenv("KERNCONFDIR");
+   if (kernconfdir != NULL) {
+   asprintf(&fnamebuf, "%s/%s", kernconfdir, fname);
+   if (fnamebuf != NULL)
+   fp = fopen(fnamebuf, "r");
}
}
if (fp == NULL) {
+   free(fnamebuf);
yyerror("cannot open included file");
return (-1);
}
cfgfile_add(fnamebuf == NULL ? fname : fnamebuf);
+   free(fnamebuf);
in = malloc(sizeof(*in));
assert(in != NULL);
in->in_prev = inclp;
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Fwd: [head tinderbox] failure on amd64/amd64

[Garrett Cooper]

FYI.. these errors LINT errors have popped up on amd64/i386/pc98.

-- Forwarded message --
From: FreeBSD Tinderbox 
Date: Tue, Nov 6, 2012 at 12:29 AM
Subject: [head tinderbox] failure on amd64/amd64
To: FreeBSD Tinderbox , curr...@freebsd.org,
am...@freebsd.org


TB --- 2012-11-06 04:40:00 - tinderbox 2.9 running on
freebsd-current.sentex.ca
TB --- 2012-11-06 04:40:00 - FreeBSD
freebsd-current.sentex.ca8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon
Mar 26 13:54:12 EDT 2012
d...@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC  amd64
TB --- 2012-11-06 04:40:00 - starting HEAD tinderbox run for amd64/amd64
TB --- 2012-11-06 04:40:00 - cleaning the object tree
TB --- 2012-11-06 04:40:00 - checking out /src from svn://
svn.freebsd.org/base/head
TB --- 2012-11-06 04:40:00 - cd /tinderbox/HEAD/amd64/amd64
TB --- 2012-11-06 04:40:00 - /usr/local/bin/svn cleanup /src
TB --- 2012-11-06 04:43:26 - /usr/local/bin/svn update /src
TB --- 2012-11-06 04:44:15 - At svn revision 242655
TB --- 2012-11-06 04:44:16 - building world
TB --- 2012-11-06 04:44:16 - CROSS_BUILD_TESTING=YES
TB --- 2012-11-06 04:44:16 - MAKEOBJDIRPREFIX=/obj
TB --- 2012-11-06 04:44:16 - PATH=/usr/bin:/usr/sbin:/bin:/sbin
TB --- 2012-11-06 04:44:16 - SRCCONF=/dev/null
TB --- 2012-11-06 04:44:16 - TARGET=amd64
TB --- 2012-11-06 04:44:16 - TARGET_ARCH=amd64
TB --- 2012-11-06 04:44:16 - TZ=UTC
TB --- 2012-11-06 04:44:16 - __MAKE_CONF=/dev/null
TB --- 2012-11-06 04:44:16 - cd /src
TB --- 2012-11-06 04:44:16 - /usr/bin/make -B buildworld
>>> Building an up-to-date make(1)
>>> World build started on Tue Nov  6 04:44:23 UTC 2012
>>> Rebuilding the temporary build tree
>>> stage 1.1: legacy release compatibility shims
>>> stage 1.2: bootstrap tools
>>> stage 2.1: cleaning up the object tree
>>> stage 2.2: rebuilding the object tree
>>> stage 2.3: build tools
>>> stage 3: cross tools
>>> stage 4.1: building includes
>>> stage 4.2: building libraries
>>> stage 4.3: make dependencies
>>> stage 4.4: building everything
>>> stage 5.1: building 32 bit shim libraries
>>> World build completed on Tue Nov  6 08:28:10 UTC 2012
TB --- 2012-11-06 08:28:10 - generating LINT kernel config
TB --- 2012-11-06 08:28:10 - cd /src/sys/amd64/conf
TB --- 2012-11-06 08:28:10 - /usr/bin/make -B LINT
TB --- 2012-11-06 08:28:10 - cd /src/sys/amd64/conf
TB --- 2012-11-06 08:28:10 - /usr/sbin/config -m LINT
TB --- 2012-11-06 08:28:11 - building LINT kernel
TB --- 2012-11-06 08:28:11 - CROSS_BUILD_TESTING=YES
TB --- 2012-11-06 08:28:11 - MAKEOBJDIRPREFIX=/obj
TB --- 2012-11-06 08:28:11 - PATH=/usr/bin:/usr/sbin:/bin:/sbin
TB --- 2012-11-06 08:28:11 - SRCCONF=/dev/null
TB --- 2012-11-06 08:28:11 - TARGET=amd64
TB --- 2012-11-06 08:28:11 - TARGET_ARCH=amd64
TB --- 2012-11-06 08:28:11 - TZ=UTC
TB --- 2012-11-06 08:28:11 - __MAKE_CONF=/dev/null
TB --- 2012-11-06 08:28:11 - cd /src
TB --- 2012-11-06 08:28:11 - /usr/bin/make -B buildkernel KERNCONF=LINT
>>> Kernel build for LINT started on Tue Nov  6 08:28:11 UTC 2012
>>> stage 1: configuring the kernel
>>> stage 2.1: cleaning up the object tree
>>> stage 2.2: rebuilding the object tree
>>> stage 2.3: build tools
>>> stage 3.1: making dependencies
[...]
cc: warning: argument unused during compilation: '-falign-functions=16'
[-Wunused-command-line-argument]
env NM='nm' sh /src/sys/kern/genassym.sh ia32_genassym.o > ia32_assym.h
uudecode < /usr/share/syscons/fonts/cp850-8x16.fnt && file2c 'static u_char
dflt_font_16[16*256] = {' '};' < cp850-8x16 > font.h && uudecode <
/usr/share/syscons/fonts/cp850-8x14.fnt && file2c 'static u_char
dflt_font_14[14*256] = {' '};' < cp850-8x14 >> font.h && uudecode <
/usr/share/syscons/fonts/cp850-8x8.fnt && file2c 'static u_char
dflt_font_8[8*256] = {' '};' < cp850-8x8 >> font.h
/usr/sbin/kbdcontrol -L jp.106 | sed -e 's/^static keymap_t.* = /static
keymap_t key_map = /' -e 's/^static accentmap_t.* = /static accentmap_t
accent_map = /' > atkbdmap.h
/usr/sbin/kbdcontrol -L it.iso | sed -e 's/^static keymap_t.* = /static
keymap_t key_map = /' -e 's/^static accentmap_t.* = /static accentmap_t
accent_map = /' > ukbdmap.h
sed -e 's/^.*#include.*phy\.h.*$//' /src/sys/contrib/dev/nve/os.h >
os+%DIKED-nve.h
cc -c -x assembler-with-cpp -DLOCORE -O2 -pipe -fno-strict-aliasing
 -std=c99  -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes
 -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef
-Wno-pointer-sign -fformat-extensions  -Wmissing-include-dirs
-fdiagnostics-show-option  -Wno-error-tautological-compare
-Wno-error-empty-body  -Wno-error-parentheses-equality -nostdinc  -I.
-I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/ipfilter
-I/src/sys/dev/ath -I/src/sys/dev/ath/ath_hal -I/src/sys/contrib/ngatm
-I/src/sys/dev/twa -I/src/sys/dev/cxgb -I/src/sys/dev/cxgbe -D_KERNEL
-DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -DGPROF
-falign-functions=16 -DGPROF4 -DGUPROF -fno-builtin -fno-omit-frame-pointer
-mno-aes -mno-avx -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse
-m

Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program

[Garrett Cooper]

On Sep 24, 2012, at 1:21 AM, "Poul-Henning Kamp"  wrote:

> In message 
> 
> , Garrett Cooper writes:
> 
>> No difference proven at 95.0% confidence
> 
> This is the important bit of information...

Yeah.. It's been a few moons since I've taken statistics.

Some other pieces of data:

This was done with...
- testing was done multiuser on a lightly loaded system.
- build was done with -j24 on a my workstation with 4 SMT enabled cores.
- the scratch disk is ata enabled, whereas the sources were housed on an mfi 
backed disk.
Thanks!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program

[Garrett Cooper]

On Sat, Sep 22, 2012 at 11:30 PM, Poul-Henning Kamp  wrote:
> In message 
> 
> , Garrett Cooper writes:
>
>>Without the change:
>>
>>$ python calc_runtime.py bw.*_without.log | ministat -w 72
>>[...]
>>$ python calc_runtime.py bw.*.log | ministat -w 72
>
> Try:
>python calc_runtime.py bw.*_without.log > _without
>python calc_runtime.py bw.*.log > _with
>ministat -w 72 _with _without
>
> :-)
>
> (PS: your two chosen glob patterns are not exclusive, but I suppose
> that was for illustration only)

Forgot to mention that I ran the "with" results before the
"without" results, so technically the files didn't exist yet and hence
the results are separate. But, just for absolute clarity for archiving
sake (and provided your suggestion on how to overlay both graphs),
here are the results:

$ for i in bw.*[0-9].log; do mv $i `echo $i | sed -e
's/\.log/_with\.log/g'`; done
$ python calc_runtime.py bw.*_with.log > _with
$ python calc_runtime.py bw.*_without.log > _without
$ ministat -w 72 _with _without
x _with
+ _without
++
| +x |
|++  +   xx +   x   xx+ ++ x x  +xx +|
| ||A_MA___M___|__|  |
++
N   Min   MaxMedian   AvgStddev
x  10   940  1002   988 972.3 24.476973
+  10   919  1010   972 958.9 33.994934
No difference proven at 95.0% confidence

Thanks for the tip :)!
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program

[Garrett Cooper]

On Sun, Sep 2, 2012 at 11:01 PM, Garrett Cooper  wrote:
> Hello,
> I've been a bit busy working on porting over ATF from NetBSD, and
> one of the pieces that's currently not available in FreeBSD that's
> available in NetBSD is the ability to understand and compile multiple
> programs. In order to do this I had to refactor bsd.prog.mk (a lot).
> The attached patch is the end result so far, and I was wondering if
> anyone could please review it and/or test it (outside of me doing so).
> I wrote over 40 tests, but it's not exercising everything, and I would
> like for someone to please review/test this out who has an interest in
> NLS support (ala bsd.nls.mk) in particular. AFAICT this is the only
> gap that I couldn't resolve right away (there isn't a ton of recent
> documentation on how to use bsd.nls.mk).
> I'll run a micro benchmark and buildworld a few times (in
> progress) with and without the change to measure the performance
> effect.
> Any assistance would be much appreciated.

I've attached an updated version of the patch (run through
buildworld successfully multiple times; make universe successfully;
going to submit for an -exp run). I needed to modify bsd.crunchgen.mk
to grok OBJS.${PROG}, but apart from that backwards compatibility has
been maintained -- sans-INSTALLFLAGS_EDIT (does anyone still use that
undocumented functionality?).
Performance wise, it's slightly slower on my W3520 with the
change, but not by much (~20 seconds).
As always, questions and comments are welcome, and if someone has
a chance, please review the proposed patch.
Thanks!
-Garrett

Without the change:

$ python calc_runtime.py bw.*_without.log | ministat -w 72
x 
++
| x  |
|xx  x  x x xx  x   x|
| |_A_M| |
++
N   Min   MaxMedian   AvgStddev
x  10   919  1010   972 958.9 33.994934

With the change:

$ python calc_runtime.py bw.*.log | ministat -w 72
x 
++
|   x|
|xx   xx  x x  x   xx|
| |___A_M_|  |
++
N   Min   MaxMedian   AvgStddev
x  10   940  1002   988 972.3 24.476973

make universe results:

# make universe MAKE_JUST_WORLDS=y
--
>>> make universe started on Tue Sep 18 09:30:04 PDT 2012
--
>> amd64 started on Tue Sep 18 09:30:04 PDT 2012
>> amd64.amd64 buildworld started on Tue Sep 18 09:30:04 PDT 2012
>> amd64.amd64 buildworld completed on Tue Sep 18 11:20:19 PDT 2012
>> amd64 completed on Tue Sep 18 11:20:19 PDT 2012
>> arm started on Tue Sep 18 11:20:19 PDT 2012
>> arm.arm buildworld started on Tue Sep 18 11:20:19 PDT 2012
>> arm.arm buildworld completed on Tue Sep 18 12:25:24 PDT 2012
>> arm.armeb buildworld started on Tue Sep 18 12:25:24 PDT 2012
>> arm.armeb buildworld completed on Tue Sep 18 13:30:25 PDT 2012
>> arm.armv6 buildworld started on Tue Sep 18 13:30:25 PDT 2012
>> arm.armv6 buildworld completed on Tue Sep 18 14:35:14 PDT 2012
>> arm.armv6eb buildworld started on Tue Sep 18 14:35:14 PDT 2012
>> arm.armv6eb buildworld completed on Tue Sep 18 15:40:05 PDT 2012
>> arm completed on Tue Sep 18 15:40:05 PDT 2012
>> i386 started on Tue Sep 18 15:40:05 PDT 2012
>> i386.i386 buildworld started on Tue Sep 18 15:40:05 PDT 2012
>> i386.i386 buildworld completed on Tue Sep 18 16:56:06 PDT 2012
>> i386 completed on Tue Sep 18 16:56:06 PDT 2012
>> ia64 started on Tue Sep 18 16:56:06 PDT 2012
>> ia64.ia64 buildworld started on Tue Sep 18 16:56:06 PDT 2012
>> ia64.ia64 buildworld completed on Tue Sep 18 18:27:49 PDT 2012
>> ia64 completed on Tue Sep 18 18:27:49 PDT 2012
>> mips started on Tue Sep 18 18:27:49 PDT 2012
>> mips.mipsel buildworld started on Tue Sep 18 18:27:49 PDT 2012
>> mips.mipsel buildworld completed on Tue Sep 18 19:34:50 PDT 2012
>> mips.mips buildworld started on Tue Sep 18 19:34:50 PDT 2012
>> mips.mips buildworld completed on Tue Sep 18 20:41:49 PDT 2012
>> mips.mips64el buildworld started on 

Re: Compiler performance tests on FreeBSD 10.0-CURRENT

[Garrett Cooper]

On Sat, Sep 15, 2012 at 10:19 PM, Konstantin Belousov
 wrote:
> On Sun, Sep 16, 2012 at 12:34:45AM +0200, Dimitry Andric wrote:
>> Hi all,
>>
>> By request, I performed a series of kernel performance tests on FreeBSD
>> 10.0-CURRENT, particularly comparing the runtime performance of GENERIC
>> kernels compiled by gcc 4.2.1 and by clang 3.2.
>>
>> The attached text file[1] contains more information about the tests,
>> some semi-cooked performance data, and my conclusions.  Any errors and
>> omissions are also my fault, so if you notice them, please let me know.
>>
>> The executive summary: GENERIC kernels compiled with clang 3.2 are
>> slightly faster than those compiled by gcc 4.2.1, though the difference
>> will not very noticeable in practice.
>>
>> Last but not least, thanks to Gavin Atkinson for providing the required
>> hardware.
>
> Thank you very much for doing this.
>
> I tried to map the CPUID into more human-friendly family moniker, and it
> seems that these are Pentium-4 class CPUs. Am I right ?
>
> If yes, could you, please, rerun the tests on anything more recent than
> Core2, i.e. any Core i7-whatever class of Xeons ?

If you can provide the tests, I can rerun it on some Nehalem class
workstations I have access to. I unfortunately don't have access to
SNB/Romley hardware yet.
Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: Clang as default compiler November 4th

[Garrett Cooper]

On Tue, Sep 11, 2012 at 1:44 PM, Mark Linimon  wrote:
> On Tue, Sep 11, 2012 at 10:07:04AM +0100, David Chisnall wrote:
>> There is some logic in the clang driver already for knowing when it is
>> invoked as gcc.  I'd be quite tempted to make gcc a symlink to clang
>> and make clang default to gnu89 when invoked in that way.
>
> And how then does a port say "I don't compile with clang no matter how
> it is invoked"?

Here's one way:

$ clang -dumpspecs
clang: error: unsupported option '-dumpspecs'
clang: error: no input files
$ gcc -dumpspecs | grep -q gcc && echo "gotcha"
gotcha
$

Also,

$ cat Makefile
.if !empty(CC:M*clang*) || !empty(CXX:M*clang*)
IGNORE= does not compile with clang
.warning ${IGNORE}
.endif

all:
$ make CXX=clang++
"Makefile", line 3: warning: does not compile with clang
$ make CXX=g++
$

But I figured that's probably done elsewhere I bit more sanely.

Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: Clang as default compiler November 4th

[Garrett Cooper]

On Sep 11, 2012, at 8:35 AM, Daniel Eischen wrote:

> On Tue, 11 Sep 2012, Konstantin Belousov wrote:
> 
>> On Tue, Sep 11, 2012 at 02:06:49PM +0200, Roman Divacky wrote:
>>> 
>>> We currently dont compile 4680 ports (out of 23857). Top 10 ports that 
>>> prevent
>>> the most other ports from compiling together prevent  ports from
>>> compilation. So if we fixed those 10 ports we could be at around 2500 ports
>>> not compiling. Thats quite far from your claim of forking 20k programs.
>> 
>> Sorry, I cannot buy the argument. How many patches there are already
>> in the ports tree to cope with clang incompatibility with gcc ? You may
>> declare that all of them are application bugs, but it completely misses
>> the point.
> 
> [ snip ]
> 
>>> I believe majority of the broken ports is broken because their maintainer
>>> never saw them being broken with clang just because it's not the default
>>> compiler. Thus by making it the default majority of the problems would just
>>> go away.
>> 
>> Can you, please, read what I wrote ? Fixing _ports_ to compile with
>> clang is plain wrong. Upstream developers use gcc almost always for
>> development and testing. Establishing another constant cost on the
>> porting work puts burden on the ports submitters, maintainers and even
>> ports users.
> 
> This is a good point!

Alternate compilers are being used on other OS distributions, like Arch Linux, 
Gentoo Linux, etc, so encouraging external developers to correct/simplify their 
Makefiles and build infrastructures is a good thing (and plus, it makes 
switching to other compilers like icc, pcc, etc easier).

You're going to run into almost the same problem when trying to get stuff to 
cross-compile for multiple targets, so there's no reason why FreeBSD/Linux 
should not strive to get others to hardcode less.

I wouldn't consider ports to be a stopgap for the clang switchover as much as 
correctness/performance. Broken third-party software can be fixed, but if the 
underlying foundation doesn't deliver sane code or severely regresses 
performance (runtime is more important than building IMO because I'd rather 
have code take a little while longer to compile if the end-result runs faster, 
and ultimately runtime performance affects build performance), then there's no 
point in trying to switch over yet.

Thanks,
-Garrett___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [rfc] a few kern.mk and bsd.sys.mk related changes

[Garrett Cooper]

On May 31, 2011, at 11:45 AM, Dimitry Andric  wrote:

> On 2011-05-31 17:26, Alexander Best wrote:
> ...
>> so how about the following patch?
> 
> Could you please try to not mix spacing and style changes with
> functional changes in this patch?  It makes it more difficult to review.

+1
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [rfc] a few kern.mk and bsd.sys.mk related changes

[Garrett Cooper]

On May 31, 2011, at 3:46 AM, Alexander Best  wrote:

> On Tue May 31 11, Dimitry Andric wrote:
>> On 2011-05-31 11:57, Alexander Best wrote:
>> ...
> however i've often read messages - mostly by bruce evans - claiming that
> anything greater than -O will in fact decrease a kernel's ability to be
> debugged just as well as a kernel with -O.
 The critical option when -O2 is used is -fno-omit-frame-pointers, since 
 removing
 frame pointers makes debugging impossible (on i386). With -O2 code is 
 moved around and
 removed, so debugging is more difficult, but can still provide useful
 information.
>>> any reason we cannot use -O2 -fno-omit-frame-pointers -fno-strict-aliasing 
>>> as
>>> standard COPTFLAGS with debugging enabled for *all* archs?
>> 
>> Most likely, the performance gain from -O2 is rather small, except for
>> special cases, but the pain during debugging is increased a great deal.
>> 
>> Even if you add frame pointers, with -O2 large pieces of code can be
>> transformed, variables or even entire functions can be completely
>> eliminated, and so on, making debugging much more difficult.
> 
> *lol* we're moving in circles. so back to the beginning: why not use -O
> for all archs, if debugging was enabled? for amd64 -O2 is always set, no
> matter, if debugging is enabled or disabled.

I don't know, but I've run into cases where gcc has inlined or shuffled around 
code on amd64 with -O2 -fno-strict-aliasing, so I changed my make.conf to use 
-O0 when DEBUG_FLAGS was defined.
Thanks,
-Garrett___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding -Wmissing-include-dirs to CWARNFLAGS

[Garrett Cooper]

On Thu, Apr 7, 2011 at 2:22 PM, Alexander Best  wrote:
> On Thu Apr  7 11, Garrett Cooper wrote:
>> On Thu, Apr 7, 2011 at 1:53 PM, Garrett Cooper  wrote:
>> > On Thu, Apr 7, 2011 at 11:55 AM, Alexander Best  
>> > wrote:
>> >> hi there,
>> >>
>> >> i'd like to propose adding -Wmissing-include-dirs to CWARNFLAGS. this 
>> >> will let
>> >> tinderbox fail, if any new kernel code was committed with (a) broken 
>> >> include
>> >> dir(s).
>> >>
>> >> i ran a test via
>> >>
>> >> make toolchains
>> >> make MAKE_JUST_KERNELS=yes tinderbox
>> >>
>> >> and nothing seemed to go wrong with the extra warning enabled. i even 
>> >> found a
>> >> missing include dir, which should be fixed by the attached patch.
>> >>
>> >> opinions?
>> >>
>> >> so far i've only tested this with gcc. i think someone on #freebsd-clang 
>> >> told
>> >> me that -Wmissing-include-dirs is a noop for clang (for whatever reasons).
>> >
>> >    make -f /etc/src.conf -VMODULES_OVERRIDE and make -f /etc/src.conf
>> > -VMODULES_OVERRIDE say... (tinderbox should also really ignore these
>> > files, but it doesn't currently)?
>
> otaku% make -f /etc/src.conf -VMODULES_OVERRIDE
> netgraph/netgraph netgraph/socket netgraph/bluetooth/bluetooth  
> netgraph/bluetooth/hci netgraph/bluetooth/l2cap netgraph/bluetooth/socket  
> netgraph/bluetooth/ubt tmpfs sound/sound sound/driver/hda usb/uhid  procfs 
> pseudofs linux linprocfs linsysfs lindev usb/quirk geom  opensolaris dtrace 
> cyclic nfsclient krpc nfs_common nfslock
> otaku% make -f /etc/make.conf -VMODULES_OVERRIDE
> netgraph/netgraph netgraph/socket netgraph/bluetooth/bluetooth  
> netgraph/bluetooth/hci netgraph/bluetooth/l2cap netgraph/bluetooth/socket  
> netgraph/bluetooth/ubt tmpfs sound/sound sound/driver/hda usb/uhid  procfs 
> pseudofs linux linprocfs linsysfs lindev usb/quirk geom  opensolaris dtrace 
> cyclic nfsclient krpc nfs_common nfslock
>
> ...however i checked and tinderbox *does* ignore src.conf and make.conf. the
> _.* log files show that modules were being build which are not returned by
> the commands above.
>
> i think having this flag would be very useful, because it would force people 
> to
> make sure their include dirs are correct.
>
>>
>> Sorry. Second invocation should have been make.conf, not src.conf.

Ok then. I stand corrected for not having tested out my claim beforehand.

Yes, I agree that adding this flag in the default set is a good idea.

Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding -Wmissing-include-dirs to CWARNFLAGS

[Garrett Cooper]

On Thu, Apr 7, 2011 at 1:53 PM, Garrett Cooper  wrote:
> On Thu, Apr 7, 2011 at 11:55 AM, Alexander Best  wrote:
>> hi there,
>>
>> i'd like to propose adding -Wmissing-include-dirs to CWARNFLAGS. this will 
>> let
>> tinderbox fail, if any new kernel code was committed with (a) broken include
>> dir(s).
>>
>> i ran a test via
>>
>> make toolchains
>> make MAKE_JUST_KERNELS=yes tinderbox
>>
>> and nothing seemed to go wrong with the extra warning enabled. i even found a
>> missing include dir, which should be fixed by the attached patch.
>>
>> opinions?
>>
>> so far i've only tested this with gcc. i think someone on #freebsd-clang told
>> me that -Wmissing-include-dirs is a noop for clang (for whatever reasons).
>
>    make -f /etc/src.conf -VMODULES_OVERRIDE and make -f /etc/src.conf
> -VMODULES_OVERRIDE say... (tinderbox should also really ignore these
> files, but it doesn't currently)?

Sorry. Second invocation should have been make.conf, not src.conf.
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: [RFC] adding -Wmissing-include-dirs to CWARNFLAGS

[Garrett Cooper]

On Thu, Apr 7, 2011 at 11:55 AM, Alexander Best  wrote:
> hi there,
>
> i'd like to propose adding -Wmissing-include-dirs to CWARNFLAGS. this will let
> tinderbox fail, if any new kernel code was committed with (a) broken include
> dir(s).
>
> i ran a test via
>
> make toolchains
> make MAKE_JUST_KERNELS=yes tinderbox
>
> and nothing seemed to go wrong with the extra warning enabled. i even found a
> missing include dir, which should be fixed by the attached patch.
>
> opinions?
>
> so far i've only tested this with gcc. i think someone on #freebsd-clang told
> me that -Wmissing-include-dirs is a noop for clang (for whatever reasons).

make -f /etc/src.conf -VMODULES_OVERRIDE and make -f /etc/src.conf
-VMODULES_OVERRIDE say... (tinderbox should also really ignore these
files, but it doesn't currently)?
Thanks,
-Garrett
___
freebsd-toolchain@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"