[CFT/RFC] Make crunched compatible with external linkers
Hello hackers, The following PR patches crunchide(1) to accept object files produced by the gold and mclinker linkers: http://www.freebsd.org/cgi/query-pr.cgi?pr=bin%2F174011 On behalf of the submitter, I'd like to request a review of the patch, and testing of crunchide/crunchgen especially on SPARC and ARM. Thanks, Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: pgbench performance is lagging compared to Linux and DragonflyBSD?
Den 07/11/2012 kl. 15.35 skrev Nikolay Denev nde...@gmail.com: Actually MAXBSIZE is 64k, MAXPHYS is 128k. There was a thread about NFS performance where it was mentioned that bigger MAXBSIZE leads to KVA fragmentation. That thread starts here: http://lists.freebsd.org/pipermail/freebsd-arch/2010-April/010143.html Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: opensolaris B_TRUE and B_FALSE
Hello Ian, Den 29/10/2012 kl. 00.24 skrev Ian Lepore free...@damnhippie.dyndns.org: Look further up in sys/cddl/compat/opensolaris/sys/types.h, they're also defined (as macros rather than enum) in the KERNEL case. They're also defined (as enum) in sys/gnu/fs/xfs/xfs_types.h. (Once again, SlickEdit pays for itself by answering with one right-click a question that would have been a pain to use grep for.) The code in the report is /sbin/zpool, so I assume it's not KERNEL code. As I wrote in my email, I can see B_TRUE and B_FALSE are defined as boolean_t in sys/cddl/compat/opensolaris/sys/types.h But I can't see that boolean_t is defined anywhere in the included headers as long as KERNEL is not defined. As far as I can see, sys/gnu/fs/xfs/xfs_types.h isn't referenced directly or indirectly in any of the headers included in zpool_main.c. Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: opensolaris B_TRUE and B_FALSE
Den 29/10/2012 kl. 11.30 skrev Dimitry Andric d...@freebsd.org:
On 2012-10-29 09:12, Erik Cederstrand wrote:
The code in the report is /sbin/zpool, so I assume it's not KERNEL code. As
I wrote in my email, I can see B_TRUE and B_FALSE are defined as boolean_t
in sys/cddl/compat/opensolaris/sys/types.h But I can't see that boolean_t is
defined anywhere in the included headers as long as KERNEL is not defined.
In sys/cddl/compat/opensolaris/sys/types.h, there is:
typedef enum { B_FALSE, B_TRUE }boolean_t;
This line defines the boolean_t type. Maybe the type itself is never
used, but only the enum values. Sort of like a an anonymous enum in C++.
Ok, so I expected B_FALSE to be defined as 0 explicitly somewhere, completely
missing the point of enums. Embarrassing. Thanks for being easy on me :-)
Erik
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
opensolaris B_TRUE and B_FALSE
Hello, I'm looking at this Clang analyzer report: http://scan.freebsd.your.org/freebsd-head/WORLD/2012-10-24-amd64/report-uH6BjZ.html.gz#EndPath Apart from the actual error, which is a apse positive, it seems like Clang can't find the macro definitions for B_TRUE and B_FALSE (if it did, hovering over them would show the macro definition). These are defined in sys/cddl/compat/opensolaris/sys/types.h as an enum of type boolean_t as long as _KERNEL is not defined. The only definition for boolean_t I can find is in sys/sys/types.h but it's only defined if _KERNEL is defined. I'm sure that ZFS wouldn't work if B_TRUE or B_FALSE were undefined, I just can't figure out where it's happening. I need a hint :-) Thanks, Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: FreeBSD in Google Code-In 2012? You can help too!
Den 16/10/2012 kl. 12.19 skrev Wojciech A. Koszek wkos...@freebsd.org: (cross-posted message; please keep discussion on freebsd-hackers@) Those of you who have Wiki access, please spent 2 more minutes and submit straight to Wiki: http://wiki.freebsd.org/GoogleCodeIn/2012Tasks There are lots of smallish tasks in the code-quality department: * Analyze and fix Clang Static Analyzer warnings * Analyze and fix compiler warnings to increase WARNS level * Write regression tests for src/tools/regression * Run include-what-you-use to clean up header inclusion * Verify bugs with patches I think they're too open-ended to enter in the wiki as-is, but I'd also like to not spam the wiki with lots of almost-identical tasks. What's the best way to suggest them for CodeIn? Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
curcpu false positive?
Hello, I'm looking at some Clang Static Analyzer reports in the kernel, and a lot of them point back to a null pointer dereference in __pcpu_type (sys/amd64/include/pcpu.h:102) which is defined as: 102 /* 103 * Evaluates to the type of the per-cpu variable name. 104 */ 105 #define __pcpu_type(name) \ 106 __typeof(((struct pcpu *)0)-name) Which indeed looks like a NULL pointer dereference. Looking at the latest commit message there, I'm sure the code is correct, but I'm unsure why the null pointer is OK. I'd appreciate an explanation :-) Thanks, Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
time_t when used as timedelta
Hi list, I'm looking at this possible divide-by zero in dhclient: http://scan.freebsd.your.org/freebsd-head/WORLD/2012-10-07-amd64/report-nBhqE2.html.gz#EndPath In this specific case, it's obvious from the intention of the code that ip-client-interval is always 0, but it's not obvious to me in the code. I could add an assert before the possible divide-by-zero: assert(ip-client-interval 0); But looking at the code, I'm not sure it's very elegant. ip-client-interval is defined as time_t (see src/sbin/dhclient/dhcpd.h), which is a signed integer type, if I'm correct. However, some time_t members of struct client_state and struct client_config (see said header file) are assumed in the code to be positive and possibly non-null. Instead of plastering the code with asserts, is there something like an utime_t type? Or are there better ways to enforce the invariant? Thanks, Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Allocator sizeof operand mismatch
Clang Analyzer reports about 100 cases of Allocator sizeof operand mismatch, for example: http://scan.freebsd.your.org/freebsd-head/sbin.umount/2012-09-23-amd64/report-k4ThD9.html#EndPath The reports seem to be valid, but I'm no export. I can work out that the above should probably be fixed by changing sizeof(int) to sizeof(char) and that it incidentally works anyway since both types have the same size in most cases. But I probably won't detect if the error is somewhere else than the arguments to malloc/calloc. Is someone willing to help me fix these, or should I start filing PR's with patches? Thanks, Erik___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
NDEBUG and assert()
Hello, I'm sifting through the Clang Analyzer reports and decided to take a look at this one: http://scan.freebsd.your.org/freebsd-head/lib.ncurses.menu/2012-09-16-amd64/report-3vc5Zu.html#Path5 If you mouseover the assert() in line 236, the problem is that assert() resolves to (void)0 so the analyzer doesn't know that item is non-null later on. Since this is contrib code, my best bet to fix it is to enable assert()'s in our build code for ncurses. That way, users will get an assertion error describing the real problem, instead of a null pointer dereference or garbage data. My assumption here is that nurses is not performance-sensitive so an extra assert() here and there is OK. lib/ncurses/config.mk has CFLAGS+= -DNDEBUG. If I remove this, contrib/ncurses/include/MKncurses_def.sh just makes sure NDEBUG is set to 0 instead of 1. However, include/assert.h just tests for ifdef NDEBUG to decide whether to actually assert or just return (void)0, so it also returns (void)0 when NDEBUG=0. This was obviously not what the ncurses author intended. By changing ifdef NDEBUG to if NDEBUG the value of NDEBUG is evaluated to true or false. The below below patch will let the analyzer reason correctly about the code, and removes the report mentioned above (and a handful others in ncurses). It doesn't touch contrib code, but I'm not happy about changing include/assert.h since it's used so many other places. Any other ideas for how to best solve this? Kind regards, Erik Cederstrand Index: lib/ncurses/ncurses/ncurses_cfg.h === --- lib/ncurses/ncurses/ncurses_cfg.h (revision 240638) +++ lib/ncurses/ncurses/ncurses_cfg.h (working copy) @@ -145,7 +145,7 @@ #define NCURSES_NO_PADDING 1 #define NCURSES_PATHSEP ':' #define NCURSES_VERSION_STRING 5.7.20081102 -#define NDEBUG 1 +#define NDEBUG 0 #define RETSIGTYPE void #define SIG_ATOMIC_T volatile sig_atomic_t #define SIZEOF_SIGNED_CHAR 1 Index: lib/ncurses/config.mk === --- lib/ncurses/config.mk (revision 240638) +++ lib/ncurses/config.mk (working copy) @@ -27,8 +27,6 @@ CFLAGS+= -Wall -CFLAGS+= -DNDEBUG - CFLAGS+= -DHAVE_CONFIG_H # everyone needs this Index: include/assert.h === --- include/assert.h(revision 240638) +++ include/assert.h(working copy) @@ -45,7 +45,7 @@ #undef assert #undef _assert -#ifdef NDEBUG +#if NDEBUG #defineassert(e) ((void)0) #define_assert(e) ((void)0) #else ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NDEBUG and assert()
Den 19/09/2012 kl. 11.19 skrev Erik Cederstrand e...@cederstrand.dk: The below below patch will let the analyzer reason correctly about the code, and removes the report mentioned above (and a handful others in ncurses). It doesn't touch contrib code, but I'm not happy about changing include/assert.h since it's used so many other places. Any other ideas for how to best solve this? An alternative that doesn't touch assert.h but contains a patch to /contrib: Erik Index: lib/ncurses/ncurses/ncurses_cfg.h === --- lib/ncurses/ncurses/ncurses_cfg.h (revision 240638) +++ lib/ncurses/ncurses/ncurses_cfg.h (working copy) @@ -145,7 +145,6 @@ #define NCURSES_NO_PADDING 1 #define NCURSES_PATHSEP ':' #define NCURSES_VERSION_STRING 5.7.20081102 -#define NDEBUG 1 #define RETSIGTYPE void #define SIG_ATOMIC_T volatile sig_atomic_t #define SIZEOF_SIGNED_CHAR 1 Index: lib/ncurses/config.mk === --- lib/ncurses/config.mk (revision 240638) +++ lib/ncurses/config.mk (working copy) @@ -27,8 +27,6 @@ CFLAGS+= -Wall -CFLAGS+= -DNDEBUG - CFLAGS+= -DHAVE_CONFIG_H # everyone needs this Index: contrib/ncurses/include/ncurses_defs === --- contrib/ncurses/include/ncurses_defs(revision 240638) +++ contrib/ncurses/include/ncurses_defs(working copy) @@ -171,7 +171,6 @@ NCURSES_EXT_FUNCS NCURSES_NO_PADDING NCURSES_PATHSEP':' -NDEBUG NEED_PTEM_H NO_LEAKS PURE_TERMINFO ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Change vfork() to posix_spawn()?
Hello hackers, I'm looking through the Clang Analyzer scans on http://scan.freebsd.your.org/freebsd-head looking for false positives to report back to LLVM. There are quite a list of reports suggesting to change vfork() calls to posix_spawn(). Example from /bin/rpc: http://scan.freebsd.your.org/freebsd-head/bin.rcp/2012-09-12-amd64/report-nsOV80.html#EndPath I know nothing about this but I can see fork and posix_spawn have been discussed on this list previously. Is this a legitimate warning (in this case and in general in FreeBSD base)? Thanks, Erik___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Change vfork() to posix_spawn()?
Den 14/09/2012 kl. 13.03 skrev Ivan Voras ivo...@freebsd.org: On 14/09/2012 09:49, Erik Cederstrand wrote: Hello hackers, I'm looking through the Clang Analyzer scans on http://scan.freebsd.your.org/freebsd-head looking for false positives to report back to LLVM. There are quite a list of reports suggesting to change vfork() calls to posix_spawn(). Example from /bin/rpc: http://scan.freebsd.your.org/freebsd-head/bin.rcp/2012-09-12-amd64/report-nsOV80.html#EndPath I know nothing about this but I can see fork and posix_spawn have been discussed on this list previously. Is this a legitimate warning (in this case and in general in FreeBSD base)? Currently (on 9-stable at least), posix_spawn() is implemented as a wrapper around vfork(), so I doubt replacing one with the other would do much. The analyzer added this warning in January. The release notes link to this explanation: https://www.securecoding.cert.org/confluence/display/seccode/POS33-C.+Do+not+use+vfork() I guess this is the important part: Because of the implementation of the vfork() function, the parent process is suspended while the child process executes. If a user sends a signal to the child process, delaying its execution, the parent process (which is privileged) is also blocked. This means that an unprivileged process can cause a privileged process to halt, which is a privilege inversion resulting in a denial of service. Erik___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: How to do benchmark test for my new kernel.
Den 03/09/2012 kl. 09.25 skrev Junior White efi...@gmail.com: hi all, I build a new kernel and install it, but don't known how to test the my new kernel's performance. I have read the Regressin and Performance Testing Guide in developer's handbook. But where is the test program is, and how do i invoke them? The handbook is deliberately vague on this point because choosing a benchmark program requires you to think about what you want to optimize, i.e. which type of workload your system will be handling. If you mostly want to play around and try out different optimizations, I would suggest to install some of the benchmark programs in /usr/ports/benchmarks/. Try /usr/ports/benchmarks/unixbench/ for a set of micro-benchmarks. Kind regards, Erik___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Performance mesurment tools
Hi Ali, Den 17/04/2012 kl. 12.35 skrev ali mousa: Hi all I'm working on a new tool that may increase Operating System performance. I'm looking for a tool that can measure OS performance especially FreeBSD, so I can compare after and before applying the patch. Take a look at the benchmark tools in /usr/ports/benchmarks/ and find one that measures the area you are trying to improve. You might want to consult http://wiki.freebsd.org/BenchmarkAdvice before you run any measurements. Thanks, Erik___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: 14 month old regression (how?)
Den 18/01/2012 kl. 03.21 skrev Devin Teske: Looking at bin/164192... I'm left wondering to myself... How on Earth did a regression-by-typo introduced in SVN r214735 go 14 months without being noticed? Because the regression tests in FreeBSD don't cover this part of the code? :-) Erik___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Deterministic builds, part 2
Hi Ryan, Den 02/12/2010 kl. 05.01 skrev Ryan Stone: asn1.c uses the assert macro, while I believe uses __FILE__. Thanks for the help! asn1.c does indeed import src/include/assert.h, which optionally uses __FILE__ if NDEBUG is not defined. I've tried adding -DNDEBUG to my CFLAGS, but apart from a build error I'm investigating now, this doesn't help with all cases. __FILE__ is used liberally in at least src/sys/sys/. I wonder if I could hack __FILE__ to be a path relative to src/. That would be a way to fix all the source file paths I see. Erik
Re: Deterministic builds, part 2
Hi Joerg, Den 02/12/2010 kl. 13.49 skrev Joerg Sonnenberger: On Thu, Dec 02, 2010 at 11:08:09AM +0100, Erik Cederstrand wrote: I wonder if I could hack __FILE__ to be a path relative to src/. That would be a way to fix all the source file paths I see. I have a patch for that in NetBSD's gcc. Fantastic! Would you mind pointing me to the patch? Thanks, Erik
Re: Deterministic builds, part 2
Den 25/11/2010 kl. 13.08 skrev Erik Cederstrand: rodata.str1.4: -- Some *.o files (all?) contain the path to the corresponding source file: Contents of section .rodata.str1.4: 2f757372 2f686f6d 652f6572 696b2f66 /usr/home/erik/f 0010 72656562 73642f68 6561642f 7372632f reebsd/head1/src/ 0020 6c69622f 6c696273 74616e64 2f6f7065 lib/libstand/ope 0030 6e2e6300 n.c. I've got the deterministic build working with different OBJDIRs now, and I'm trying different SRCDIR locations. This means I have to remove the above file path from the .rodata.str1.4 section. How did it get in there, and how do I remove it again? Thanks, Erik
Re: Deterministic builds, part 2
Den 02/12/2010 kl. 03.44 skrev Ryan Stone: Does the C file use the __FILE__ macro? There are a bit over 1000 files with checksum mismatches, but they seem to differ in the same way. I've picked an example: /usr/lib/libbsnmp.a It contains the string /usr/home/erik/freebsd/head1/src/lib/libbsnmp/libbsnmp/../../../contrib/bsnmp/lib/asn1.c in the section .rodata.str1.8 of the archive member asn1.o. asn1.o was created during buildworld with this command: cc -O2 -pipe -DSTRIP_FBSDID -I/usr/home/erik/freebsd/head1/src/lib/libbsnmp/libbsnmp/../../../contrib/bsnmp/lib -DHAVE_ERR_H -DHAVE_GETADDRINFO -DHAVE_STRLCPY -DHAVE_STDINT_H -DHAVE_INTTYPES_H -DQUADFMT='llu' -DQUADXFMT='llx' -std=gnu99 -fstack-protector -Wsystem-headers -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -c /usr/home/erik/freebsd/head1/src/lib/libbsnmp/libbsnmp/../../../contrib/bsnmp/lib/asn1.c I don't see __FILE__ anywhere in asn1.c or in dependent files. Erik
Deterministic builds, part 2
Hello hackers, With a simple version of deterministic builds done (see my previous post here - anyone willing to comment the patch?), I have started to look at the more general case where OBJDIR and SRCDIR change between builds. The following are my findings. Kernel modules: -- In the ELF section .gnu-debuglink, there is a link to the corresponding *.ko.symbols file. It seems to be an inode or such rather than a file path since nothing shows up in strings(1). I have commented out makeoptions DEBUG=-g in the GENERIC kernel conf file which I am testing now, but I'd like to know what is actually going on. Symbol tables: For example, libstand.a shows up in a diff. Looking with objdump, I see the contained _setjmp.o file has the following symbol table: SYMBOL TABLE: ldf *ABS* /usr/home/erik/freebsd/head1/src/lib/libstand/i386/_setjmp.S ldf *ABS* ./machine/asm.h ldf *ABS* /home/erik/freebsd/obj1/usr/home/erik/freebsd/head1/src/tmp/usr/include/sys/cdefs.h ldf *ABS* ./machine/asm.h ldf *ABS* /usr/home/erik/freebsd/head1/src/lib/libstand/i386/_setjmp.S ldf *ABS* command-line ldf *ABS* built-in ldf *ABS* /usr/home/erik/freebsd/head1/src/lib/libstand/i386/_setjmp.S ld .text ld .data ld .bss g F .text 001d _setjmp 0020 g F .text 0024 _longjmp There are absolute paths within SRCDIR and OBJDIR. I'm pretty sure libarchive.a will still work at runtime if I blow away those directories. I could really use some help changing the paths to be relative to SRJ/OBJDIR if that makes sense, or even removing them if that's better. rodata.str1.4: -- Some *.o files (all?) contain the path to the corresponding source file: Contents of section .rodata.str1.4: 2f757372 2f686f6d 652f6572 696b2f66 /usr/home/erik/f 0010 72656562 73642f68 6561642f 7372632f reebsd/head1/src/ 0020 6c69622f 6c696273 74616e64 2f6f7065 lib/libstand/ope 0030 6e2e6300 n.c. As with the symbol tables, is it possible to change these to relative paths, or just remove them? If there are no flags to gcc or other tricks to do this, I've thought about calling strip(1) within the build process to remove the things I don't want. Thanks, Erik
Re: Deterministic builds, part 2
Den 25/11/2010 kl. 13.08 skrev Erik Cederstrand: Symbol tables: For example, libstand.a shows up in a diff. Looking with objdump, I see the contained _setjmp.o file has the following symbol table: SYMBOL TABLE: ldf *ABS* /usr/home/erik/freebsd/head1/src/lib/libstand/i386/_setjmp.S ldf *ABS* ./machine/asm.h ldf *ABS* /home/erik/freebsd/obj1/usr/home/erik/freebsd/head1/src/tmp/usr/include/sys/cdefs.h ldf *ABS* ./machine/asm.h ldf *ABS* /usr/home/erik/freebsd/head1/src/lib/libstand/i386/_setjmp.S ldf *ABS* command-line ldf *ABS* built-in ldf *ABS* /usr/home/erik/freebsd/head1/src/lib/libstand/i386/_setjmp.S ld .text ld .data ld .bss g F .text 001d _setjmp 0020 g F .text 0024 _longjmp There are absolute paths within SRCDIR and OBJDIR. I'm pretty sure libarchive.a will still work at runtime if I blow away those directories. I could really use some help changing the paths to be relative to SRJ/OBJDIR if that makes sense, or even removing them if that's better. I have an example of this in _umtx_op_err.po contained in libpthread_p.a. The object file is created with the following command: cc -DPROF -O2 -pipe -DPTHREAD_KERNEL -I/usr/home/erik/freebsd/head/src/lib/libthr/../libc/include -I/usr/home/erik/freebsd/head/src/lib/libthr/thread -I/usr/home/erik/freebsd/head/src/lib/libthr/../../include -I/usr/home/erik/freebsd/head/src/lib/libthr/arch/amd64/include -I/usr/home/erik/freebsd/head/src/lib/libthr/sys -I/usr/home/erik/freebsd/head/src/lib/libthr/../../libexec/rtld-elf -I/usr/home/erik/freebsd/head/src/lib/libthr/../../libexec/rtld-elf/amd64 -I/usr/home/erik/freebsd/head/src/lib/libthr/../libthread_db -Winline -fexceptions -D_PTHREAD_FORCED_UNWIND -D_PTHREADS_INVARIANTS -DSYSCALL_COMPAT -std=gnu99 -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -c /usr/home/erik/freebsd/head/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S -o _umtx_op_err.po This command leaves absolute paths in the symbol table. Erik
Re: Deterministic builds, part 2
Den 25/11/2010 kl. 20.17 skrev Mark Johnston: On Thu, Nov 25, 2010 at 01:08:58PM +0100, Erik Cederstrand wrote: Kernel modules: -- In the ELF section .gnu-debuglink, there is a link to the corresponding *.ko.symbols file. It seems to be an inode or such rather than a file path since nothing shows up in strings(1). I have commented out makeoptions DEBUG=-g in the GENERIC kernel conf file which I am testing now, but I'd like to know what is actually going on. The .gnu_debuglink segment contains the name of the debug symbols file (i.e. not a full path or an inode number). I'm not sure why it doesn't show up with strings(1), but you can see it with a hex editor. When gdb loads an object file with a gnu_debuglink segment, it looks in a few pre-defined locations for the corresponding symbols file. The gdb docs explain it pretty well: http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html Thanks. I was explained in private email that I should be using strings -a which indeed turns up the filename. The checksum mismatch actually occurs because the section also contains a CRC of the symbols file, which means the symbols files aren't identical. I'll investigate that separately. Erik
Re: Deterministic builds?
Den 22/10/2010 kl. 12.01 skrev Ulrich Spörlein: Why do you make this a requirement? Of course it's usually easier to build different releases from different source directories, but I think requiring the following conditions are fine: 1. If you build a specific svn revision, 2. sitting in /usr/src with 3. the default make.conf (ie., no special flags, no frobbing of OBJDIR) 4. at different times then you get the same binaries. Let's start with an achievable, not-so-intrusive goal, right? Attached is a patch which addresses the simple case where OBJDIR, SRCDIR and SVN revision are constant, and DESTDIR and build time are different across builds. It does the following: * Patches ranlib to produce generic symbol table (-D option, patch also sent to Kai Wang) * Patches sendmail config scripts to output generic headers for config files * Patches a swath of Makefiles in /contrib that override $ARFLAGS * Removes debugging flag from bthidd Makefile * Adds -D to ARFLAGS and RANLIB * Adds -frandom-seed to CXXflags All the above are only activated if WITH_DETERMINISTIC=true is passed to make: make WITH_DETERMINISTIC=true buildworld/kernel If I could get sendmail config scripts to see src.conf, then it would be possible to place the flag in src.conf. Normal make buildworld should be unaffected by the patch. I have attempted to keep the diff as small as possible. Next goal is making buildworld immune to changing OBJDIRs. Erik deterministic.patch Description: Binary data
Re: Deterministic builds?
Den 15/11/2010 kl. 12.40 skrev Tom Evans: The important things for us are that given a binary, you should be able to easily reproduce the source environment that the binary was produced from, and any two binaries produced from the same sources should be identical. I'm leaning towards not even recording the svn rev. within the binary. A commit only changing comments or style(9) would not change the bits of the binary, but the differing revision number would. A solution could be to have an external file, e.g. /etc/kernel-buildinfo and /etc/world-buildinfo, containing the output of svn stat, svn diff, src.conf, make.conf, SRCDIR and OBJDIR locations, the full buildworld/kernel command and whatever else could affect the build outcome. Erik
Re: Deterministic builds?
Den 12/11/2010 kl. 21.20 skrev Giorgos Keramidas: Since the SVN rev. is recorded, I think a timestamp is redundant. Any ideas where I can disable the timestamps in the source? The timestamp is not 'redundant'. It records _when_ you compiled the sources of the kernel, which in itself is a useful bit of information. I'm curious as to why this might be useful? Would the mtime of the file not be be sufficient? I can only think of debugging purposes, but apart from the timestamp, two kernels with the same rev. would be bitwise identical, so I think the rev. number is more useful. Is the timestamp not just a remnant from the CVS days? If it is useful, why not brand all binaries with a rev. number and a timestamp? Erik
Re: Deterministic builds?
Den 14/11/2010 kl. 21.32 skrev Dimitry Andric: On 2010-11-14 21:22, Erik Cederstrand wrote: I'm curious as to why this might be useful? Would the mtime of the file not be be sufficient? I can only think of debugging purposes, but apart from the timestamp, two kernels with the same rev. would be bitwise identical, This does not have to be the case. For example, if you have have local modifications, or use different settings in make.conf or src.conf. In this case the timestamp + rev. is not sufficient to reproduce the kernel anyway. You'd need to store externally the non-standard contents of conf files, local diffs etc. on all your non-standard builds. You could do all sorts of fun stuff, even fool the rev. number or timestamp if you wanted. I'm just saying that for the standard user on a standard GENERIC kernel (and world for that matter) - the revision number should be sufficient for e.g. filing a PR. If you need the timestamp, there's the mtime. Erik
Re: Deterministic builds?
Den 22/10/2010 kl. 12.01 skrev Ulrich Spörlein: Why do you make this a requirement? Of course it's usually easier to build different releases from different source directories, but I think requiring the following conditions are fine: 1. If you build a specific svn revision, 2. sitting in /usr/src with 3. the default make.conf (ie., no special flags, no frobbing of OBJDIR) 4. at different times then you get the same binaries. Let's start with an achievable, not-so-intrusive goal, right? :) Ok, here's a new attempt with SRCDIR and OBJDIR constant between the two builds. This time, /boot/kernel/kernel, /boot/loader, /boot/pxeboot and /boot/zfsloader differ. According to strings(1), the only difference is the timestamp. E.g. the kernel: @(#)FreeBSD 9.0-CURRENT #0 r215143: Thu Nov 11 22:58:34 CET 2010 FreeBSD 9.0-CURRENT #0 r215143: Thu Nov 11 22:58:34 CET 2010 --- @(#)FreeBSD 9.0-CURRENT #0 r215143: Thu Nov 11 23:29:17 CET 2010 FreeBSD 9.0-CURRENT #0 r215143: Thu Nov 11 23:29:17 CET 2010 Since the SVN rev. is recorded, I think a timestamp is redundant. Any ideas where I can disable the timestamps in the source? Also, /usr/bin/[clang|clang++|tblgen] differ, i.e.: 248735,248736c248735,248736 N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp__E8B12D4D15PrintModulePassE N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp__E8B12D4D17PrintFunctionPassE --- N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp__BDCFB9C615PrintModulePassE N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp__BDCFB9C617PrintFunctionPassE I'm not sure what to do with this except pass it on to the LLVM lists. Erik
Re: Deterministic builds?
Den 21/10/2010 kl. 19.57 skrev Ulrich Spörlein:
On Mon, 11.10.2010 at 11:35:42 +0200, Erik Cederstrand wrote:
I'm beginning to think that it should at least be optional. Removing e.g.
build times, mtimes and path to OBJDIR or SRCDIR might not make everyone
happy.
The problem with making it optional is, that we already have enough
flags and knobs. No need in adding more.
Besides, why would people want to know the date of the build? Far more
important is the date and state of the source for the time of build. So
you might want to replace ${BUILDATE} with ${SRCDATE} somehow (time of
last commit?).
Otherwise, please go for it. It would be nice if two people compiling
GENERIC for the same source-base would get identical binaries.
It goes without saying that I agree with you on this. But it seems the feature
will require fairly invasive changes to a standard FreeBSD, e.g. changing
standard ar behavior, and building kernels and some other tools without
debugging symbols. I'm not experienced enough to determine if this is fine with
the majority of users, but hiding the changes under a knob would at least let
the feature prove itself and put off bikeshedding for a while. If the project
works out, we can discuss changing the default.
I'm still not sure what to do with debugging symbols. Currently absolute paths
to source files are used. I would like to change that to absolute paths , i.e.
usr.bin/ar/ar.c. That might even be beneficial if someone is debugging the
binary on a system that doesn't have the source tree located in
/some/obscure/directory/src. It's my understanding that gdb uses the path to
look up source code related to stack traces etc. It seems gdb can handle
relative paths, but I have no idea if it actually works, or if it's a nuisance
to use compared to absolute paths. Any hints?
Thanks,
Erik
Re: Deterministic builds?
Den 11/10/2010 kl. 10.47 skrev Kostik Belousov: My personal opinion that the feature is nice to have. Unless the changes to get this working are too large, and, more importantly, unless the maintenance cost of having this in good shape is too high, sure we would better have deterministic build results. Also, the deterministic builds require somebody who would monitor the feature, either manually, or by setting some bot that automatically checks it. Otherwise, I suspect, it will degrade. I might want to adopt the task of monitoring the feature. I'm beginning to think that it should at least be optional. Removing e.g. build times, mtimes and path to OBJDIR or SRCDIR might not make everyone happy. Any hints to why kernel module checksums don't match? Thanks, Erik
Re: Timestamps in static libraries
Den 06/10/2010 kl. 14.35 skrev Erik Cederstrand:
Den 06/10/2010 kl. 13.07 skrev Erik Cederstrand:
Is something like the following acceptable? Without risking changes to
buildworld/distribution just now, this would allow me to dump contents of an
archive and re-insert them with '0' for mtime, uid and gid before checking
checksums, without affecting normal ar behaviour.
Great, I didn't even see the discussion on this list recently:
http://lists.freebsd.org/pipermail/freebsd-hackers/2010-September/033005.html
Anyway, I added file mode to the patch. Apparently recent binutils also added
this feature. I haven't looked at their patch, but chance has it I used the
same option '-D'.
I'm sure the file mode I'm setting in line 175 of write.c is wrong (obj-md =
100644), but I couldn't quite figure out how to set the value to rw-r--r--.
Here's the new patch:
Index: ar.1
===
--- ar.1 (revision 213478)
+++ ar.1 (working copy)
@@ -62,6 +62,7 @@
.Op Fl a Ar position-after
.Op Fl b Ar position-before
.Op Fl c
+.Op Fl D
.Op Fl i Ar position-before
.Op Fl j
.Op Fl s
@@ -179,6 +180,14 @@
.Ar archive .
The archive's symbol table, if present, is updated to reflect
the new contents of the archive.
+.It Fl D
+When used in combination with the
+.Fl r
+option, insterts 0's instead of the real mtime, uid and gid values
+and 644 instead of file mode from the members named by arguments
+.Ar files ... .
+This ensures that checksums on the resulting archives are reproducible
+when member contents are identical.
.It Fl f
Synonymous with option
.Fl T .
Index: ar.c
===
--- ar.c (revision 213478)
+++ ar.c (working copy)
@@ -154,7 +154,7 @@
}
}
- while ((opt = getopt_long(argc, argv, abCcdfijlMmopqrSsTtuVvxz,
+ while ((opt = getopt_long(argc, argv, abCcdDfijlMmopqrSsTtuVvxz,
longopts, NULL)) != -1) {
switch(opt) {
case 'a':
@@ -173,6 +173,9 @@
case 'd':
set_mode(bsdar, opt);
break;
+ case 'D':
+ bsdar-options |= AR_D;
+ break;
case 'f':
case 'T':
bsdar-options |= AR_TR;
@@ -357,8 +360,8 @@
(void)fprintf(stderr, \tar -m [-Tabijsvz] position archive file
...\n);
(void)fprintf(stderr, \tar -p [-Tv] archive [file ...]\n);
(void)fprintf(stderr, \tar -q [-Tcjsvz] archive file ...\n);
- (void)fprintf(stderr, \tar -r [-Tcjsuvz] archive file ...\n);
- (void)fprintf(stderr, \tar -r [-Tabcijsuvz] position archive file
...\n);
+ (void)fprintf(stderr, \tar -r [-TcDjsuvz] archive file ...\n);
+ (void)fprintf(stderr, \tar -r [-TabcDijsuvz] position archive file
...\n);
(void)fprintf(stderr, \tar -s [-jz] archive\n);
(void)fprintf(stderr, \tar -t [-Tv] archive [file ...]\n);
(void)fprintf(stderr, \tar -x [-CTouv] archive [file ...]\n);
Index: ar.h
===
--- ar.h (revision 213478)
+++ ar.h (working copy)
@@ -43,6 +43,7 @@
#define AR_U 0x0200 /* only extract or update newer members.*/
#define AR_V 0x0400 /* verbose mode */
#define AR_Z 0x0800 /* gzip compression */
+#define AR_D 0x1000 /* insert members with dummy mode, mtime, uid
and gid */
#define DEF_BLKSZ 10240 /* default block size */
Index: write.c
===
--- write.c (revision 213478)
+++ write.c (working copy)
@@ -163,11 +163,23 @@
if (mtime != 0 bsdar-options AR_U sb.st_mtime = mtime)
goto giveup;
- obj-uid = sb.st_uid;
- obj-gid = sb.st_gid;
- obj-md = sb.st_mode;
+ /*
+ * When option '-D' is specified, mtime and UID / GID from the file
+ * will be replaced with 0, and file mode with 644. This ensures that
+ * checksums will match for two archives containing the exact same
files.
+ */
+ if (bsdar-options AR_D) {
+ obj-uid = 0;
+ obj-gid = 0;
+ obj-mtime = 0;
+ obj-md = 100644;
+ } else {
+ obj-uid = sb.st_uid;
+ obj-gid = sb.st_gid;
+ obj-mtime = sb.st_mtime;
+ obj-md = sb.st_mode;
+ }
obj-size = sb.st_size;
- obj-mtime = sb.st_mtime;
obj-dev = sb.st_dev;
obj-ino = sb.st_ino;
@@ -621,7 +633,10 @@
bsdar-options AR_S) {
entry = archive_entry_new();
archive_entry_copy_pathname(entry, /);
- archive_entry_set_mtime(entry, time(NULL), 0);
+ if (bsdar-options AR_O
Deterministic builds?
Hi hackers As a followup to the Timestamps in static libraries thread which resulted in a '-D' option to ar(1), I'd like to discuss if it is a worthy goal of the Project to create deterministic builds. By that I mean for two make build+install world+kernel+distribution runs, every contained file is bitwise identical between the two runs. Deterministic builds would be useful for me, since I'm creating binary diffs against lots of FreeBSD builds, and smaller diffs are good. Also, I'd like to detect which files have changed between two commits. I imagine it would also be useful for things like IDS and freebsd-update. Currently, this does not hold for static libraries (*.a), kernel modules (*.ko / *.ko.symbols) and the following: bthidd cc1 cc1obj cc1plus clang clang++ ctfconvert freebsd.cf freebsd.submit.cf kernel kernel.symbols libcrypto.so.6 libufs.so.5 loader pxeboot sendmail.cf submit.cf tblgen zfsloader Most of the libraries can be brought to be identical by using ar -D. Some record the absolute OBJDIR path to header files, though (libc.a for example). I tried adding 'D' to ARFLAGS in share/mk/sys.mk, but that's only part of the solution. ARFLAGS are overridden hundreds of places in the source code, and in some places ARFLAGS isn't even used (or AR for that matter). Is it worthwhile to go through the whole tree, fixing up these calls to ar? A lot of this is in contrib/ code. Another option is to add a WITH_DETERMINISTIC_AR knob to the build to compile ar with D as default behaviour. This would make the above changes unnecessary, but is more intrusive. A third option is that this is not a priority for the community, or directly unwanted, and that I just post-process my builds myself. I don't know what causes the checksum difference in .ko files - there is no size difference, and no difference according to strings(1). A bsdiff on the two is typically around 160B. .ko.symbols have some unique identifiers or addresses internally. kernel, loader, zfsloader and pxeboot have a build date recorded, kernel also has absolute path to GENERIC. OK for the kernel, I think, although it would be easier for me if this was just stored in a separate file since binary diffs on large files are expensive. clang, clang++ and tblgen store some absolute paths to .cpp files in the src repo internally, plus unique identifiers. freebsd.cf, freebsd.submit.cf, sendmail.cf and submit.cf record the absolute OBJDIR path to sendmail What do you think? Thanks, Erik
Re: Timestamps in static libraries
Den 06/10/2010 kl. 08.00 skrev Tim Kientzle: That's the timestamp on the pseudo-entry used to store the archive symbol table. (GNU/SysV ar files use a pseudo-entry named / to store the symbol table. Since '/' is added to end of names in this format, this is essentially an entry with an empty name.) The current ar code generates this entry and sets the timestamp around line 624 of usr.bin/ar/write.c. As far as I can tell, ar itself doesn't care about the timestamp here; it's possible that ranlib or ld do care. If they don't, we could set this timestamp to zero always. Yeah, I had a look at the code too. I was thinking maybe it would help to add a modifier to replace timestamps, uids and gids with '0' when inserting a file, to respect POLA. Thanks, Erik
Re: Timestamps in static libraries
Den 06/10/2010 kl. 10.06 skrev per...@pluto.rain.com: Erik Cederstrand e...@cederstrand.dk wrote: It seems I can at least normalize the .a files using something like the following to weed out timestamps and uid/gid: % ar -x /usr/lib/libfetch.a % chown 0:0 * % touch -t 19700101 * % ar -r libfetch.a `ar -t /usr/lib/libfetch.a` ... Unfortunately it seems there's still a creation time of the archive itself that I cant alter using the above, so the md5 sums still don't match: % diff mod.strings orig.strings 2c2 / 1286312209 0 0 0 958 ` --- / 1269146263 0 0 0 958 ` Any particular reason to recollect them into an archive, if the point is just to check md5 signatures? I'm pretty sure collecting them with tar instead will avoid this problem. That's of course another option. I could unpack the archive and be satisfied if all contained files have matching md5's. I guess the perfectionist in me is protesting. If I build FreeBSD twice from the same source code, the result should be exactly the same. It would be useful in a number of cases, e.g. IDS. ccache also relies on the assumption that checksums will match on identical source code and compiler version, although I know that's a level beneath where ar operates. Thanks, Erik
Re: Timestamps in static libraries
Den 06/10/2010 kl. 08.00 skrev Tim Kientzle:
% diff mod.strings orig.strings
2c2
/ 1286312209 0 0 0 958 `
---
/ 1269146263 0 0 0 958 `
That's the timestamp on the pseudo-entry used to store the archive
symbol table. (GNU/SysV ar files use a pseudo-entry named / to
store the symbol table. Since '/' is added to end of names in this
format, this is essentially an entry with an empty name.)
The current ar code generates this entry and sets the timestamp
around line 624 of usr.bin/ar/write.c.
As far as I can tell, ar itself doesn't care about the timestamp here;
it's possible that ranlib or ld do care. If they don't, we could set
this timestamp to zero always.
Is something like the following acceptable? Without risking changes to
buildworld/distribution just now, this would allow me to dump contents of an
archive and re-insert them with '0' for mtime, uid and gid before checking
checksums, without affecting normal ar behaviour.
erik% svn diff
Index: ar.1
===
--- ar.1(revision 213478)
+++ ar.1(working copy)
@@ -62,6 +62,7 @@
.Op Fl a Ar position-after
.Op Fl b Ar position-before
.Op Fl c
+.Op Fl D
.Op Fl i Ar position-before
.Op Fl j
.Op Fl s
@@ -179,6 +180,14 @@
.Ar archive .
The archive's symbol table, if present, is updated to reflect
the new contents of the archive.
+.It Fl D
+When used in combination with the
+.Fl r
+option, insterts 0's instead of the real mtime, uid and gid values
+from the members named by arguments
+.Ar files ... .
+This ensures that checksums on the resulting archives are reproducible
+when member contents are identical.
.It Fl f
Synonymous with option
.Fl T .
Index: ar.c
===
--- ar.c(revision 213478)
+++ ar.c(working copy)
@@ -154,7 +154,7 @@
}
}
- while ((opt = getopt_long(argc, argv, abCcdfijlMmopqrSsTtuVvxz,
+ while ((opt = getopt_long(argc, argv, abCcdDfijlMmopqrSsTtuVvxz,
longopts, NULL)) != -1) {
switch(opt) {
case 'a':
@@ -173,6 +173,9 @@
case 'd':
set_mode(bsdar, opt);
break;
+ case 'D':
+ bsdar-options |= AR_D;
+ break;
case 'f':
case 'T':
bsdar-options |= AR_TR;
@@ -357,8 +360,8 @@
(void)fprintf(stderr, \tar -m [-Tabijsvz] position archive file
...\n);
(void)fprintf(stderr, \tar -p [-Tv] archive [file ...]\n);
(void)fprintf(stderr, \tar -q [-Tcjsvz] archive file ...\n);
- (void)fprintf(stderr, \tar -r [-Tcjsuvz] archive file ...\n);
- (void)fprintf(stderr, \tar -r [-Tabcijsuvz] position archive file
...\n);
+ (void)fprintf(stderr, \tar -r [-TcDjsuvz] archive file ...\n);
+ (void)fprintf(stderr, \tar -r [-TabcDijsuvz] position archive file
...\n);
(void)fprintf(stderr, \tar -s [-jz] archive\n);
(void)fprintf(stderr, \tar -t [-Tv] archive [file ...]\n);
(void)fprintf(stderr, \tar -x [-CTouv] archive [file ...]\n);
Index: ar.h
===
--- ar.h(revision 213478)
+++ ar.h(working copy)
@@ -43,6 +43,7 @@
#define AR_U 0x0200 /* only extract or update newer members.*/
#define AR_V 0x0400 /* verbose mode */
#define AR_Z 0x0800 /* gzip compression */
+#define AR_D 0x1000 /* insert members with dummy mtime, uid and gid
*/
#define DEF_BLKSZ 10240/* default block size */
Index: write.c
===
--- write.c (revision 213478)
+++ write.c (working copy)
@@ -163,11 +163,22 @@
if (mtime != 0 bsdar-options AR_U sb.st_mtime = mtime)
goto giveup;
- obj-uid = sb.st_uid;
- obj-gid = sb.st_gid;
+ /*
+* When modifier -D is specified, mtime and UID / GID from the file
+* will be replaced with 0. This ensures that checksums will match
+* for two archives containing the exact same files.
+ */
+ if (bsdar-options AR_D) {
+ obj-uid = 0;
+ obj-gid = 0;
+ obj-mtime = 0;
+ } else {
+ obj-uid = sb.st_uid;
+ obj-gid = sb.st_gid;
+ obj-mtime = sb.st_mtime;
+ }
obj-md = sb.st_mode;
obj-size = sb.st_size;
- obj-mtime = sb.st_mtime;
obj-dev = sb.st_dev;
obj-ino = sb.st_ino;
@@ -621,7 +632,10 @@
bsdar-options AR_S) {
entry = archive_entry_new();
archive_entry_copy_pathname(entry, /);
- archive_entry_set_mtime(entry, time(NULL), 0);
Re: Timestamps in static libraries
Den 06/10/2010 kl. 13.07 skrev Erik Cederstrand:
Is something like the following acceptable? Without risking changes to
buildworld/distribution just now, this would allow me to dump contents of an
archive and re-insert them with '0' for mtime, uid and gid before checking
checksums, without affecting normal ar behaviour.
Great, I didn't even see the discussion on this list recently:
http://lists.freebsd.org/pipermail/freebsd-hackers/2010-September/033005.html
Anyway, I added file mode to the patch. Apparently recent binutils also added
this feature. I haven't looked at their patch, but chance has it I used the
same option '-D'.
I'm sure the file mode I'm setting in line 175 of write.c is wrong (obj-md =
100644), but I couldn't quite figure out how to set the value to rw-r--r--.
Here's the new patch:
Index: ar.1
===
--- ar.1(revision 213478)
+++ ar.1(working copy)
@@ -62,6 +62,7 @@
.Op Fl a Ar position-after
.Op Fl b Ar position-before
.Op Fl c
+.Op Fl D
.Op Fl i Ar position-before
.Op Fl j
.Op Fl s
@@ -179,6 +180,14 @@
.Ar archive .
The archive's symbol table, if present, is updated to reflect
the new contents of the archive.
+.It Fl D
+When used in combination with the
+.Fl r
+option, insterts 0's instead of the real mtime, uid and gid values
+and 644 instead of file mode from the members named by arguments
+.Ar files ... .
+This ensures that checksums on the resulting archives are reproducible
+when member contents are identical.
.It Fl f
Synonymous with option
.Fl T .
Index: ar.c
===
--- ar.c(revision 213478)
+++ ar.c(working copy)
@@ -154,7 +154,7 @@
}
}
- while ((opt = getopt_long(argc, argv, abCcdfijlMmopqrSsTtuVvxz,
+ while ((opt = getopt_long(argc, argv, abCcdDfijlMmopqrSsTtuVvxz,
longopts, NULL)) != -1) {
switch(opt) {
case 'a':
@@ -173,6 +173,9 @@
case 'd':
set_mode(bsdar, opt);
break;
+ case 'D':
+ bsdar-options |= AR_D;
+ break;
case 'f':
case 'T':
bsdar-options |= AR_TR;
@@ -357,8 +360,8 @@
(void)fprintf(stderr, \tar -m [-Tabijsvz] position archive file
...\n);
(void)fprintf(stderr, \tar -p [-Tv] archive [file ...]\n);
(void)fprintf(stderr, \tar -q [-Tcjsvz] archive file ...\n);
- (void)fprintf(stderr, \tar -r [-Tcjsuvz] archive file ...\n);
- (void)fprintf(stderr, \tar -r [-Tabcijsuvz] position archive file
...\n);
+ (void)fprintf(stderr, \tar -r [-TcDjsuvz] archive file ...\n);
+ (void)fprintf(stderr, \tar -r [-TabcDijsuvz] position archive file
...\n);
(void)fprintf(stderr, \tar -s [-jz] archive\n);
(void)fprintf(stderr, \tar -t [-Tv] archive [file ...]\n);
(void)fprintf(stderr, \tar -x [-CTouv] archive [file ...]\n);
Index: ar.h
===
--- ar.h(revision 213478)
+++ ar.h(working copy)
@@ -43,6 +43,7 @@
#define AR_U 0x0200 /* only extract or update newer members.*/
#define AR_V 0x0400 /* verbose mode */
#define AR_Z 0x0800 /* gzip compression */
+#define AR_D 0x1000 /* insert members with dummy mode, mtime, uid
and gid */
#define DEF_BLKSZ 10240/* default block size */
Index: write.c
===
--- write.c (revision 213478)
+++ write.c (working copy)
@@ -163,11 +163,23 @@
if (mtime != 0 bsdar-options AR_U sb.st_mtime = mtime)
goto giveup;
- obj-uid = sb.st_uid;
- obj-gid = sb.st_gid;
- obj-md = sb.st_mode;
+ /*
+* When option '-D' is specified, mtime and UID / GID from the file
+* will be replaced with 0, and file mode with 644. This ensures that
+* checksums will match for two archives containing the exact same
files.
+ */
+ if (bsdar-options AR_D) {
+ obj-uid = 0;
+ obj-gid = 0;
+ obj-mtime = 0;
+ obj-md = 100644;
+ } else {
+ obj-uid = sb.st_uid;
+ obj-gid = sb.st_gid;
+ obj-mtime = sb.st_mtime;
+ obj-md = sb.st_mode;
+ }
obj-size = sb.st_size;
- obj-mtime = sb.st_mtime;
obj-dev = sb.st_dev;
obj-ino = sb.st_ino;
@@ -621,7 +633,10 @@
bsdar-options AR_S) {
entry = archive_entry_new();
archive_entry_copy_pathname(entry, /);
- archive_entry_set_mtime(entry, time(NULL), 0);
+ if (bsdar-options AR_O)
+ archive_entry_set_mtime(entry
Timestamps in static libraries
Hello hackers, I got reminded of a problem I had a couple of years back compressing FreeBSD jails. I was using bsdiff for the compression and found out that md5 sums of static libraries (.a files) in /usr/lib and /usr/local/lib didn't match between jails, even though the source code used to create the jails hadn't changed. One of my goals is to detect which files in a distribution change between two commits. It turns out that timestamps are stored in the library: # strings DIR1/usr/lib/libfetch.a tmp1 # strings DIR2/usr/lib/libfetch.a tmp2 # diff tmp1 tmp2 2c2 / 1200728973 0 0 0 954 ` --- / 1200723259 0 0 0 954 ` 57c57 file.o/ 1200728973 0 0 100644 2356 ` --- file.o/ 1200723259 0 0 100644 2356 ` 86c86 http.o/ 1200728973 0 0 100644 17180 ` --- http.o/ 1200723258 0 0 100644 17180 ` [...] I'm wondering if this is necessary, or if this can possibly be turned of with a knob somewhere. Thanks, Erik
Re: Timestamps in static libraries
Den 05/10/2010 kl. 15.59 skrev Erik Trulsson: On Tue, Oct 05, 2010 at 03:28:36PM +0200, Erik Cederstrand wrote: Hello hackers, I got reminded of a problem I had a couple of years back compressing FreeBSD jails. I was using bsdiff for the compression and found out that md5 sums of static libraries (.a files) in /usr/lib and /usr/local/lib didn't match between jails, even though the source code used to create the jails hadn't changed. One of my goals is to detect which files in a distribution change between two commits. It turns out that timestamps are stored in the library: Yes, they are. That is because the file format used for static libraries include a timestamp for each object file stored in the archive. You can use 'ar -tv /PATH/TO/LIBRARY/libfoo.a' to get a list of the objects stored in the archive and the corresponding timestamps. See the ar(5) manpage for a description of the file format, and the ar(1) manpage for information on how to manage such files. Thanks, that was very helpful. It seems I can at least normalize the .a files using something like the following to weed out timestamps and uid/gid: % ar -x /usr/lib/libfetch.a % chown 0:0 * % touch -t 19700101 * % ar -r libfetch.a `ar -t /usr/lib/libfetch.a` The above takes ca. 10 seconds for all static libraries on my aging machine, which is OK in my case. Unfortunately it seems there's still a creation time of the archive itself that I cant alter using the above, so the md5 sums still don't match: % diff mod.strings orig.strings 2c2 / 1286312209 0 0 0 958 ` --- / 1269146263 0 0 0 958 ` Using Python to replace 1286312209 with 1269146263 in the binary file now gives me matching md5 sums. It seems a bit complicated just to get rid of some timestamps, though. Thanks, Erik
Re: GSoC: Making ports work with clang
Den 30/05/2010 kl. 14.51 skrev Andrius Morkūnas:
On Sun, 30 May 2010 14:58:05 +0300, Volodymyr Kostyrko c.kw...@gmail.com
wrote:
1. __dso not found after link. Some symbols seems to be omitted from
libraries and linking of plugins fails badly.
Known problem with known fix.
2. Assembler errors. Xorg has some in x11-servers/xorg-server
x11-drivers/xf86-video-vesa x11-drivers/xf86-video-ati, everything else
compiles and works.
Assembler errors often aren't similar to each other, so fixing them may
be very easy or difficult. Hopefully we will fix them for big stuff like
xorg (not really as part of this GSoC project).
3. Big bunch of compile errors or config errors. This means incorrectly
written code, like not correctly declaring variables. This also means
some automake stupidities like testing c++ compiler with c style code -
for example clang++ refuses to compile int main(void) {}.
$ cat main.cc
int main(void) {}
$ clang main.cc -o test ./test echo No, it works.
No, it works.
Other than that, yes, many problems are related to insane configure
scripts.
4. Some ports specify that thay need at least gcc 3.3.
This is another of those insane configure scripts, testing for specific
version of specific compiler, rather than testing if it can compile
anything.
5. Some ports needs --dumpspecs.
It's a bit uglier than some ports:
$ grep dumpspecs /usr/ports/Mk/bsd.gecko.mk
GECKO_PTHREAD_LIBS!=${CC} -dumpspecs | ${GREP} -m 1 pthread: | ${SED} -e
's|^.*%{\!pg: %{pthread:|| ; s|}.*$$||' || ${TRUE}
audio/libmad - distorted sound
lang/python26 - compiling any gir dumps core
textproc/expat2 - dbus dumps core at launch
Python and expat shout i386 in my face, clang/llvm tends to not like
i386 too much. But I think few miscompilations were fixed recently,
so some of these may already be working fine.
And this all data is not current. It's one month old. Since then
dumpspecs was implemented. And maybe some other problems begone - I just
have not enough time to look at this thoroughly.
Some problems from a month ago are definitely gone, but I don't think
dumpspecs is one of them.
Andrius, would it make sense to create e.g. a wiki page tracking the status and
current known problems with compiling ports with clang? Just like there's a
wiki page ClangBSD status.
I think it would make it easier for lurkers to jump in and test things, and
help whittle away at the problems.
Thanks,
Erik
Re: Permissive licensed toolchain
Den 30/05/2010 kl. 16.15 skrev C. Bergström: What's the real status of a fully permissive licensed toolchain? You mean ClangBSD? 1) Benchmarks - (I mean emperical evidence on FBSD and per target with no anecdotal comments or speculation.. I admit benchmarks can actually be misleading since many companies optimize for them specifically) I'm working (slowly) on comparing FreeBSD and ClangBSD for various benchmarks. 3) Which assembler is being used? The same as FreeBSD: GNU as. There is ongoing work in the llvm-mc project to provide an assembler for LLVM, but ELF support is low priority for Apple. There was an experimental patch on the mailing list a couple of weeks back. 4) Which linker is being used? GNU ld. LLVM also provides a linker, but last time I checked it wasn't functional on FreeBSD. It provides LTO that would be interesting to benchmark. What's the best way to make a plan which will get feedback if someone wanted to try alternative approach to the above? Make it easily available, e.g. as a Subversion repo like ClangBSD, or an install CD, and provide instructions for easy installation and test. I'm testing ClangBSD in a VirtualBox VM which makes it easy to recover after hosing the operating system. Erik
Re: Clang: now available from a SVN server near you!
Den 04/06/2009 kl. 18.06 skrev Tim Kientzle: Erik Cederstrand wrote: LLVM provides a linker (http://llvm.org/cmds/llvm-ld.html) but it doesn't interact correctly with conventional nm/ar/etc (http://lists.cs.uiuc.edu/pipermail/cfe-dev/2009-June/005296.html ). In what way does it not interact correctly? There was not much more help to get from the Clang list, unfortunately. The code lives at http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/llvm-ld/ and looks very well-documented and structured. Unfortunately, it is a bit over my head to see what needs to be fixed and actually fix it. Thanks, Erik
Re: Clang: now available from a SVN server near you!
Hi Roman Den 04/06/2009 kl. 14.38 skrev Roman Divacky: you could use llvm-ld (see the wiki for instructions how to do it). there's also some effort to make gnu ld usable with llvm LTO and I guess the patch could be backported to our ld. I guess As I understand the reply from Eli Friedman[1] this is not possible without at least some hacking. The LTO work in GNU ld[2] is under GPLv3[3], as is gold[4], which makes backporting patches a sticky issue. Erik [1] http://lists.cs.uiuc.edu/pipermail/cfe-dev/2009-June/005296.html [2] http://gcc.gnu.org/wiki/LinkTimeOptimization [3] http://gcc.gnu.org/svn/gcc/branches/lto/lto-plugin/lto-plugin.c [4] http://sourceware.org/cgi-bin/cvsweb.cgi/src/gold/gold.cc?rev=1.63content-type=text/x-cvsweb-markupcvsroot=src
Re: Clang: now available from a SVN server near you!
Den 04/06/2009 kl. 11.38 skrev Ed Schouten: You can now build your very own version of FreeBSD with Clang installed as /usr/bin/cc as follows: Thanks for your hard work, Ed. This is great news! You might want to mention that a few parts are still GCC-compiled due to bugs in Clang ( see http://wiki.freebsd.org/ BuildingFreeBSDWithClang). Also, it's very encouraging that the ports run you did with Erwin (http://lists.cs.uiuc.edu/pipermail/cfe-dev/2009-June/005274.html ) compiles over 7000 ports with Clang. I've asked in the Clang list, but I'd like an opinion from FreeBSD folks, too: Clang supports LTO (http://llvm.org/docs/LinkTimeOptimization.html ), which has a potential for performance improvements. It doesn't work on FreeBSD because the linker we have (in binutils) doesn't know about the libLTO that LLVM provides. There's a new linker from GNU called Gold, but as far as I know it's GPLv3 licensed and therefore undesirable at least to have in base. LLVM provides a linker (http://llvm.org/cmds/llvm-ld.html ) but it doesn't interact correctly with conventional nm/ar/etc (http://lists.cs.uiuc.edu/pipermail/cfe-dev/2009-June/005296.html ). There's the ELF toolchain project (elftoolchain.sourceforge.net/) but a BSD-licensed ld hasn't been developed yet. What would be the best way to get LTO to work on FreeBSD? Thanks, Erik
Re: PXE on 7.0 problem and solution
Tim Clewlow wrote: Hi there, Installing 7.0 via PXE has a slight problem that is easily worked around. The file /boot/mfsroot.gz on the installation media needs to be unzipped to make PXE boot via tftp/nfs work. Otherwise the loader ultimately complains that it cant find the device to boot from. For example, if you have the installation media living at /usr/pxe/nfs/ on the PXE server, then do: gzip -d /usr/pxe/nfs/boot/mfsroot.gz After doing this it now loads the kernel and starts the installation procedure as expected. Someone more knowledgeable than me might want to let whoever needs to know about this. Apart from that, it looks great, the work is appreciated, thanks for the new release :-) I've had problems with other files fetched over NFS in the past. In theory (at least that's what a tcpdump says), it should be possible to provide all files gzipped, and even split the files in smaller chunks. Files are tried in this order: boot/loader.rc.split boot/loader.rc.gz.split boot/loader.rc.gz boot/loader.rc However, I only got the plain files to work reliably. Erik ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
