Re: LTO and libelf (and FreeBSD)

2010-05-23 Thread Kai Wang
On Sun, May 23, 2010 at 12:48:20AM +0200, Gerald Pfeifer wrote:
> On Thu, 20 May 2010, Kai Wang wrote:
> > The elf_getbase() API in FreeBSD libelf can only be called using an 
> > archive member ELF descriptor. It will return -1 (indicates an error) 
> > when called with a "regular" ELF object.
> > 
> > The lto_obj_build_section_table() function in lto-elf.c calls
> > elf_getbase() to get the offset base for a "regular" ELF object. On
> > FreeBSD it gets -1.  (Side notes: lto-elf.c should really check return
> > values of libelf APIs) And later it uses -1 as base_offset for all the
> > ELF sections thus results in an off-by-one error when passing the
> > section data to zlib to inflate.
> > 
> > Could you please try applying the attached patch to the libelf in
> > FreeBSD 7.3 and see if it fixes gcc4.6 lto?
> 
> Thanks, Kai!
> 
> This patch to FreeBSD's libelf significantly improves test results
> from http://gcc.gnu.org/ml/gcc-testresults/2010-05/msg02081.html
> to http://gcc.gnu.org/ml/gcc-testresults/2010-05/msg02150.html

Thank you all for testing the patch!

> Based on this, it will be great if you can apply your patch to -CURRENT, 
> 8-STABLE and 7-STABLE.

I'll see what I can do.

> >From what I can tell, this patch of yours fixes LTO in GCC.  Looking
> at the remaining failures, it is now only the somewhat related WHOPR 
> optimization in GCC that remains broken:
> 
>   % cat x.c
>   int main() { }
>   % gccvs -flto x.c
>   % gccvs -fwhopr x.c
>   lto1: fatal error: elf_update() failed: Layout constraint violation
>   compilation terminated.
>   lto-wrapper: gccvs returned 1 exit status
>   collect2: lto-wrapper returned 1 exit status

The elf_update() failure is caused by an alignment check inside
FreeBSD elf_update().

FreeBSD elf_update() requires that all Elf_Data d_align must be at
least as equally strict as the section alignment. Function
lto_obj_append_data() in lto-elf.c only set the first Elf_Data d_align
to section alignment while the rest to 1. It seems that this different
alignment is desired because some .gnu_lto_XXX section contains
different kinds of data. For example, gnu.lto_.decls contains
intergers and a string table; My guess is the Elf_Data containing the
string table is set to have alignment 1 intead of padding the string
table to section alignement. I'm not sure if this is the right thing
to do... Maybe FreeBSD elf_update() aligment check is indeed too
strict.

Anyway, I attached a patch for FreeBSD libelf which removes the
d_align check against section aligment. Please try applying it to a
vanilla FreeBSD 7.3 libelf (The patch includes the previous
elf_getbase() change) and run the gcc test suite again...

I've also built a patched libelf here:
 http://people.freebsd.org/~kaiw/libelf.so.1

You can instead download this one and set your LD_LIBRARY_PATH to
avoid replacing the system libelf.

Thanks,
Kai
diff -urN libelf.orig/elf_getbase.c libelf/elf_getbase.c
--- libelf.orig/elf_getbase.c	2010-05-20 18:49:43.0 +0200
+++ libelf/elf_getbase.c	2010-05-20 18:52:49.0 +0200
@@ -34,12 +34,14 @@
 off_t
 elf_getbase(Elf *e)
 {
-	if (e == NULL ||
-	e->e_parent == NULL) {
+	if (e == NULL) {
 		LIBELF_SET_ERROR(ARGUMENT, 0);
 		return (off_t) -1;
 	}
 
-	return ((off_t) ((uintptr_t) e->e_rawfile -
-	(uintptr_t) e->e_parent->e_rawfile));
+	if (e->e_parent)
+		return ((off_t) ((uintptr_t) e->e_rawfile -
+		(uintptr_t) e->e_parent->e_rawfile));
+	else
+		return (off_t) 0;
 }
diff -urN libelf.orig/elf_update.c libelf/elf_update.c
--- libelf.orig/elf_update.c	2010-05-20 18:49:43.0 +0200
+++ libelf/elf_update.c	2010-05-23 16:29:22.0 +0200
@@ -149,11 +149,7 @@
 			LIBELF_SET_ERROR(VERSION, 0);
 			return (0);
 		}
-		if ((d_align = d->d_align) % sh_align) {
-			LIBELF_SET_ERROR(LAYOUT, 0);
-			return (0);
-		}
-		if (d_align == 0 || (d_align & (d_align - 1))) {
+		if ((d_align = d->d_align) == 0 || (d_align & (d_align - 1))) {
 			LIBELF_SET_ERROR(DATA, 0);
 			return (0);
 		}
@@ -168,7 +164,7 @@
 			if ((uint64_t) d->d_off + d->d_size > scn_size)
 scn_size = d->d_off + d->d_size;
 		} else {
-			scn_size = roundup2(scn_size, scn_alignment);
+			scn_size = roundup2(scn_size, d->d_align);
 			d->d_off = scn_size;
 			scn_size += d->d_size;
 		}


Re: LTO and libelf (and FreeBSD)

2010-05-20 Thread Kai Wang
On Sun, May 02, 2010 at 11:38:39PM +0200, Gerald Pfeifer wrote:
> As http://gcc.gnu.org/ml/gcc-testresults/2010-05/msg00120.html shows,
> *-unknown-freebsd* exhibits tons of failures around LTO.
> 
> I dug a bit deeper, and even the most trivial test program
>   int main() { }
> fails with
>   lto1: internal compiler error: compressed stream: data error
>   Please submit a full bug report,
>   with preprocessed source if appropriate.
>   See  for instructions.
>   lto-wrapper: /files/pfeifer/gcc/bin/gcc returned 1 exit status
>   collect2: lto-wrapper returned 1 exit status
> when compiled with gcc -flto x.c.

Hi Gerald,

First sorry about the late reply.

The problem is indeed caused by a small incomptibility of FreeBSD
libelf with other libelf implementations.

The elf_getbase() API in FreeBSD libelf can only be called using
an archive member ELF descriptor. It will return -1 (indicates an error)
when called with a "regular" ELF object.

The lto_obj_build_section_table() function in lto-elf.c calls
elf_getbase() to get the offset base for a "regular" ELF object. On
FreeBSD it gets -1.  (Side notes: lto-elf.c should really check return
values of libelf APIs) And later it uses -1 as base_offset for all the
ELF sections thus results in an off-by-one error when passing the
section data to zlib to inflate.

Could you please try applying the attached patch to the libelf in
FreeBSD 7.3 and see if it fixes gcc4.6 lto?

Thanks,
Kai
diff -urN libelf.orig/elf_getbase.c libelf/elf_getbase.c
--- libelf.orig/elf_getbase.c	2010-05-20 18:49:43.0 +0200
+++ libelf/elf_getbase.c	2010-05-20 18:52:49.0 +0200
@@ -34,12 +34,14 @@
 off_t
 elf_getbase(Elf *e)
 {
-	if (e == NULL ||
-	e->e_parent == NULL) {
+	if (e == NULL) {
 		LIBELF_SET_ERROR(ARGUMENT, 0);
 		return (off_t) -1;
 	}
 
-	return ((off_t) ((uintptr_t) e->e_rawfile -
-	(uintptr_t) e->e_parent->e_rawfile));
+	if (e->e_parent)
+		return ((off_t) ((uintptr_t) e->e_rawfile -
+		(uintptr_t) e->e_parent->e_rawfile));
+	else
+		return (off_t) 0;
 }