Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

Control: severity 873508 normal
Control: retitle 873508 Fix FTBFS for m68k, hurd, x32 and ppc64

On Thu, Sep 21, 2017 at 08:58:23PM +0200, Uwe Kleine-König wrote:
> I still expect some platforms to fail with the wrapper, too, because
> cgcc doesn't know about all platforms yet. But I intend to upload a new
> sparse package soon that includes a build time check for that, and the
> respective fixes are easy.

I did that now, and at least all official Debian ports pass that new
check. I downgraded the severity accordingly to normal.
 
Current build failures can be seen at

https://buildd.debian.org/status/package.php?p=sparse=unstable

hurd fails because it doesn't define PATH_MAX and NAME_MAX. m68k fails
with

sparse: ptrlist.c:125: __add_ptr_list: Assertion `(3 & (unsigned 
long)ptr) == 0' failed.

(Didn't look into this one yet.) And ppc64 and x32 need the respective
cpp defines added I think. If noone beats me to it, I will look into the
latter at least during the next few days.

Best regards
Uwe


signature.asc
Description: PGP signature

Processed: Re: Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Debian Bug Tracking System]

Processing control commands:

> severity 873508 normal
Bug #873508 [src:sparse] parsing horst source code fails on s390x and ppc64el
Severity set to 'normal' from 'serious'
> retitle 873508 Fix FTBFS for m68k, hurd, x32 and ppc64
Bug #873508 [src:sparse] parsing horst source code fails on s390x and ppc64el
Changed Bug title to 'Fix FTBFS for m68k, hurd, x32 and ppc64' from 'parsing 
horst source code fails on s390x and ppc64el'.

-- 
873508: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873508
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

Control: clone 873508 -1
Control: retitle -1 Please use cgcc to check hosted C code instead of sparse
Control: severity -1 normal
Control: reassign -1 horst

On Fri, Sep 01, 2017 at 09:46:44AM +0200, Uwe Kleine-König wrote:
> @anarcat: Given that cgcc seems to work, would you agree to apply the
> following patch to horst:
> 
> diff --git a/Makefile b/Makefile
> index 4f924fa..d563652 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -110,7 +110,7 @@ $(NAME): $(OBJS)
>  $(OBJS): .buildflags
>  
>  check:
> - sparse $(CFLAGS) *.[ch]
> + cgcc -no-compile $(CFLAGS) *.[ch]
>  
>  clean:
>   -rm -f *.o radiotap/*.o *~
> 

In the meantime I learned from upstream that sparse is not expected to
grok arbitrary hosted code. For that it is needed to use the cgcc
wrapper to handle the required cpp symbols. That it works on some
architectures with plain sparse is mostly luck.

I still expect some platforms to fail with the wrapper, too, because
cgcc doesn't know about all platforms yet. But I intend to upload a new
sparse package soon that includes a build time check for that, and the
respective fixes are easy.

> and downgrade the bug to "important"? That would be a compromise that
> buys us a bit of time.

I'd say sparse failing on hosted code isn't "important", but cgcc should
have all necessary definitions for Debian platforms. So I'm keeping this
bug at important and intend to close it once all platforms are known to
it.

Best regards
Uwe


signature.asc
Description: PGP signature

Processed: Re: Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Debian Bug Tracking System]

Processing control commands:

> clone 873508 -1
Bug #873508 [src:sparse] parsing horst source code fails on s390x and ppc64el
Bug 873508 cloned as bug 876402
> retitle -1 Please use cgcc to check hosted C code instead of sparse
Bug #876402 [src:sparse] parsing horst source code fails on s390x and ppc64el
Changed Bug title to 'Please use cgcc to check hosted C code instead of sparse' 
from 'parsing horst source code fails on s390x and ppc64el'.
> severity -1 normal
Bug #876402 [src:sparse] Please use cgcc to check hosted C code instead of 
sparse
Severity set to 'normal' from 'serious'
> reassign -1 horst
Bug #876402 [src:sparse] Please use cgcc to check hosted C code instead of 
sparse
Bug reassigned from package 'src:sparse' to 'horst'.
No longer marked as found in versions sparse/0.5.0-4.
Ignoring request to alter fixed versions of bug #876402 to the same values 
previously set

-- 
873508: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873508
876402: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876402
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Tue, Sep 12, 2017 at 2:27 AM, Uwe Kleine-König  
wrote:
>> BTW, if I want to get a PPC64 machine for Linux testing purpose, is the
>> used apple G5 a good place to start?
>
> Honestly I don't know. https://wiki.debian.org/ppc64el tells
>
> Debian/ppc64el requires, at minimum, a POWER8 processor machine.
> Although Debian was initially bootstrapped on a POWER7 set of
> servers. this class of server is not supported anymore, and you
> are not able to run Debian/ppc64el on a POWER7 processor without
> hitting an illegal instruction fault.
>
> Hm https://en.wikipedia.org/wiki/POWER8 tells:
>
> Systems based on POWER8 became available from IBM in June
> 2014. Systems and POWER8 processor designs made by other
> OpenPOWER members was available in early 2015.
>
> So I think this rules out a G5.

Thanks for the tip. I am glad I asked before I pull the trigger.

> https://wiki.debian.org/ppc64el/Installation mentions you can run this
> under qemu however.

Good idea. I will give that a try too.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

On Tue, Sep 12, 2017 at 01:59:47AM -0400, Christopher Li wrote:
> On Wed, Sep 6, 2017 at 11:36 AM, Uwe Kleine-König  
> wrote:
> > There is https://dsa.debian.org/doc/guest-account/ which would give you
> > the possibility to access some Debian machines. Other than that I intend
> 
> Sorry for the delay. Thanks for the pointer of the guest account.

Tell me if you want to do that. And plan for a delay because there is
some "paper work" to be done before; mainly by other people, so it might
(or might not) take a moment.

> > to upload 0.5.1 to Debian soon and then can provide you links to build
> > failures in the build server farm :-) And if you have a patch, I can
> > volunteer to make the test monkey for you.
> 
> BTW, if I want to get a PPC64 machine for Linux testing purpose, is the
> used apple G5 a good place to start?

Honestly I don't know. https://wiki.debian.org/ppc64el tells

Debian/ppc64el requires, at minimum, a POWER8 processor machine.
Although Debian was initially bootstrapped on a POWER7 set of
servers. this class of server is not supported anymore, and you
are not able to run Debian/ppc64el on a POWER7 processor without
hitting an illegal instruction fault.

Hm https://en.wikipedia.org/wiki/POWER8 tells:

Systems based on POWER8 became available from IBM in June
2014. Systems and POWER8 processor designs made by other
OpenPOWER members was available in early 2015.

So I think this rules out a G5.

https://wiki.debian.org/ppc64el/Installation mentions you can run this
under qemu however.

Best regards
Uwe


signature.asc
Description: PGP signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Wed, Sep 6, 2017 at 11:36 AM, Uwe Kleine-König  
wrote:
> There is https://dsa.debian.org/doc/guest-account/ which would give you
> the possibility to access some Debian machines. Other than that I intend

Sorry for the delay. Thanks for the pointer of the guest account.

> to upload 0.5.1 to Debian soon and then can provide you links to build
> failures in the build server farm :-) And if you have a patch, I can
> volunteer to make the test monkey for you.

BTW, if I want to get a PPC64 machine for Linux testing purpose, is the
used apple G5 a good place to start?

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Luc Van Oostenryck]

On Sun, Sep 10, 2017 at 10:43 AM, Uwe Kleine-König
 wrote:
>
>> Meanwhile, is it possible to have the build logs but with 'make V=1 ...' ?
>> It would also be useful to have:
>> - the output of 'uname -a'
>> - the details about the version of LLVM you're using
>
> Sure, can do. Attached is a build from the ppc64el machine with Chris'
> patch applied. Tell me if it contains everything you need.

I've taken a look at it what happens.
The problem is easy to identify and very annoying to solve:
in sparsec (a wrapper for sparse-llvm + llc + as [+ ld]) there
is a discrepancy between the defaults for llc and as.
'llc' seems to default to the sub-architecture of the build machine
(possibly including the most modern features) while 'as' defaults
to the minimal features for the build machine architecture.

The problem can be solved (on the machine I have access to) by either:
- using the option "-mgeneric" for llc
- using the option "-mpower8" for as

Something smarter would be better.

-- Luc

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Luc Van Oostenryck]

On Sun, Sep 10, 2017 at 10:43 AM, Uwe Kleine-König
 wrote:
> On 09/10/2017 03:22 AM, Luc Van Oostenryck wrote:
>>
>> I fully test on x86, x86-64, arm & ARM64 (with LLVM 3.9 or 4.0).
>> I also test on ppc64 but not the LLVM part because the machines I have
>> access to have not LLVM installed and I never bothered to install it myself.
>>
>> Would it be possible to have access to a machine with the architectures
>> you care about?
>
> Debian provides access to porter boxes for such problems. See
> https://dsa.debian.org/doc/guest-account/.

OK.
I'll first try to install LLVM on what I have already access, it
should be faster.

>> Meanwhile, is it possible to have the build logs but with 'make V=1 ...' ?
>> It would also be useful to have:
>> - the output of 'uname -a'
>> - the details about the version of LLVM you're using
>
> Sure, can do. Attached is a build from the ppc64el machine with Chris'
> patch applied. Tell me if it contains everything you need.

Yes, enough to investigate the problem. Thanks.

>> On the other hand, you/us should disable the sparse-llvm part since:
>> - it's something that is bundled and build by default but absolutely not
>>   needed (or even useful) to use sparse.
>> - it hasn't been written for anything else than x86/x86-64 (no 'layout'
>>   for anything else than those architectures.
>
> With your patch applied I get (independent of having Chris' patch
> applied or not):
>
> Out of 265 tests, 255 passed, 10 failed (10 of them are known to fail)

Perfect.
With this, you should be unblocked.

@Chris,
can you apply the patch, please?


Best regards,
-- Luc Van Oostenryck

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

On 09/10/2017 03:22 AM, Luc Van Oostenryck wrote:
> On Sat, Sep 9, 2017 at 11:02 PM, Uwe Kleine-König  
> wrote:
>>
>> I tried this on ppc64le and it fixes 2 tests, so were at
>>
>> Out of 287 tests, 273 passed, 14 failed (10 of them are known to 
>> fail)
>>
>> The repaired tests are:
>>
>> backend/hello.c
>> backend/sum.c
>>
>> unexpected failures are:
>>
>> backend/arithmetic-ops.c
>> backend/cmp-ops.c
>> backend/int-cond.c
>> backend/logical-ops.c
>>
>> These are not about missing preprocessor tokens as there are no system
>> includes used, but the error there is
>>
>> Error: unrecognized opcode: `...`
>>
>> . I didn't look into what the problem is there, but attached the test
>> log.
> 
> It clearly looks as the code generated by LLVM  (the machine code/assembly
> not LLVM's bytecode) is not understood by the assembler (or at least some
> instructions). Probably a mismatch with the architecture version or something
> like that.
> 
>> I did a build test on a few other Debian machines, arm64 was fine, mips
>> and mipx64el had 15 failures, ppc64 (i.e. big endian) had 12. I didn't
>> look in more detail and suggest to tackle one after the other :-)
> 
> I fully test on x86, x86-64, arm & ARM64 (with LLVM 3.9 or 4.0).
> I also test on ppc64 but not the LLVM part because the machines I have
> access to have not LLVM installed and I never bothered to install it myself.
> 
> Would it be possible to have access to a machine with the architectures
> you care about?

Debian provides access to porter boxes for such problems. See
https://dsa.debian.org/doc/guest-account/.

> Meanwhile, is it possible to have the build logs but with 'make V=1 ...' ?
> It would also be useful to have:
> - the output of 'uname -a'
> - the details about the version of LLVM you're using

Sure, can do. Attached is a build from the ppc64el machine with Chris'
patch applied. Tell me if it contains everything you need.

> On the other hand, you/us should disable the sparse-llvm part since:
> - it's something that is bundled and build by default but absolutely not
>   needed (or even useful) to use sparse.
> - it hasn't been written for anything else than x86/x86-64 (no 'layout'
>   for anything else than those architectures.

With your patch applied I get (independent of having Chris' patch
applied or not):

Out of 265 tests, 255 passed, 10 failed (10 of them are known to fail)

Best regards
Uwe


buildlog-23a393b1cd48ea50bff94fa4a1e2c02a5d78d9cb+
Description: Binary data


signature.asc
Description: OpenPGP digital signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Luc Van Oostenryck]

On Sat, Sep 9, 2017 at 11:02 PM, Uwe Kleine-König  
wrote:
>
> I tried this on ppc64le and it fixes 2 tests, so were at
>
> Out of 287 tests, 273 passed, 14 failed (10 of them are known to fail)
>
> The repaired tests are:
>
> backend/hello.c
> backend/sum.c
>
> unexpected failures are:
>
> backend/arithmetic-ops.c
> backend/cmp-ops.c
> backend/int-cond.c
> backend/logical-ops.c
>
> These are not about missing preprocessor tokens as there are no system
> includes used, but the error there is
>
> Error: unrecognized opcode: `...`
>
> . I didn't look into what the problem is there, but attached the test
> log.

It clearly looks as the code generated by LLVM  (the machine code/assembly
not LLVM's bytecode) is not understood by the assembler (or at least some
instructions). Probably a mismatch with the architecture version or something
like that.

> I did a build test on a few other Debian machines, arm64 was fine, mips
> and mipx64el had 15 failures, ppc64 (i.e. big endian) had 12. I didn't
> look in more detail and suggest to tackle one after the other :-)

I fully test on x86, x86-64, arm & ARM64 (with LLVM 3.9 or 4.0).
I also test on ppc64 but not the LLVM part because the machines I have
access to have not LLVM installed and I never bothered to install it myself.

Would it be possible to have access to a machine with the architectures
you care about?

Meanwhile, is it possible to have the build logs but with 'make V=1 ...' ?
It would also be useful to have:
- the output of 'uname -a'
- the details about the version of LLVM you're using

On the other hand, you/us should disable the sparse-llvm part since:
- it's something that is bundled and build by default but absolutely not
  needed (or even useful) to use sparse.
- it hasn't been written for anything else than x86/x86-64 (no 'layout'
  for anything else than those architectures.

Best regards,
-- Luc Van Ooostenryck

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

On Thu, Aug 31, 2017 at 08:47:55PM -0400, Christopher Li wrote:
> On Thu, Aug 31, 2017 at 4:55 PM, Uwe Kleine-König  Yes
> that works. So to address the Debian bug I can do:
> >
> >  - move sparse to /usr/lib
> >  - teach cgcc about the move of sparse
> >  - make /usr/bin/sparse call cgcc -no-compile "$@"
> 
> I don't like that. It means the user can't invoke sparse directly.
> 
> >
> > or is it easier to teach sparse about the architecture stuff?
> 
> First of all. It is not very trivial to teach sparse about the architecture
> stuff. To my mind, we need to move all the cgcc logic into sparse.
> 
> For this case, I think it is easier to teach the sparse validation
> code to use cgcc on those back end testing. Most validation don't
> need to include system header file at all so it does not have
> this problem.
> 
> How about this patch?
> I know my patch is white space damaged in email.
> Git branch is at:
> https://git.kernel.org/pub/scm/devel/sparse/chrisl/sparse.git/log/?h=llvm-cgcc
> 
> Please let me know if that fix your problem. It pass check
> on my local machine running x86_64. I don't have ppc64 to
> test with.
> 
> Chris
> 
> diff --git a/sparsec b/sparsec
> index 9dc96c9..2990d26 100755
> --- a/sparsec
> +++ b/sparsec
> @@ -32,7 +32,8 @@ done
>  TMPLLVM=`mktemp -t tmp.XX`".llvm"
>  TMPFILE=`mktemp -t tmp.XX`".o"
> 
> -$DIRNAME/sparse-llvm $SPARSEOPTS > $TMPLLVM
> +env CHECK=$DIRNAME/sparse-llvm $DIRNAME/cgcc -no-compile \
> +   $SPARSEOPTS > $TMPLLVM
> 
>  LLC=`"${LLVM_CONFIG:-llvm-config}" --bindir`/llc
> 
> diff --git a/sparsei b/sparsei
> index 3431a9f..3abd00f 100755
> --- a/sparsei
> +++ b/sparsei
> @@ -10,4 +10,4 @@ if [ $# -eq 0 ]; then
>exit 1
>  fi
> 
> -$DIRNAME/sparse-llvm $@ | $LLI
> +env CHECK=$DIRNAME/sparse-llvm $DIRNAME/cgcc -no-compile $@ | $LLI

I tried this on ppc64le and it fixes 2 tests, so were at

Out of 287 tests, 273 passed, 14 failed (10 of them are known to fail)

The repaired tests are:

backend/hello.c
backend/sum.c

unexpected failures are:

backend/arithmetic-ops.c
backend/cmp-ops.c
backend/int-cond.c
backend/logical-ops.c

These are not about missing preprocessor tokens as there are no system
includes used, but the error there is

Error: unrecognized opcode: `...`

. I didn't look into what the problem is there, but attached the test
log.

I did a build test on a few other Debian machines, arm64 was fine, mips
and mipx64el had 15 failures, ppc64 (i.e. big endian) had 12. I didn't
look in more detail and suggest to tackle one after the other :-)

Best regards
Uwe
I: Started sh -c make && make check
O: make: Nothing to be done for 'all'.
O:  TESTWoverride-init-def (Woverride-init-def.c)
O:  TESTWoverride-init-no (Woverride-init-no.c)
O:  TESTWoverride-init-yes (Woverride-init-yes.c)
O:  TESTwarn-unknown-attribute (Wunknown-attribute-def.c)
O:  TESTwarn-unknown-attribute-no (Wunknown-attribute-no.c)
O:  TESTwarn-unknown-attribute-yes (Wunknown-attribute-yes.c)
O:  TEST__func__ (__func__.c)
O:  TESTabstract array declarator static 
(abstract-array-declarator-static.c)
O:  TESTaddress_space attribute (address_space.c)
O:  TESTalias distinct symbols (alias-distinct.c)
O:  TESTalias symbol/pointer (alias-mixed.c)
O:  TESTalias same symbols (alias-same.c)
O:  TESTattribute __alloc_align__ (alloc-align.c)
O:  TESTalternate keywords (alternate-keywords.c)
O:  TESTtest anonymous union initializer (anon-union.c)
O:  TESTAsm with goto labels. (asm-empty-clobber.c)
O:  TESTAsm with goto labels. (asm-goto-lables.c)
O:  TESTasm-toplevel.c (asm-toplevel.c)
O:  TESTinline attributes (attr-inline.c)
O:  TESTattribute no_sanitize_address (attr-no_sanitize_address.c)
O:  TESTattribute noclone (attr-noclone.c)
O:  TESToptimize attributes (attr-optimize.c)
O:  TESTattribute warning (attr-warning.c)
O:  TESTattribute assume_aligned (attr_aligned.c)
O:  TESTattribute after ( in direct-declarator (attr_in_parameter.c)
O:  TESTattribute vector_size (attr_vector_size.c)
O:  TESTArithmetic operator code generation (backend/arithmetic-ops.c)
O: error: actual error text does not match expected error text.
O: error: see backend/arithmetic-ops.c.error.* for further investigation.
O: --- backend/arithmetic-ops.c.error.expected  2017-09-09 20:44:47.964306005 
+
O: +++ backend/arithmetic-ops.c.error.got   2017-09-09 20:44:47.960305943 
+
O: @@ -0,0 +1,10 @@
O: +{standard input}: Assembler messages:
O: +{standard input}:38: Error: unrecognized opcode: `xsaddsp'
O: +{standard input}:52: Error: unrecognized opcode: `xsadddp'
O: +{standard input}:94: Error: unrecognized opcode: `xssubsp'
O: +{standard input}:108: Error: unrecognized opcode: `xssubdp'
O: +{standard input}:150: Error: 

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

Hello Christopher,

On Wed, Sep 06, 2017 at 11:18:04AM -0400, Christopher Li wrote:
> On Wed, Sep 6, 2017 at 10:44 AM, Uwe Kleine-König  
> wrote:
> > and while it's ok to test the core stuff and not wanting the system
> > includes to interfere, there should also be tests that check "ordinary"
> > userspace programs which naturally depend on the system headers.
> 
> There is one. The "selfcheck" target was checking sparse on its own
> source file. That will definitively use the system header file. However,
> there are some warning trigger in the system header file can't be fixed
> in the sparse source code. It need to change the system header to make
> some warning go away, or disable that warning.
> 
> >
> > No, it's not solved. But given that it is somehow known that sparse
> > (without cgcc) fails to work well on userspace stuff, I think the
> > following would be fine for the Debian side:
> >
> >  a) let horst use cgcc instead of sparse
> >  b) downgrade bug to important (or even normal) pointing out that
> > cgcc should be used for userspace programs
> 
> That seems to be the right thing to do for now. That is until
> sparse are smarter on user space header regarding architecture stuff.

ok, Antoine, can you talk to the horst people and ask them to switch to
cgcc then?

> > For sparse it would be cool to:
> >
> >  c) drop the #weak_define of __amd64__ to make this "problem" more
> > apparent. (Assuming this doesn't break e.g. a kernel build.)
> 
> You mean remove define of "__x86_64__".

ack.

> It will likely break some other stuff. For the record the "selfcheck" target
> already using cgcc. We still need to fix the breakage.
> 
> Any suggestion how to test sparse running on other platform headfile
> without have to get access to ppc64 for example? I think that is the biggest
> obstacle right now. I can make some changes, but I don't have a good way
> to test it other than x86 platform.

There is https://dsa.debian.org/doc/guest-account/ which would give you
the possibility to access some Debian machines. Other than that I intend
to upload 0.5.1 to Debian soon and then can provide you links to build
failures in the build server farm :-) And if you have a patch, I can
volunteer to make the test monkey for you.

> >  d) fix the test suite, at least mark the tests that are expected to
> > fail on !x86 as such to make $(make check) succeed. (Otherwise I'd
> > have to disable or ignore the testsuite which isn't that great.)
> 
> We can make the test-suite not depend on system header files.
> That seems to be the right think to do. I also send out a patch
> to let the llvm back end test-suite use cgcc last week. Removing
> system header usage in test suite is better.

Testing that patch is on my todo list.

Best regards
Uwe


signature.asc
Description: PGP signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Wed, Sep 6, 2017 at 10:44 AM, Uwe Kleine-König  
wrote:
>
> and while it's ok to test the core stuff and not wanting the system
> includes to interfere, there should also be tests that check "ordinary"
> userspace programs which naturally depend on the system headers.
>

There is one. The "selfcheck" target was checking sparse on its own
source file. That will definitively use the system header file. However,
there are some warning trigger in the system header file can't be fixed
in the sparse source code. It need to change the system header to make
some warning go away, or disable that warning.

>
> No, it's not solved. But given that it is somehow known that sparse
> (without cgcc) fails to work well on userspace stuff, I think the
> following would be fine for the Debian side:
>
>  a) let horst use cgcc instead of sparse
>  b) downgrade bug to important or even normal pointing out that cgcc
> should be used for userspace programs

That seems to be the right thing to do for now. That is until
sparse are smarter on user space header regarding architecture stuff.

> For sparse it would be cool to:
>
>  c) drop the #weak_define of __amd64__ to make this "problem" more
> apparent. (Assuming this doesn't break e.g. a kernel build.)

You mean remove define of "__x86_64__".

It will likely break some other stuff. For the record the "selfcheck" target
already using cgcc. We still need to fix the breakage.

Any suggestion how to test sparse running on other platform headfile
without have to get access to ppc64 for example? I think that is the biggest
obstacle right now. I can make some changes, but I don't have a good way
to test it other than x86 platform.

>  d) fix the test suite, at least mark the tests that are expected to
> fail on !x86 as such to make $(make check) succeed. (Otherwise I'd
> have to disable or ignore the testsuite which isn't that great.)

We can make the test-suite not depend on system header files.
That seems to be the right think to do. I also send out a patch
to let the llvm back end test-suite use cgcc last week. Removing
system header usage in test suite is better.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

[readding people to Cc assuming that's ok]

Hello Luc,

On Mon, Sep 04, 2017 at 10:36:47PM +0200, Luc Van Oostenryck wrote:
> On Mon, Sep 4, 2017 at 10:57 AM, Uwe Kleine-König  
> wrote:
> > On 09/03/2017 11:14 PM, Luc Van Oostenryck wrote:
> >> On Fri, Sep 1, 2017 at 9:02 AM, Josh Triplett  
> >> wrote:
> >>> Related to that: while it would mean we couldn't necessarily just rely
> >>> entirely on GCC's definitions for a target platform, I think in an ideal
> >>> world we could have a sparse binary that understood *all* target
> >>> platforms at once, such that you could ask Sparse on x86_64 to "compile"
> >>> as though targeting any arbitrary architecture. That would also have the
> >>> major advantage of making it easy to run the Sparse testsuite for
> >>> *every* target architecture without needing compilers for every such
> >>> architecture.
> >>
> >> I really think that the testsuite should not depend on system or library
> >> header.
> >
> > Assuming it's intended that sparse should be able to check userspace
> > programs, I don't agree here.
> 
> I understand this.
> I'll explain a bit better my point of view.
> First, I make a distinction between 'sparse core functionalities' and
> general usage.
> I was talking about this core usage and the testsuite is currently for this 
> core
> usage too.

and while it's ok to test the core stuff and not wanting the system
includes to interfere, there should also be tests that check "ordinary"
userspace programs which naturally depend on the system headers.

> Asking for the testsuite to not depends on system or library header is exactly
> the same as GCC people asking bug reports to be done on pre-processed file
> (so that they focus on the core problem and not some problem with an header).
> This, of course, doesn't mean that GCC should only be used on standalone
> source files nor that GCC shouldn't be tested on real code, using
> system headers.
> It's just something different.
> 
> So to answer to your objection: yes, you're right but it should be done in 
> some
> specific tests, not the core ones.

ah, we agree. Fine.

> Secondly, about "intended to check userspace programs":
> It's clear that sparse's main use is for the kernel, but it's also
> clear that it can
> and is used on other (userspace) projects.
> However, as you have seen yourself, you can't use sparse as is and expect
> to work on any environment, on any architecture. Even for the kernel it 
> doesn't:
> each architecture has to specify a few flags (like -m32/-m64) and a few 
> defines
> (-D__arm__, ...). For userspace, cgcc can do a part of this job for you.
> 
> Josh proposal to have what I called a 'universal' sparse, won't solve this,
> on the contrary.
> Compilers eschew part of this problem by having to configure the build
> 
> I'm all in favor to move cgcc logic to sparse and/or it's build system so that
> *for a native build* it can be used as-is in most cases.
> This would solve your problem, I think.
> 
> BTW, sorry I didn't follow last week but is your problem solved now?

No, it's not solved. But given that it is somehow known that sparse
(without cgcc) fails to work well on userspace stuff, I think the
following would be fine for the Debian side:

 a) let horst use cgcc instead of sparse
 b) downgrade bug to important or even normal pointing out that cgcc
should be used for userspace programs

For sparse it would be cool to:

 c) drop the #weak_define of __amd64__ to make this "problem" more
apparent. (Assuming this doesn't break e.g. a kernel build.)
 d) fix the test suite, at least mark the tests that are expected to
fail on !x86 as such to make $(make check) succeed. (Otherwise I'd
have to disable or ignore the testsuite which isn't that great.)
 
Best regards
Uwe


signature.asc
Description: PGP signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Sun, Sep 3, 2017 at 5:14 PM, Luc Van Oostenryck
 wrote:
>
> I really think that the testsuite should not depend on system or library
> header.

I think that is a good point. We can start cleaning up the system header
file dependency in the existing test suite. See how it goes.

>
> Otherwise, I'm not at all opposed to sparse being universal but I would like
> to note that things can become very quickly very very messy.
> For example, for the current problem here I understood that it was
> at least partially based on the lack of a definition of _CALL_ELF
> but do we need to define it to 1 or to 2, in other words, do we need
> to support the ELFv1 ABI or the ELFv2? GCC has some flags for this
> (-mabi=elfv[12]) but what default value do we want? ELFv1 is the default

I think we can just sparse default to as late as the latest release
version of gcc.

> for big-endian platform and ELFv2 for little-endian platform, so yes,
> we need a flag for the endianness but which endianness we want as default?

I am tempting to make the endianness the same as the host gcc by default.
Then it can be overwrite by architecture flags.

>
> Things become even more fun when taking in account the difference
> between GCC version. Do we want to be universal there too (and thus
> have some flags for to specify which gcc's version we want to mimick)?
> What about other compilers?

I purpose just sync to the latest gcc version (or a late enough version
we can agree on. e.g. the one that supported by kernel compile.) Sparse
current try to sync to the latest gcc attributes already.

> I think that part of the needed info can be auto-extracted from GCC
> when doing a native build. Using some sort of spec file or a .sparserc

Is there a way to do auto-extract? That would be a good starting point.

> I also note that currently, sparse is already largely universal *because*
> it *doesn't* need those platform details (or only the very minimal: word 
> size).

Sparse is not universal, it just support a very small sub set of the C
source file that haven't expose to those platform detail macros. Adding
those architecture macro support will not make it any less universal.
Might slow things down, that is some thing we need to watch out for.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Luc Van Oostenryck]

On Fri, Sep 1, 2017 at 9:02 AM, Josh Triplett  wrote:
> On Thu, Aug 31, 2017 at 08:47:55PM -0400, Christopher Li wrote:
>> On Thu, Aug 31, 2017 at 4:55 PM, Uwe Kleine-König  Yes
>> that works. So to address the Debian bug I can do:
>> >
>> >  - move sparse to /usr/lib
>> >  - teach cgcc about the move of sparse
>> >  - make /usr/bin/sparse call cgcc -no-compile "$@"
>>
>> I don't like that. It means the user can't invoke sparse directly.
>>
>> >
>> > or is it easier to teach sparse about the architecture stuff?
>>
>> First of all. It is not very trivial to teach sparse about the architecture
>> stuff. To my mind, we need to move all the cgcc logic into sparse.
>
> Related to that: while it would mean we couldn't necessarily just rely
> entirely on GCC's definitions for a target platform, I think in an ideal
> world we could have a sparse binary that understood *all* target
> platforms at once, such that you could ask Sparse on x86_64 to "compile"
> as though targeting any arbitrary architecture. That would also have the
> major advantage of making it easy to run the Sparse testsuite for
> *every* target architecture without needing compilers for every such
> architecture.

I really think that the testsuite should not depend on system or library
header.

Otherwise, I'm not at all opposed to sparse being universal but I would like
to note that things can become very quickly very very messy.
For example, for the current problem here I understood that it was
at least partially based on the lack of a definition of _CALL_ELF
but do we need to define it to 1 or to 2, in other words, do we need
to support the ELFv1 ABI or the ELFv2? GCC has some flags for this
(-mabi=elfv[12]) but what default value do we want? ELFv1 is the default
for big-endian platform and ELFv2 for little-endian platform, so yes,
we need a flag for the endianness but which endianness we want as default?
And so on.

Things become even more fun when taking in account the difference
between GCC version. Do we want to be universal there too (and thus
have some flags for to specify which gcc's version we want to mimick)?
What about other compilers?


I think that part of the needed info can be auto-extracted from GCC
when doing a native build. Using some sort of spec file or a .sparserc
can help too.

I also note that currently, sparse is already largely universal *because*
it *doesn't* need those platform details (or only the very minimal: word size).

-- Luc Van Oostenryck

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Josh Triplett]

On Fri, Sep 01, 2017 at 09:57:09AM +0200, Uwe Kleine-König wrote:
> On Fri, Sep 01, 2017 at 12:02:12AM -0700, Josh Triplett wrote:
> > On Thu, Aug 31, 2017 at 08:47:55PM -0400, Christopher Li wrote:
> > > On Thu, Aug 31, 2017 at 4:55 PM, Uwe Kleine-König  Yes
> > > that works. So to address the Debian bug I can do:
> > > >
> > > >  - move sparse to /usr/lib
> > > >  - teach cgcc about the move of sparse
> > > >  - make /usr/bin/sparse call cgcc -no-compile "$@"
> > > 
> > > I don't like that. It means the user can't invoke sparse directly.
> > > 
> > > >
> > > > or is it easier to teach sparse about the architecture stuff?
> > > 
> > > First of all. It is not very trivial to teach sparse about the 
> > > architecture
> > > stuff. To my mind, we need to move all the cgcc logic into sparse.
> > 
> > Related to that: while it would mean we couldn't necessarily just rely
> > entirely on GCC's definitions for a target platform, I think in an ideal
> > world we could have a sparse binary that understood *all* target
> > platforms at once, such that you could ask Sparse on x86_64 to "compile"
> > as though targeting any arbitrary architecture. That would also have the
> > major advantage of making it easy to run the Sparse testsuite for
> > *every* target architecture without needing compilers for every such
> > architecture.
> 
> You'd need the target arch's system headers though.

Only for building userspace code, not for building standalone/kernel
code, or the Sparse testsuite.

- Josh Triplett

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Fri, Sep 1, 2017 at 3:02 AM, Josh Triplett  wrote:
>> First of all. It is not very trivial to teach sparse about the architecture
>> stuff. To my mind, we need to move all the cgcc logic into sparse.
>
> Related to that: while it would mean we couldn't necessarily just rely
> entirely on GCC's definitions for a target platform, I think in an ideal
> world we could have a sparse binary that understood *all* target
> platforms at once, such that you could ask Sparse on x86_64 to "compile"

Yes, that is what I want to have. It is list as one of the project in
project idea document as well. I have  a related question. How do we
test the different architecture handling without actually run sparse
on different platform? I am thinking maybe using gcc cross platform
compiler and compare some macro against the sparse one.

> as though targeting any arbitrary architecture. That would also have the
> major advantage of making it easy to run the Sparse testsuite for
> *every* target architecture without needing compilers for every such
> architecture.

Another way to fix the test suite for now would be let testsuite
specify using cgcc instead of sparse directly, for the test source
that needs it. That will buy us some time. Fixing sparse properly
in the long term obvious would be better.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Fri, Sep 1, 2017 at 3:46 AM, Uwe Kleine-König  wrote:
\>
> Nearly right. I'm responsible for the sparse Debian package and the
> problem at hand is https://bugs.debian.org/873508. This bug report has
> "Severity: serious" wihch might eventually result in the removal of
> sparse from the Debian archive.
>
> @anarcat: Given that cgcc seems to work, would you agree to apply the
> following patch to horst:
>
> diff --git a/Makefile b/Makefile
> index 4f924fa..d563652 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -110,7 +110,7 @@ $(NAME): $(OBJS)
>  $(OBJS): .buildflags
>
>  check:
> -   sparse $(CFLAGS) *.[ch]
> +   cgcc -no-compile $(CFLAGS) *.[ch]

You patch definitely works.
I think it is the better way to fix it.
Sparse "selfcheck" target was doing the same thing.
It is using "cgcc -no-compile" already.

I suggest in your patch you can do some thing
similar to "selfcheck" target:

CHECKER = cgcc -no-compile

 $(CHECKER) $(CFLAGS) *.[ch]

Later when we update sparse to handle architecture
properly. We can just invoke make with "CHECKER="
to test.


> and downgrade the bug to "important"? That would be a compromise that
> buys us a bit of time.

Agree.

>
> This only solves a part of the problem because the bug report is about
> sparse itself, not it's llvm part.

I agree with that too.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Antoine Beaupré]

On 2017-09-01 09:46:44, Uwe Kleine-König wrote:
> Hello,
>
> On Thu, Aug 31, 2017 at 11:43:53PM +0100, Ramsay Jones wrote:
>> On 31/08/17 21:55, Uwe Kleine-König wrote:
>> > On Wed, Aug 30, 2017 at 08:11:49PM -0400, Christopher Li wrote:
>> >> That is very much like on x86_64 missing define "#weak_define __x86_64__ 
>> >> 1"
>> >>
>> >> Does cgcc work for you? In the future we do want to move the archetecture
>> >> related define from cgcc into sparse by itself. For now you can set
>> >> "sparse" as "cgcc -no-compile"
>> > 
>> > Yes that works. So to address the Debian bug I can do:
>> > 
>> >  - move sparse to /usr/lib
>> >  - teach cgcc about the move of sparse
>> >  - make /usr/bin/sparse call cgcc -no-compile "$@"
>> 
>> Hmm, I don't think that would be a good idea ...
>> 
>> > or is it easier to teach sparse about the architecture stuff?
>> 
>> I now understand (I think!) that you are building a sparse
>> package (presumably a .deb) and you are concerned that sparse
>> does not pass it's own testsuite on those platforms.
>
> Nearly right. I'm responsible for the sparse Debian package and the
> problem at hand is https://bugs.debian.org/873508. This bug report has
> "Severity: serious" wihch might eventually result in the removal of
> sparse from the Debian archive.
>
> @anarcat: Given that cgcc seems to work, would you agree to apply the
> following patch to horst:
>
> diff --git a/Makefile b/Makefile
> index 4f924fa..d563652 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -110,7 +110,7 @@ $(NAME): $(OBJS)
>  $(OBJS): .buildflags
>  
>  check:
> - sparse $(CFLAGS) *.[ch]
> + cgcc -no-compile $(CFLAGS) *.[ch]
>  
>  clean:
>   -rm -f *.o radiotap/*.o *~
>
> and downgrade the bug to "important"? That would be a compromise that
> buys us a bit of time.

Well right now I have simply disabled the checks for those broken
architectures, so sparse isn't as affected anymore.

Frankly, I don't quite know what to do with this - but I'd be happy to
happly that patch to sparse if it fixes the issue better.

A.

-- 
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin, 1755

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

On Fri, Sep 01, 2017 at 12:02:12AM -0700, Josh Triplett wrote:
> On Thu, Aug 31, 2017 at 08:47:55PM -0400, Christopher Li wrote:
> > On Thu, Aug 31, 2017 at 4:55 PM, Uwe Kleine-König  Yes
> > that works. So to address the Debian bug I can do:
> > >
> > >  - move sparse to /usr/lib
> > >  - teach cgcc about the move of sparse
> > >  - make /usr/bin/sparse call cgcc -no-compile "$@"
> > 
> > I don't like that. It means the user can't invoke sparse directly.
> > 
> > >
> > > or is it easier to teach sparse about the architecture stuff?
> > 
> > First of all. It is not very trivial to teach sparse about the architecture
> > stuff. To my mind, we need to move all the cgcc logic into sparse.
> 
> Related to that: while it would mean we couldn't necessarily just rely
> entirely on GCC's definitions for a target platform, I think in an ideal
> world we could have a sparse binary that understood *all* target
> platforms at once, such that you could ask Sparse on x86_64 to "compile"
> as though targeting any arbitrary architecture. That would also have the
> major advantage of making it easy to run the Sparse testsuite for
> *every* target architecture without needing compilers for every such
> architecture.

You'd need the target arch's system headers though. But still it would
be great. In a first attempt something like:

#ifdef __powerpc__
add_pre_buffer("#weak_define __powerpc__ " __powerpc__ "\n");
#ifdef _CALL_ELF
add_pre_buffer("#weak_define _CALL_ELF " _CALL_ELF "\n");
#endif
#endif

would be helpful already.

Best regards
Uwe


signature.asc
Description: PGP signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

Hello,

On Thu, Aug 31, 2017 at 11:43:53PM +0100, Ramsay Jones wrote:
> On 31/08/17 21:55, Uwe Kleine-König wrote:
> > On Wed, Aug 30, 2017 at 08:11:49PM -0400, Christopher Li wrote:
> >> That is very much like on x86_64 missing define "#weak_define __x86_64__ 1"
> >>
> >> Does cgcc work for you? In the future we do want to move the archetecture
> >> related define from cgcc into sparse by itself. For now you can set
> >> "sparse" as "cgcc -no-compile"
> > 
> > Yes that works. So to address the Debian bug I can do:
> > 
> >  - move sparse to /usr/lib
> >  - teach cgcc about the move of sparse
> >  - make /usr/bin/sparse call cgcc -no-compile "$@"
> 
> Hmm, I don't think that would be a good idea ...
> 
> > or is it easier to teach sparse about the architecture stuff?
> 
> I now understand (I think!) that you are building a sparse
> package (presumably a .deb) and you are concerned that sparse
> does not pass it's own testsuite on those platforms.

Nearly right. I'm responsible for the sparse Debian package and the
problem at hand is https://bugs.debian.org/873508. This bug report has
"Severity: serious" wihch might eventually result in the removal of
sparse from the Debian archive.

@anarcat: Given that cgcc seems to work, would you agree to apply the
following patch to horst:

diff --git a/Makefile b/Makefile
index 4f924fa..d563652 100644
--- a/Makefile
+++ b/Makefile
@@ -110,7 +110,7 @@ $(NAME): $(OBJS)
 $(OBJS): .buildflags
 
 check:
-   sparse $(CFLAGS) *.[ch]
+   cgcc -no-compile $(CFLAGS) *.[ch]
 
 clean:
-rm -f *.o radiotap/*.o *~

and downgrade the bug to "important"? That would be a compromise that
buys us a bit of time.
 
> As I said before, the additional failures you are seeing are
> in the 'llvm backend' code (which, as far as I know, only passes
> on x86_64 Linux), and in my opinion the llvm-backend programs should
> not be installed. (The Makefile will build them automatically if
> you have llvm installed, likewise for c2xml/libxml and test-inspect/gtk).

Currently the sparse package contains /usr/include/sparse/, c2xml, cgcc,
sparse, sparse-llvm, sparsec and a separate package sparse-test-inspect
includes test-inspect. (That's how I inherited the package some time
ago.)
 
> [I would like to see build variable(s) to allow the user to suppress
> the build (or installation) of the other 'non-primary' sparse programs.]
> 
> Anyway, if you were to un-install llvm, sparse-llvm etc., would not
> be built, and the tests would not be run ... ;-)
> 
> Christopher, as the project maintainer, has the joy of making these
> kinds of decisions! :-D

This only solves a part of the problem because the bug report is about
sparse itself, not it's llvm part.

Best regards
Uwe


signature.asc
Description: PGP signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Josh Triplett]

On Thu, Aug 31, 2017 at 08:47:55PM -0400, Christopher Li wrote:
> On Thu, Aug 31, 2017 at 4:55 PM, Uwe Kleine-König  Yes
> that works. So to address the Debian bug I can do:
> >
> >  - move sparse to /usr/lib
> >  - teach cgcc about the move of sparse
> >  - make /usr/bin/sparse call cgcc -no-compile "$@"
> 
> I don't like that. It means the user can't invoke sparse directly.
> 
> >
> > or is it easier to teach sparse about the architecture stuff?
> 
> First of all. It is not very trivial to teach sparse about the architecture
> stuff. To my mind, we need to move all the cgcc logic into sparse.

Related to that: while it would mean we couldn't necessarily just rely
entirely on GCC's definitions for a target platform, I think in an ideal
world we could have a sparse binary that understood *all* target
platforms at once, such that you could ask Sparse on x86_64 to "compile"
as though targeting any arbitrary architecture. That would also have the
major advantage of making it easy to run the Sparse testsuite for
*every* target architecture without needing compilers for every such
architecture.

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Thu, Aug 31, 2017 at 6:43 PM, Ramsay Jones
>>  - move sparse to /usr/lib
>>  - teach cgcc about the move of sparse
>>  - make /usr/bin/sparse call cgcc -no-compile "$@"
>
> Hmm, I don't think that would be a good idea ...
>

Agree.

>
> Anyway, if you were to un-install llvm, sparse-llvm etc., would not
> be built, and the tests would not be run ... ;-)

I think Uwe already confirm using cgcc to invoke sparse will
fix the problem.

In this test, we just need to use cgcc to invoke sparse-llvm.
I have a patch to fix the usage in test-suite in the other email.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Thu, Aug 31, 2017 at 4:55 PM, Uwe Kleine-König  Yes
that works. So to address the Debian bug I can do:
>
>  - move sparse to /usr/lib
>  - teach cgcc about the move of sparse
>  - make /usr/bin/sparse call cgcc -no-compile "$@"

I don't like that. It means the user can't invoke sparse directly.

>
> or is it easier to teach sparse about the architecture stuff?

First of all. It is not very trivial to teach sparse about the architecture
stuff. To my mind, we need to move all the cgcc logic into sparse.

For this case, I think it is easier to teach the sparse validation
code to use cgcc on those back end testing. Most validation don't
need to include system header file at all so it does not have
this problem.

How about this patch?
I know my patch is white space damaged in email.
Git branch is at:
https://git.kernel.org/pub/scm/devel/sparse/chrisl/sparse.git/log/?h=llvm-cgcc

Please let me know if that fix your problem. It pass check
on my local machine running x86_64. I don't have ppc64 to
test with.

Chris

diff --git a/sparsec b/sparsec
index 9dc96c9..2990d26 100755
--- a/sparsec
+++ b/sparsec
@@ -32,7 +32,8 @@ done
 TMPLLVM=`mktemp -t tmp.XX`".llvm"
 TMPFILE=`mktemp -t tmp.XX`".o"

-$DIRNAME/sparse-llvm $SPARSEOPTS > $TMPLLVM
+env CHECK=$DIRNAME/sparse-llvm $DIRNAME/cgcc -no-compile \
+   $SPARSEOPTS > $TMPLLVM

 LLC=`"${LLVM_CONFIG:-llvm-config}" --bindir`/llc

diff --git a/sparsei b/sparsei
index 3431a9f..3abd00f 100755
--- a/sparsei
+++ b/sparsei
@@ -10,4 +10,4 @@ if [ $# -eq 0 ]; then
   exit 1
 fi

-$DIRNAME/sparse-llvm $@ | $LLI
+env CHECK=$DIRNAME/sparse-llvm $DIRNAME/cgcc -no-compile $@ | $LLI

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Ramsay Jones]

On 31/08/17 21:55, Uwe Kleine-König wrote:
> On Wed, Aug 30, 2017 at 08:11:49PM -0400, Christopher Li wrote:
>> That is very much like on x86_64 missing define "#weak_define __x86_64__ 1"
>>
>> Does cgcc work for you? In the future we do want to move the archetecture
>> related define from cgcc into sparse by itself. For now you can set
>> "sparse" as "cgcc -no-compile"
> 
> Yes that works. So to address the Debian bug I can do:
> 
>  - move sparse to /usr/lib
>  - teach cgcc about the move of sparse
>  - make /usr/bin/sparse call cgcc -no-compile "$@"

Hmm, I don't think that would be a good idea ...

> or is it easier to teach sparse about the architecture stuff?

I now understand (I think!) that you are building a sparse
package (presumably a .deb) and you are concerned that sparse
does not pass it's own testsuite on those platforms.

As I said before, the additional failures you are seeing are
in the 'llvm backend' code (which, as far as I know, only passes
on x86_64 Linux), and in my opinion the llvm-backend programs should
not be installed. (The Makefile will build them automatically if
you have llvm installed, likewise for c2xml/libxml and test-inspect/gtk).

[I would like to see build variable(s) to allow the user to suppress
the build (or installation) of the other 'non-primary' sparse programs.]

Anyway, if you were to un-install llvm, sparse-llvm etc., would not
be built, and the tests would not be run ... ;-)

Christopher, as the project maintainer, has the joy of making these
kinds of decisions! :-D

ATB,
Ramsay Jones

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

Hello Christopher,

On Wed, Aug 30, 2017 at 08:11:49PM -0400, Christopher Li wrote:
> On Wed, Aug 30, 2017 at 1:36 PM, Uwe Kleine-König  
> wrote:
> >> >
> >> > diff --git a/validation/backend/sum.c b/validation/backend/sum.c
> >> > index 0604299..d0be8dd 100644
> >> > --- a/validation/backend/sum.c
> >> > +++ b/validation/backend/sum.c
> >> > @@ -1,3 +1,5 @@
> >> > +#define __powerpc64__
> >> > +#define _CALL_ELF 2
> >> >  #include 
> >> >  #include 
> >> >
> >> >
> >>
> >> Yep, sparse/sparsec do not define various macros that gcc/clang define
> >> by default on a given architecture. This is a known problem (that I have
> >> been meaning to fix ...). The 'workaround' for the time being is to use
> >> the cgcc front-end to sparse. (for example 'make CC=cgcc', or perhaps
> >> 'cgcc -no-compile').
> >>
> >> [You didn't mention your usage - is this for a kernel build?]
> >
> > This problem became visible during the make check phase when creating 
> > packaged
> > on the listed archs for horst[1]. You can see a build logs at
> >
> > 
> > https://buildd.debian.org/status/fetch.php?pkg=horst=s390x=5.0-1=1503905687=0
> > 
> > https://buildd.debian.org/status/fetch.php?pkg=horst=ppc64el=5.0-1=1503906226=0
> >
> > The error message looks identical (I checked the ppc64el log) to the
> > problem with backend/sum.c:
> >
> > sparse -g -O2 -fdebug-prefix-map=/<>=. 
> > -fstack-protector-strong -Wformat -Werror=format-security -std=gnu99 -Wall 
> > -Wextra -g -I. -DDO_DEBUG -I/usr/include/libnl3 *.[ch]
> > /usr/include/powerpc64le-linux-gnu/gnu/stubs.h:8:12: error: unable 
> > to open 'gnu/stubs-32.h'
> 
> That is very much like on x86_64 missing define "#weak_define __x86_64__ 1"
> 
> Does cgcc work for you? In the future we do want to move the archetecture
> related define from cgcc into sparse by itself. For now you can set
> "sparse" as "cgcc -no-compile"

Yes that works. So to address the Debian bug I can do:

 - move sparse to /usr/lib
 - teach cgcc about the move of sparse
 - make /usr/bin/sparse call cgcc -no-compile "$@"

or is it easier to teach sparse about the architecture stuff?

Best regards
Uwe


signature.asc
Description: PGP signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Wed, Aug 30, 2017 at 1:36 PM, Uwe Kleine-König  
wrote:
>> >
>> > diff --git a/validation/backend/sum.c b/validation/backend/sum.c
>> > index 0604299..d0be8dd 100644
>> > --- a/validation/backend/sum.c
>> > +++ b/validation/backend/sum.c
>> > @@ -1,3 +1,5 @@
>> > +#define __powerpc64__
>> > +#define _CALL_ELF 2
>> >  #include 
>> >  #include 
>> >
>> >
>>
>> Yep, sparse/sparsec do not define various macros that gcc/clang define
>> by default on a given architecture. This is a known problem (that I have
>> been meaning to fix ...). The 'workaround' for the time being is to use
>> the cgcc front-end to sparse. (for example 'make CC=cgcc', or perhaps
>> 'cgcc -no-compile').
>>
>> [You didn't mention your usage - is this for a kernel build?]
>
> This problem became visible during the make check phase when creating packaged
> on the listed archs for horst[1]. You can see a build logs at
>
> 
> https://buildd.debian.org/status/fetch.php?pkg=horst=s390x=5.0-1=1503905687=0
> 
> https://buildd.debian.org/status/fetch.php?pkg=horst=ppc64el=5.0-1=1503906226=0
>
> The error message looks identical (I checked the ppc64el log) to the
> problem with backend/sum.c:
>
> sparse -g -O2 -fdebug-prefix-map=/<>=. 
> -fstack-protector-strong -Wformat -Werror=format-security -std=gnu99 -Wall 
> -Wextra -g -I. -DDO_DEBUG -I/usr/include/libnl3 *.[ch]
> /usr/include/powerpc64le-linux-gnu/gnu/stubs.h:8:12: error: unable to 
> open 'gnu/stubs-32.h'

That is very much like on x86_64 missing define "#weak_define __x86_64__ 1"

Does cgcc work for you? In the future we do want to move the archetecture
related define from cgcc into sparse by itself. For now you can set
"sparse" as "cgcc -no-compile"

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

Hello,

On Wed, Aug 30, 2017 at 05:55:00PM +0100, Ramsay Jones wrote:
> On 30/08/17 17:14, Uwe Kleine-König wrote:
> > ukleinek@plummer:~/sparse$ make check V=1
> 
> It would be easier to see the results if you _didn't_ add V=1. ;-)

noted for the next time.

> [snip]
> > Out of 287 tests, 272 passed, 15 failed (10 of them are known to fail)
> > Makefile:232: recipe for target 'check' failed
> > make: *** [check] Error 1
> > ukleinek@plummer:~/sparse$ 
> 
> The additional five failures are all in the llvm backend (sparsec),
> which you do not need to use sparse as a 'checker'.
> 
> > The problem is that some cpp symbols are not defined in sparse that are
> > expected to exist. So I can "fix" backend/sum.c with the following
> > patch:
> > 
> > diff --git a/validation/backend/sum.c b/validation/backend/sum.c
> > index 0604299..d0be8dd 100644
> > --- a/validation/backend/sum.c
> > +++ b/validation/backend/sum.c
> > @@ -1,3 +1,5 @@
> > +#define __powerpc64__
> > +#define _CALL_ELF 2
> >  #include 
> >  #include 
> >  
> > 
> 
> Yep, sparse/sparsec do not define various macros that gcc/clang define
> by default on a given architecture. This is a known problem (that I have
> been meaning to fix ...). The 'workaround' for the time being is to use
> the cgcc front-end to sparse. (for example 'make CC=cgcc', or perhaps
> 'cgcc -no-compile').
> 
> [You didn't mention your usage - is this for a kernel build?]

This problem became visible during the make check phase when creating packaged
on the listed archs for horst[1]. You can see a build logs at


https://buildd.debian.org/status/fetch.php?pkg=horst=s390x=5.0-1=1503905687=0

https://buildd.debian.org/status/fetch.php?pkg=horst=ppc64el=5.0-1=1503906226=0

The error message looks identical (I checked the ppc64el log) to the
problem with backend/sum.c:

sparse -g -O2 -fdebug-prefix-map=/<>=. 
-fstack-protector-strong -Wformat -Werror=format-security -std=gnu99 -Wall 
-Wextra -g -I. -DDO_DEBUG -I/usr/include/libnl3 *.[ch]
/usr/include/powerpc64le-linux-gnu/gnu/stubs.h:8:12: error: unable to 
open 'gnu/stubs-32.h'
Makefile:113: recipe for target 'check' failed
make[1]: *** [check] Error 1

Best regards
Uwe

[1] https://packages.debian.org/sid/horst


signature.asc
Description: PGP signature

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Ramsay Jones]

On 30/08/17 17:14, Uwe Kleine-König wrote:
> Hello,
> 
> Antoine Beaupre (on Cc:) noticed that sparse doesn't work on some not so
> common architectures like ppc32le, s390x, ppc64 and sparc64[1]. This is
> nicely catched by the testsuite, e.g.:

The only architecture, from the above list, that is not supported
by cgcc seems to be ppc32le.

>   ukleinek@plummer:~/sparse$ git rev-parse HEAD
>   958c11c35d98417eb6b948bffe2dffed14eb3320
>   ukleinek@plummer:~/sparse$ uname -a
>   Linux plummer 4.9.0-3-powerpc64le #1 SMP Debian 4.9.30-2+deb9u3 
> (2017-08-06) ppc64le GNU/Linux
>   ukleinek@plummer:~/sparse$ make check V=1

It would be easier to see the results if you _didn't_ add V=1. ;-)

[snip]
>   Out of 287 tests, 272 passed, 15 failed (10 of them are known to fail)
>   Makefile:232: recipe for target 'check' failed
>   make: *** [check] Error 1
>   ukleinek@plummer:~/sparse$ 

The additional five failures are all in the llvm backend (sparsec),
which you do not need to use sparse as a 'checker'.

> The problem is that some cpp symbols are not defined in sparse that are
> expected to exist. So I can "fix" backend/sum.c with the following
> patch:
> 
> diff --git a/validation/backend/sum.c b/validation/backend/sum.c
> index 0604299..d0be8dd 100644
> --- a/validation/backend/sum.c
> +++ b/validation/backend/sum.c
> @@ -1,3 +1,5 @@
> +#define __powerpc64__
> +#define _CALL_ELF 2
>  #include 
>  #include 
>  
> 

Yep, sparse/sparsec do not define various macros that gcc/clang define
by default on a given architecture. This is a known problem (that I have
been meaning to fix ...). The 'workaround' for the time being is to use
the cgcc front-end to sparse. (for example 'make CC=cgcc', or perhaps
'cgcc -no-compile').

[You didn't mention your usage - is this for a kernel build?]

ATB,
Ramsay Jones

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Uwe Kleine-König]

Hello,

Antoine Beaupre (on Cc:) noticed that sparse doesn't work on some not so
common architectures like ppc32le, s390x, ppc64 and sparc64[1]. This is
nicely catched by the testsuite, e.g.:

ukleinek@plummer:~/sparse$ git rev-parse HEAD
958c11c35d98417eb6b948bffe2dffed14eb3320
ukleinek@plummer:~/sparse$ uname -a
Linux plummer 4.9.0-3-powerpc64le #1 SMP Debian 4.9.30-2+deb9u3 
(2017-08-06) ppc64le GNU/Linux
ukleinek@plummer:~/sparse$ make check V=1
cd validation && ./test-suite
 TESTWoverride-init-def (Woverride-init-def.c)
Using command   : ../sparse Woverride-init-def.c
Expecting exit value: 0
 TESTWoverride-init-no (Woverride-init-no.c)
Using command   : ../sparse -Wno-override-init 
Woverride-init-no.c
Expecting exit value: 0
 TESTWoverride-init-yes (Woverride-init-yes.c)
Using command   : ../sparse -Woverride-init 
Woverride-init-yes.c
Expecting exit value: 0
 TESTwarn-unknown-attribute (Wunknown-attribute-def.c)
Using command   : ../sparse Wunknown-attribute-def.c
Expecting exit value: 0
 TESTwarn-unknown-attribute-no (Wunknown-attribute-no.c)
Using command   : ../sparse -Wno-unknown-attribute 
Wunknown-attribute-no.c
Expecting exit value: 0
 TESTwarn-unknown-attribute-yes (Wunknown-attribute-yes.c)
Using command   : ../sparse -Wunknown-attribute 
Wunknown-attribute-yes.c
Expecting exit value: 0
 TEST__func__ (__func__.c)
Using command   : ../sparse -Wall __func__.c
Expecting exit value: 0
 TESTabstract array declarator static 
(abstract-array-declarator-static.c)
Using command   : ../sparse 
abstract-array-declarator-static.c
Expecting exit value: 0
 TESTaddress_space attribute (address_space.c)
Using command   : ../sparse address_space.c
Expecting exit value: 0
 TESTalias distinct symbols (alias-distinct.c)
Using command   : ../test-linearize alias-distinct.c
Expecting exit value: 0
 TESTalias symbol/pointer (alias-mixed.c)
Using command   : ../test-linearize alias-mixed.c
Expecting exit value: 0
 TESTalias same symbols (alias-same.c)
Using command   : ../test-linearize alias-same.c
Expecting exit value: 0
 TESTattribute __alloc_align__ (alloc-align.c)
Using command   : ../sparse alloc-align.c
Expecting exit value: 0
 TESTalternate keywords (alternate-keywords.c)
Using command   : ../sparse alternate-keywords.c
Expecting exit value: 0
 TESTtest anonymous union initializer (anon-union.c)
Using command   : ../sparse anon-union.c
Expecting exit value: 0
 TESTAsm with goto labels. (asm-empty-clobber.c)
Using command   : ../sparse asm-empty-clobber.c
Expecting exit value: 0
 TESTAsm with goto labels. (asm-goto-lables.c)
Using command   : ../sparse asm-goto-lables.c
Expecting exit value: 0
 TESTasm-toplevel.c (asm-toplevel.c)
Using command   : ../test-linearize asm-toplevel.c
Expecting exit value: 0
 TESTinline attributes (attr-inline.c)
Using command   : ../sparse attr-inline.c
Expecting exit value: 0
 TESTattribute no_sanitize_address (attr-no_sanitize_address.c)
Using command   : ../sparse attr-no_sanitize_address.c
Expecting exit value: 0
 TESTattribute noclone (attr-noclone.c)
Using command   : ../sparse attr-noclone.c
Expecting exit value: 0
 TESToptimize attributes (attr-optimize.c)
Using command   : ../sparse attr-optimize.c
Expecting exit value: 0
 TESTattribute warning (attr-warning.c)
Using command   : ../sparse attr-warning.c
Expecting exit value: 0
 TESTattribute assume_aligned (attr_aligned.c)
Using command   : ../sparse attr_aligned.c
Expecting exit value: 0
 TESTattribute after ( in direct-declarator 
(attr_in_parameter.c)
Using command   : ../sparse attr_in_parameter.c
Expecting exit value: 0
 TESTattribute vector_size (attr_vector_size.c)