Re: [PATCH] Silence misguided GCC warning about alignment issues
Hi Emmanuel, On Tue, 8 Aug 2017, Emmanuel Deloget wrote: > On Tue, Aug 8, 2017 at 1:19 PM, Johannes Schindelin < > johannes.schinde...@gmx.de> wrote: > > > On Mon, 7 Aug 2017, Emmanuel Deloget wrote: > > > > > And yes, its seems that the get_le32() macro in xz_private.h is a > > > bit illegal with respect to strict aliasing, as it casts a uint8_t * > > > into a const uint32_t *. > > > > It would seem so. > > > > Until you realize that it is used only on s->temp.buf (or `s->temp.buf + 4 > > * `), and that the `buf` field was carefully placed after two > > fields of type `size_t` (i.e. naturally aligned *at least* to a 32-bit > > boundary). > > > > If you are worried about that magic, you can even mark the `buf` field > > using ALIGN4. But that *still* does not fix GCC's compiler warning. What > > fixes it is my workaround of defining a static ALWAYS_INLINE function. > > > > > No, I'm not worried by the magic :) > > Yet the problem is not the aliasing per see, it's the way the compiler > handles it and what it means to him. Please do not refer to the compiler as a "he". You are a "he", I assume, and I am a "he" (I am certain about the latter). But the compiler is an "it". Writing "he" for the compiler makes me stumble every single time, which means I had a much harder time understanding your otherwise excellent explanation. I was under the impression that the C standard explicitly forbade the duplication/reordering you mentioned when it comes to struct fields. But maybe I was wrong on that. Having been educated about this, I agree that the easier fix is to simply drop the optimization, even if it is a pity to give up on performance. Ciao, Johannes___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
Johannes Schindelin wrote: When compiling xz_dec_stream.c with GCC 7.1.0, it complains thusly: In function 'dec_stream_footer': error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing] if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf)) ^~ The thing is, the `buf` field was put just after two fields of type size_t for the express purpose of avoiding alignment issues, as per the comment above the `temp` struct. +#if defined(__GNUC__) + /* +* The temp.buf field is put just after two fields of type size_t for +* the express purpose of avoiding alignment issues. But GCC complains +* about it nevertheless... so: shut GCC up for a few lines. +*/ +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wstrict-aliasing" +#endif In this discussion, you write repeatedly about the alignment of the variable, you even propose a comment to that effect. Note, this is not about the alignment, it is about aliasing, which means two different pointers may access the same memory location. ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
Hi Johannes On Tue, Aug 8, 2017 at 1:19 PM, Johannes Schindelin < johannes.schinde...@gmx.de> wrote: > Hi Emmanuel, > > On Mon, 7 Aug 2017, Emmanuel Deloget wrote: > > > And yes, its seems that the get_le32() macro in xz_private.h is a bit > > illegal with respect to strict aliasing, as it casts a uint8_t * into a > > const uint32_t *. > > It would seem so. > > Until you realize that it is used only on s->temp.buf (or `s->temp.buf + 4 > * `), and that the `buf` field was carefully placed after two > fields of type `size_t` (i.e. naturally aligned *at least* to a 32-bit > boundary). > > If you are worried about that magic, you can even mark the `buf` field > using ALIGN4. But that *still* does not fix GCC's compiler warning. What > fixes it is my workaround of defining a static ALWAYS_INLINE function. > > No, I'm not worried by the magic :) Yet the problem is not the aliasing per see, it's the way the compiler handles it and what it means to him. Strict aliasing has been defined as a way to allow the compiler to do some clever code optimization (mostly reordering). If the compiler knows that two memory areas are distinct, there is no R/W dependency on these areas and it can reorder the code as he wants. But when aliasing comes into play, such optimizations should be forbidden since an undetected R/W dependency might exists. Yet, since the -fstrict-aliasing is given to him, we explicitely tell the compiler to force strict aliasing rules and the compiler still apply them -- ditching any unproved R/W dependency in the process. Needless to say, this can be the source of hard-to-catch bugs :) Consider this code: uint32_t k = 0; uint8_t *p = (uint8_t *) k = 0x; if (*p == 0xff) do_some_magic_stuff(); Without strict aliasing, the compiler must order the code so that *p is indeed 0xff when the if condition is evaluated because it cannot be sure that there is a R/W dependency between k and p. With forced strict aliasing, and p are assumed to not point to the same memory area, thus there cannot be any R/W dependency, so changing k is not supposed to have any effect on the *p test. As a consequence, the compiler is free to reorder the code and for exemple do the following: if (*p = 0xff) do_some_magic_stuff(); k = 0x; Which, of course, will not work as expected. Thus, this aliasing problem may have other unintended consequences. Silencing the warning will hide a possible bug, and this is not a good thing. It would be far better to fix the aliasing issue correctly. Similarly, the fact that ALWAY_INLINE seems to hide the warning looks more like a GCC bug to me than the expected behavior, as ALWAYS_INLINE does not change the way the compiler plays with the aliasing rules. Something along the line of (not tested): #define get_le32(p) \ ({ \ union { const uint32_t *x32; const uint8_t *x8; } v = { .x8 = (p), }; \ le32_to_cpup(v.x32); \ }) (for the occurence at xz_private.h@24) Maybe it's even better to create an aliastype macro looking like: #define aliasvalue(type, value) \ ({ \ union { \ type *aliased; \ typeof(value) *original; \ } v = { \ .original = &(value), \ }; \ v.aliased; \ }) And then: #define get_le32(p) le32_to_cpup(aliasvalue(uint32_t, (p))) Or #define get_le32(p) (*(aliasvalue(uint32_t,(p I'm not even sure the compiler is not able to fully optimize out this kind of cod since it will look like a move from one register to another register to him (my own tests seems to show that the compiler emits optimized code in this situation). The aliasvalue() macro above is probably not perfect (it might run ok when p in a scalar or an array of scalar but will break if p is a pointer; not to mention its horrible name...) so more though might be needed to come up with a correct version. > Denys' fix works around the problem, alright, but it also makes the code > slower. And it is *not* necessary to make the code slower. > > Ciao, > Johannes > Best regards, -- Emmanuel Deloget ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
Hi Emmanuel, On Mon, 7 Aug 2017, Emmanuel Deloget wrote: > And yes, its seems that the get_le32() macro in xz_private.h is a bit > illegal with respect to strict aliasing, as it casts a uint8_t * into a > const uint32_t *. It would seem so. Until you realize that it is used only on s->temp.buf (or `s->temp.buf + 4 * `), and that the `buf` field was carefully placed after two fields of type `size_t` (i.e. naturally aligned *at least* to a 32-bit boundary). If you are worried about that magic, you can even mark the `buf` field using ALIGN4. But that *still* does not fix GCC's compiler warning. What fixes it is my workaround of defining a static ALWAYS_INLINE function. Denys' fix works around the problem, alright, but it also makes the code slower. And it is *not* necessary to make the code slower. Ciao, Johannes ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
On Mon, Aug 7, 2017 at 10:37 PM, Emmanuel Delogetwrote: > > > > On Mon, Aug 7, 2017 at 9:46 PM, Johannes Schindelin > wrote: >> >> Hi Denys, >> >> On Mon, 7 Aug 2017, Denys Vlasenko wrote: >> >> > On Mon, Aug 7, 2017 at 12:09 PM, Johannes Schindelin >> > wrote: >> > > When compiling xz_dec_stream.c with GCC 7.1.0, it complains thusly: >> > > >> > > In function 'dec_stream_footer': >> > > error: dereferencing type-punned pointer will break >> > > strict-aliasing >> > > rules [-Werror=strict-aliasing] >> > >if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf)) >> > >^~ >> > > >> > > The thing is, the `buf` field was put just after two fields of type >> > > size_t for the express purpose of avoiding alignment issues, as per the >> > > comment above the `temp` struct. >> > > >> > > Meaning: GCC gets this all wrong and should not complain. So let's force >> > > it to be quiet for a couple of lines. >> > >> > xz_crc32 is: >> > >> > xz_crc32(const uint8_t *buf, size_t size, uint32_t crc) >> > >> > I think gcc is not complaining about xz_crc32, >> > it complains about get_le32! >> > >> > Can you test this theory by splitting that statement in two? >> >> I, too, thought at first that it is clearly about get_le32(), but GCC was >> really adamant that it was the xz_crc32() call. >> >> And a little further investigation suggests that GCC really meant >> xz_crc32(), which is defined as a static function in >> archival/libarchive/decompress_unxz.c that gets inlined by GCC because it >> simply hands off to crc32_block_endian0() which in turn takes an >> `uint32_t *` as first parameter. And that's where the type-punning is >> broken. >> > > This would be quite weird - the static function is by definition local to the > compilation unit, and the compiler cannot know about the content of > decompress_unxz.c from xz_dec_stream.c. My bad. It's the other way round -- sorry for the noise. decompress_unxz.c includes xz_dec_stream.c (sometimes, the Busybox code is a bit hard to follow :)) And yes, its seems that the get_le32() macro in xz_private.h is a bit illegal with respect to strict aliasing, as it casts a uint8_t * into a const uint32_t *. > > BR, > > -- Emmanuel Deloget > ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
Hi Emmanuel, On Mon, 7 Aug 2017, Emmanuel Deloget wrote: > On Mon, Aug 7, 2017 at 12:25 PM, Lauri Kasanenwrote: > > On Mon, 7 Aug 2017 12:09:45 +0200 (CEST) > > Johannes Schindelin wrote: > >> Meaning: GCC gets this all wrong and should not complain. So let's force > >> it to be quiet for a couple of lines. > > > > I believe this is both wrong and dangerous. If gcc warns about it, it > > will also very likely try to mis-optimize the same case. > > > > - Lauri > > I tried to carefully read the code, and I don't see where the aliasing > issue migh be. > * buf is an uin8_t [] > * the code either use it as is, or promote it to const uint8_t *, or > cast it to (const) char *. > > All of these types are compatible w.r.t. aliasing (and the C99 > standard). I may have missed something though and my eyes might be > rusted. > > Now, it seems that some might have seen a regression in GCC with > respect to aliasing rules [1][2]. This might be an instance of the bug > (according to the bug report, it's no longer reproducible in GCC > trunk). I dug a little deeper, and it seems this issue may indeed have been reported as https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80593 (and been fixed in the meantime). However... > Anyway, it might be a good idea to avoid unsetting the warning for one > buggy GCC version :) ... I get this warning *also* with the default GCC version of Ubuntu 16.04.3 LTS, gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609. So it is not just one buggy GCC version. I do agree, however, that it is a super-ugly patch, even if it does the job for the described purpose. So I set out to find a more elegant solution. More elegant, even, than removing the get_le32() macro as Denys did in 76b65624b (unxz: get_le32 macro is obviously wrong, 2017-08-07): the macro was there for a reason, namely to avoid accessing the (purposefully aligned) data using get_unaligned_le32() which is slow and unlikely to get optimized. It turns out that GCC simply has a hard time with the cast, without assigning the pointer explicitly to a variable. Turning get_le32() into an inline function lets GCC figure out that everything is groovy, and it also adds a bit more of concrete compile-time safety. I will send out v2 in a moment, assuming that Denys will agree that direct, provably aligned 32-bit access is better than an unnecessary assembly-from-individual-bytes. Ciao, Johannes ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
On Mon, Aug 7, 2017 at 9:46 PM, Johannes Schindelin < johannes.schinde...@gmx.de> wrote: > Hi Denys, > > On Mon, 7 Aug 2017, Denys Vlasenko wrote: > > > On Mon, Aug 7, 2017 at 12:09 PM, Johannes Schindelin > >wrote: > > > When compiling xz_dec_stream.c with GCC 7.1.0, it complains thusly: > > > > > > In function 'dec_stream_footer': > > > error: dereferencing type-punned pointer will break > strict-aliasing > > > rules [-Werror=strict-aliasing] > > >if (xz_crc32(s->temp.buf + 4, 6, 0) != > get_le32(s->temp.buf)) > > >^~ > > > > > > The thing is, the `buf` field was put just after two fields of type > > > size_t for the express purpose of avoiding alignment issues, as per the > > > comment above the `temp` struct. > > > > > > Meaning: GCC gets this all wrong and should not complain. So let's > force > > > it to be quiet for a couple of lines. > > > > xz_crc32 is: > > > > xz_crc32(const uint8_t *buf, size_t size, uint32_t crc) > > > > I think gcc is not complaining about xz_crc32, > > it complains about get_le32! > > > > Can you test this theory by splitting that statement in two? > > I, too, thought at first that it is clearly about get_le32(), but GCC was > really adamant that it was the xz_crc32() call. > > And a little further investigation suggests that GCC really meant > xz_crc32(), which is defined as a static function in > archival/libarchive/decompress_unxz.c that gets inlined by GCC because it > simply hands off to crc32_block_endian0() which in turn takes an > `uint32_t *` as first parameter. And that's where the type-punning is > broken. > > This would be quite weird - the static function is by definition local to the compilation unit, and the compiler cannot know about the content of decompress_unxz.c from xz_dec_stream.c. BR, -- Emmanuel Deloget ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
Hi, On Mon, 7 Aug 2017, Johannes Schindelin wrote: > On Mon, 7 Aug 2017, Denys Vlasenko wrote: > > > On Mon, Aug 7, 2017 at 12:09 PM, Johannes Schindelin > >wrote: > > > When compiling xz_dec_stream.c with GCC 7.1.0, it complains thusly: > > > > > > In function 'dec_stream_footer': > > > error: dereferencing type-punned pointer will break > > > strict-aliasing > > > rules [-Werror=strict-aliasing] > > >if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf)) > > >^~ > > > > > > The thing is, the `buf` field was put just after two fields of type > > > size_t for the express purpose of avoiding alignment issues, as per the > > > comment above the `temp` struct. > > > > > > Meaning: GCC gets this all wrong and should not complain. So let's force > > > it to be quiet for a couple of lines. > > > > xz_crc32 is: > > > > xz_crc32(const uint8_t *buf, size_t size, uint32_t crc) > > > > I think gcc is not complaining about xz_crc32, > > it complains about get_le32! > > > > Can you test this theory by splitting that statement in two? > > I, too, thought at first that it is clearly about get_le32(), but GCC was > really adamant that it was the xz_crc32() call. > > And a little further investigation suggests that GCC really meant > xz_crc32(), which is defined as a static function in > archival/libarchive/decompress_unxz.c that gets inlined by GCC because it > simply hands off to crc32_block_endian0() which in turn takes an > `uint32_t *` as first parameter. And that's where the type-punning is > broken. Okay, nevermind, I misread the xz_crc32() function. The first parameter passed to crc32_block_endian0() is not the first parameter passed to xz_crc32(). And splitting the statement in two does indeed suggest that get_le32() is the actual cause for that warning. Ciao, Johannes ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
Hi Denys, On Mon, 7 Aug 2017, Denys Vlasenko wrote: > On Mon, Aug 7, 2017 at 12:09 PM, Johannes Schindelin >wrote: > > When compiling xz_dec_stream.c with GCC 7.1.0, it complains thusly: > > > > In function 'dec_stream_footer': > > error: dereferencing type-punned pointer will break strict-aliasing > > rules [-Werror=strict-aliasing] > >if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf)) > >^~ > > > > The thing is, the `buf` field was put just after two fields of type > > size_t for the express purpose of avoiding alignment issues, as per the > > comment above the `temp` struct. > > > > Meaning: GCC gets this all wrong and should not complain. So let's force > > it to be quiet for a couple of lines. > > xz_crc32 is: > > xz_crc32(const uint8_t *buf, size_t size, uint32_t crc) > > I think gcc is not complaining about xz_crc32, > it complains about get_le32! > > Can you test this theory by splitting that statement in two? I, too, thought at first that it is clearly about get_le32(), but GCC was really adamant that it was the xz_crc32() call. And a little further investigation suggests that GCC really meant xz_crc32(), which is defined as a static function in archival/libarchive/decompress_unxz.c that gets inlined by GCC because it simply hands off to crc32_block_endian0() which in turn takes an `uint32_t *` as first parameter. And that's where the type-punning is broken. Ciao, Johannes ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
On Mon, Aug 7, 2017 at 12:09 PM, Johannes Schindelinwrote: > When compiling xz_dec_stream.c with GCC 7.1.0, it complains thusly: > > In function 'dec_stream_footer': > error: dereferencing type-punned pointer will break strict-aliasing > rules [-Werror=strict-aliasing] >if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf)) >^~ > > The thing is, the `buf` field was put just after two fields of type > size_t for the express purpose of avoiding alignment issues, as per the > comment above the `temp` struct. > > Meaning: GCC gets this all wrong and should not complain. So let's force > it to be quiet for a couple of lines. xz_crc32 is: xz_crc32(const uint8_t *buf, size_t size, uint32_t crc) I think gcc is not complaining about xz_crc32, it complains about get_le32! Can you test this theory by splitting that statement in two? ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
Hello, On Mon, Aug 7, 2017 at 12:25 PM, Lauri Kasanenwrote: > On Mon, 7 Aug 2017 12:09:45 +0200 (CEST) > Johannes Schindelin wrote: >> Meaning: GCC gets this all wrong and should not complain. So let's force >> it to be quiet for a couple of lines. > > I believe this is both wrong and dangerous. If gcc warns about it, it > will also very likely try to mis-optimize the same case. > > - Lauri I tried to carefully read the code, and I don't see where the aliasing issue migh be. * buf is an uin8_t [] * the code either use it as is, or promote it to const uint8_t *, or cast it to (const) char *. All of these types are compatible w.r.t. aliasing (and the C99 standard). I may have missed something though and my eyes might be rusted. Now, it seems that some might have seen a regression in GCC with respect to aliasing rules [1][2]. This might be an instance of the bug (according to the bug report, it's no longer reproducible in GCC trunk). Anyway, it might be a good idea to avoid unsetting the warning for one buggy GCC version :) Best regards, -- Emmanuel Deloget [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80593 [2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80633 ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: [PATCH] Silence misguided GCC warning about alignment issues
On Mon, 7 Aug 2017 12:09:45 +0200 (CEST) Johannes Schindelinwrote: > Meaning: GCC gets this all wrong and should not complain. So let's force > it to be quiet for a couple of lines. I believe this is both wrong and dangerous. If gcc warns about it, it will also very likely try to mis-optimize the same case. - Lauri ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
[PATCH] Silence misguided GCC warning about alignment issues
When compiling xz_dec_stream.c with GCC 7.1.0, it complains thusly: In function 'dec_stream_footer': error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing] if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf)) ^~ The thing is, the `buf` field was put just after two fields of type size_t for the express purpose of avoiding alignment issues, as per the comment above the `temp` struct. Meaning: GCC gets this all wrong and should not complain. So let's force it to be quiet for a couple of lines. Signed-off-by: Johannes Schindelin--- archival/libarchive/unxz/xz_dec_stream.c | 12 1 file changed, 12 insertions(+) diff --git a/archival/libarchive/unxz/xz_dec_stream.c b/archival/libarchive/unxz/xz_dec_stream.c index bf791055b..8131ee30a 100644 --- a/archival/libarchive/unxz/xz_dec_stream.c +++ b/archival/libarchive/unxz/xz_dec_stream.c @@ -423,6 +423,15 @@ static enum xz_ret XZ_FUNC dec_stream_footer(struct xz_dec *s) if (!memeq(s->temp.buf + 10, FOOTER_MAGIC, FOOTER_MAGIC_SIZE)) return XZ_DATA_ERROR; +#if defined(__GNUC__) + /* +* The temp.buf field is put just after two fields of type size_t for +* the express purpose of avoiding alignment issues. But GCC complains +* about it nevertheless... so: shut GCC up for a few lines. +*/ +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wstrict-aliasing" +#endif if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf)) return XZ_DATA_ERROR; @@ -433,6 +442,9 @@ static enum xz_ret XZ_FUNC dec_stream_footer(struct xz_dec *s) */ if ((s->index.size >> 2) != get_le32(s->temp.buf + 4)) return XZ_DATA_ERROR; +#if defined(__GNUC__) +#pragma GCC diagnostic pop +#endif if (s->temp.buf[8] != 0 || s->temp.buf[9] != s->check_type) return XZ_DATA_ERROR; base-commit: 4dea1edd08a45c5987448719e56ee61a20fb9210 -- 2.14.0.windows.1.2.g0f3342804fc Published-As: https://github.com/dscho/busybox-w32/releases/tag/busybox-type-punned-warning-v1 Fetch-It-Via: git fetch https://github.com/dscho/busybox-w32 busybox-type-punned-warning-v1 ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox