Bug#793641: glibc: too few static TLS slots

2015-11-24 Thread Carlos O'Donell 
On Mon, Nov 23, 2015 at 7:04 AM, Aurelien Jarno  wrote:
> On 2015-08-17 15:04, Carlos O'Donell wrote:
>> On Sun, Aug 16, 2015 at 12:48 PM, Andreas Cadhalpun
>>  wrote:
>> >> until there's a better tested and working way to transition
>> >> to ffmpeg?
>> >
>> > This really doesn't have that much to do with the transition to ffmpeg.
>> > Any other library that (indirectly) links against sufficiently many
>> > STATIC_TLS using ones and is used in some plugin is also affected.
>> >
>> > Note that there are currently 14 STATIC_TLS slots and gst-libav uses 6,
>> > while it used 4 previously. It is a mere coincidence that this increase
>> > causes totem to hit the arbitrary limit in glibc.
>>
>> The math can't be done that easily, it's actually a heuristic, but
>> pretend it works this way for the sake of this discussion.
>>
>> The only immediate solution is for debian's glibc to increase
>> DTV_SURPLUS to 32 or higher. This is exactly what I did for Fedora.
>>
>> The other alternative is to backport Alex's fixes for Sourceware bugs
>> BZ #17090, BZ #17620, BZ #17621, BZ #17628, which corrects the
>> heuristics behind DTV_SURPLUS which prevent the loading of STATIC_TLS.
>
> I have backported these patches to 2.21. Unfortunately they now cause
> nptl/tst-cancel24-static to fail on at least armhf, armel, mips and
> mipsel. The binary segfaults.

Please submit a bug for this.

Is it segfaulting when accessing locales for the first time in the
newly dlopen'd libc.so.6?

There are several known issues regarding statically linked and
dlopened binaries that may impact this.

> According to the wiki [1], the failure seems to be a known issue
> in the 2.22 release, though its cause was marked as unknown. Do you know
> by chance if the problem has been fixed or addressed in the meantime?

No, nobody has fixed this upstream.

Any triage you can provide would be appreciated.

Cheers,
Carlos.

Bug#793641: glibc: too few static TLS slots

2015-11-23 Thread Aurelien Jarno 
On 2015-08-17 15:04, Carlos O'Donell wrote:
> On Sun, Aug 16, 2015 at 12:48 PM, Andreas Cadhalpun
>  wrote:
> >> until there's a better tested and working way to transition
> >> to ffmpeg?
> >
> > This really doesn't have that much to do with the transition to ffmpeg.
> > Any other library that (indirectly) links against sufficiently many
> > STATIC_TLS using ones and is used in some plugin is also affected.
> >
> > Note that there are currently 14 STATIC_TLS slots and gst-libav uses 6,
> > while it used 4 previously. It is a mere coincidence that this increase
> > causes totem to hit the arbitrary limit in glibc.
> 
> The math can't be done that easily, it's actually a heuristic, but
> pretend it works this way for the sake of this discussion.
> 
> The only immediate solution is for debian's glibc to increase
> DTV_SURPLUS to 32 or higher. This is exactly what I did for Fedora.
> 
> The other alternative is to backport Alex's fixes for Sourceware bugs
> BZ #17090, BZ #17620, BZ #17621, BZ #17628, which corrects the
> heuristics behind DTV_SURPLUS which prevent the loading of STATIC_TLS.

I have backported these patches to 2.21. Unfortunately they now cause
nptl/tst-cancel24-static to fail on at least armhf, armel, mips and
mipsel. The binary segfaults.

According to the wiki [1], the failure seems to be a known issue
in the 2.22 release, though its cause was marked as unknown. Do you know
by chance if the problem has been fixed or addressed in the meantime?

Thanks,
Aurelien

[1] https://sourceware.org/glibc/wiki/Release/2.22

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Bug#793641: glibc: too few static TLS slots

2015-11-23 Thread Aurelien Jarno 
On 2015-11-23 13:04, Aurelien Jarno wrote:
> On 2015-08-17 15:04, Carlos O'Donell wrote:
> > On Sun, Aug 16, 2015 at 12:48 PM, Andreas Cadhalpun
> >  wrote:
> > >> until there's a better tested and working way to transition
> > >> to ffmpeg?
> > >
> > > This really doesn't have that much to do with the transition to ffmpeg.
> > > Any other library that (indirectly) links against sufficiently many
> > > STATIC_TLS using ones and is used in some plugin is also affected.
> > >
> > > Note that there are currently 14 STATIC_TLS slots and gst-libav uses 6,
> > > while it used 4 previously. It is a mere coincidence that this increase
> > > causes totem to hit the arbitrary limit in glibc.
> > 
> > The math can't be done that easily, it's actually a heuristic, but
> > pretend it works this way for the sake of this discussion.
> > 
> > The only immediate solution is for debian's glibc to increase
> > DTV_SURPLUS to 32 or higher. This is exactly what I did for Fedora.
> > 
> > The other alternative is to backport Alex's fixes for Sourceware bugs
> > BZ #17090, BZ #17620, BZ #17621, BZ #17628, which corrects the
> > heuristics behind DTV_SURPLUS which prevent the loading of STATIC_TLS.
> 
> I have backported these patches to 2.21. Unfortunately they now cause
> nptl/tst-cancel24-static to fail on at least armhf, armel, mips and
> mipsel. The binary segfaults.
> 
> According to the wiki [1], the failure seems to be a known issue
> in the 2.22 release, though its cause was marked as unknown. Do you know
> by chance if the problem has been fixed or addressed in the meantime?

I have just tried to build master and the problem is also there, so the
answer is likely no.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Bug#793641: glibc: too few static TLS slots

2015-08-17 Thread Carlos O'Donell 
On Sun, Aug 16, 2015 at 12:48 PM, Andreas Cadhalpun
andreas.cadhal...@googlemail.com wrote:
 until there's a better tested and working way to transition
 to ffmpeg?

 This really doesn't have that much to do with the transition to ffmpeg.
 Any other library that (indirectly) links against sufficiently many
 STATIC_TLS using ones and is used in some plugin is also affected.

 Note that there are currently 14 STATIC_TLS slots and gst-libav uses 6,
 while it used 4 previously. It is a mere coincidence that this increase
 causes totem to hit the arbitrary limit in glibc.

The math can't be done that easily, it's actually a heuristic, but
pretend it works this way for the sake of this discussion.

The only immediate solution is for debian's glibc to increase
DTV_SURPLUS to 32 or higher. This is exactly what I did for Fedora.

The other alternative is to backport Alex's fixes for Sourceware bugs
BZ #17090, BZ #17620, BZ #17621, BZ #17628, which corrects the
heuristics behind DTV_SURPLUS which prevent the loading of STATIC_TLS.

Without Alex's fixes the implementation is limited, by a heuristic, to
a limited number of libraries with STATIC_TLS, and after Alex's fixes
(available in glibc 2.22) the limit is You may load as many
STATIC_TLS libraries as long as you have static image space for all of
them (which is a much higher arbitrary limit).

Cheers,
Carlos.

Bug#793641: glibc: too few static TLS slots

2015-08-17 Thread Andreas Cadhalpun 
Control: tag -1 fixed-upstream

Hi Carlos,

On 17.08.2015 21:04, Carlos O'Donell wrote:
 On Sun, Aug 16, 2015 at 12:48 PM, Andreas Cadhalpun
 andreas.cadhal...@googlemail.com wrote:
 until there's a better tested and working way to transition
 to ffmpeg?

 This really doesn't have that much to do with the transition to ffmpeg.
 Any other library that (indirectly) links against sufficiently many
 STATIC_TLS using ones and is used in some plugin is also affected.

 Note that there are currently 14 STATIC_TLS slots and gst-libav uses 6,
 while it used 4 previously. It is a mere coincidence that this increase
 causes totem to hit the arbitrary limit in glibc.
 
 The math can't be done that easily, it's actually a heuristic, but
 pretend it works this way for the sake of this discussion.

Apologies for the inaccuracies in my explanation. I'm not that familiar
with glibc internals. ;) 

 The only immediate solution is for debian's glibc to increase
 DTV_SURPLUS to 32 or higher. This is exactly what I did for Fedora.
 
 The other alternative is to backport Alex's fixes for Sourceware bugs
 BZ #17090, BZ #17620, BZ #17621, BZ #17628, which corrects the
 heuristics behind DTV_SURPLUS which prevent the loading of STATIC_TLS.
 
 Without Alex's fixes the implementation is limited, by a heuristic, to
 a limited number of libraries with STATIC_TLS, and after Alex's fixes
 (available in glibc 2.22) the limit is You may load as many
 STATIC_TLS libraries as long as you have static image space for all of
 them (which is a much higher arbitrary limit).

So this is already fixed upstream. Great!

In that case I don't mind working around that in ffmpeg until glibc 2.22
reaches sid.

Best regards,
Andreas

Bug#793641: glibc: too few static TLS slots

2015-08-16 Thread Andreas Henriksson 
Hello Andreas Cadhalpun.

Thanks for your interest in this issue.

On Sun, Jul 26, 2015 at 09:35:16PM +0200, Andreas Cadhalpun wrote:
 Control: severity 793641 important
[...]

I'd like to start out with questioning this gstreamer1.0-libav
is completely unuable now, breaking very popular programs like totem.
Severity should in my opinion be grave.

[...]
 The link [1] provided there is quite helpful.
[...]
 1: https://bugzilla.redhat.com/show_bug.cgi?id=1124987

It indeed is, and I'd like to point out comment #22:


  For the record it is not a glibc bug. It is a defect in the loaded
  libraries, they should not require TLS. Fundamentally it's a defect
  that we allowed developers to build shared libraries with static TLS
  in the first place.


https://bugzilla.redhat.com/show_bug.cgi?id=1124987#c22


No matter if this is a bug in glibc or not, ffmpeg maintainers should
take any 'temporary' measures at hand to ensure usability and not expect
changes to be done on a whim to critical system components like glibc.
This breakage has persisted way too long now. If there are no other
option, maybe it's time to start planning transitioning back to libav
until there's a better tested and working way to transition
to ffmpeg? Speaking to #debian-multimedia there doesn't seem to
be much interest in dealing with the breakage caused by the ffmpeg
transition right now.

Regards,
Andreas Henriksson

Bug#793641: glibc: too few static TLS slots

2015-08-16 Thread Andreas Cadhalpun 
Hi,

On 16.08.2015 17:12, Andreas Henriksson wrote:
 Thanks for your interest in this issue.

You're welcome.

 On Sun, Jul 26, 2015 at 09:35:16PM +0200, Andreas Cadhalpun wrote:
 Control: severity 793641 important
 [...]
 
 I'd like to start out with questioning this gstreamer1.0-libav
 is completely unuable now, breaking very popular programs like totem.
 Severity should in my opinion be grave.

The severity had been normal originally so I increased it.
I didn't chose grave, because this bug doesn't make glibc unusable.
You can adjust the severity, if you think that is appropriate,
but arguing about bug severity is usually not helpful.

 [...]
 The link [1] provided there is quite helpful.
 [...]
 1: https://bugzilla.redhat.com/show_bug.cgi?id=1124987
 
 It indeed is, and I'd like to point out comment #22:
 
 
   For the record it is not a glibc bug. It is a defect in the loaded
   libraries, they should not require TLS. Fundamentally it's a defect
   that we allowed developers to build shared libraries with static TLS
   in the first place.
 
 
 https://bugzilla.redhat.com/show_bug.cgi?id=1124987#c22

Then let me also point out form comment #15:
 108  /lib64/libgomp.so.1
- OK. May use STATIC_TLS, it's part of the implementation.
  Language runtime support for gomp shared across all programs.
 0/lib64/libcrypt.so.1
 0 /lib64/libm.so.6
 0 /lib64/libnsl.so.1
 0 /lib64/libnss_files.so.2
 0 /lib64/libpthread.so.0
 0 /lib64/libresolv.so.2
 0 /lib64/librt.so.1
 0 /lib64/libutil.so.1

- OK, all part of the implementation (glibc).

So it's not a bug in libgomp and libresolv and hence can only be fixed
in glibc.

 No matter if this is a bug in glibc or not, ffmpeg maintainers should
 take any 'temporary' measures at hand to ensure usability and not expect
 changes to be done on a whim to critical system components like glibc.

I had hoped that one of the glibc maintainers would comment on this bug,
before we resort to workarounds.

 This breakage has persisted way too long now.

However, surprisingly few people complained about it...

 If there are no other option,

As I wrote the other option would be to work around this bug by not linking
ffmpeg against some external libraries.
Can you confirm that removing --enable-libsoxr from ffmpeg's debian/rules
avoids this problem?
If not, also removing --enable-libssh should work.

 maybe it's time to start planning transitioning back to libav

No.

 until there's a better tested and working way to transition
 to ffmpeg?

This really doesn't have that much to do with the transition to ffmpeg.
Any other library that (indirectly) links against sufficiently many
STATIC_TLS using ones and is used in some plugin is also affected.

Note that there are currently 14 STATIC_TLS slots and gst-libav uses 6,
while it used 4 previously. It is a mere coincidence that this increase
causes totem to hit the arbitrary limit in glibc.

 Speaking to #debian-multimedia there doesn't seem to
 be much interest in dealing with the breakage caused by the ffmpeg
 transition right now.

Since the bug is in glibc, only the glibc maintainers can fix it.

Best regards,
Andreas

Bug#793641: glibc: too few static TLS slots

2015-07-26 Thread Andreas Cadhalpun 
Control: severity 793641 important
Control: reassign 793689 libc6 2.19-19
Control: reassign 793641 libc6 2.19-19
Control: forcemerge 793641 793689
Control: retitle 793641 glibc: too few static TLS slots
Control: tags 793641 patch

Hi,

On 26.07.2015 17:42, Tom Rauchenwald wrote:
 I can't play mp4 files using totem anymore, only a black screen is displayed.

This looks similar to bug #793641 for vlc thus I'm merging the two.
The link [1] provided there is quite helpful.
The problem is that ffmpeg links against more libraries than libav and some
of those use static TLS.
Using the method from the redhat bug to determine which libraries use TLS 
reveals:
$ for lib in $(ldd /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstlibav.so | 
grep = | sed 's/.*= \([/a-z0-9\._+-]*\) .*/\1/g' | sort); do echo readelf 
-d -W $lib; readelf -d -W $lib; done | egrep 'readelf|STATIC_TLS' | grep -B1 
TLS
readelf -d -W /lib/x86_64-linux-gnu/libc.so.6
 0x001e (FLAGS)  STATIC_TLS
--
readelf -d -W /lib/x86_64-linux-gnu/libm.so.6
 0x001e (FLAGS)  STATIC_TLS
--
readelf -d -W /lib/x86_64-linux-gnu/libpthread.so.0
 0x001e (FLAGS)  STATIC_TLS
readelf -d -W /lib/x86_64-linux-gnu/libresolv.so.2
 0x001e (FLAGS)  STATIC_TLS
readelf -d -W /lib/x86_64-linux-gnu/librt.so.1
 0x001e (FLAGS)  STATIC_TLS
--
readelf -d -W /usr/lib/x86_64-linux-gnu/libgomp.so.1
 0x001e (FLAGS)  STATIC_TLS


Previously that was only:
readelf -d -W /lib/x86_64-linux-gnu/libc.so.6
 0x001e (FLAGS)  STATIC_TLS
--
readelf -d -W /lib/x86_64-linux-gnu/libm.so.6
 0x001e (FLAGS)  STATIC_TLS
--
readelf -d -W /lib/x86_64-linux-gnu/libpthread.so.0
 0x001e (FLAGS)  STATIC_TLS
readelf -d -W /lib/x86_64-linux-gnu/librt.so.1
 0x001e (FLAGS)  STATIC_TLS

Thus it can happen that when totem/vlc tries to load the plugin not enough free
static TLS slots remain.

 I was told on #debian-multimedia to report the bug against ffmpeg.

The only thing ffmpeg could do is not to link against some libraries, losing
some features. I think this would be wrong.

Thus I think the only solution is to increase the DTV_SURPLUS limit in glibc.
Therefore I'm reassigning this bug there.

 I get the following message on the console:
 
 (totem:7753): GStreamer-WARNING **: Failed to load plugin 
 '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstlibav.so': dlopen: cannot load 
 any more object with static TLS
 ** Message: Missing plugin: gstreamer|1.0|totem|H.264 
 decoder|decoder-video/x-h264, stream-format=(string)avc, 
 alignment=(string)au, level=(string)3.1, profile=(string)high, 
 parsed=(boolean)true (H.264 decoder)
 ** Message: Automatic missing codec installation not supported (helper
 script missing)
 
 The file is present on my system:
 -rw-r--r-- 1 root root 236016 Jul 20 20:29
 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstlibav.so
 
 Any help would be appreciated!

This works again for me, after installing a libc6 from glibc rebuilt with
the following patch:
---8---
--- a/sysdeps/generic/ldsodefs.h
+++ b/sysdeps/generic/ldsodefs.h
@@ -393,7 +393,7 @@ struct rtld_global
 #define TLS_SLOTINFO_SURPLUS (62)
 
 /* Number of additional slots in the dtv allocated.  */
-#define DTV_SURPLUS(14)
+#define DTV_SURPLUS(64)
 
   /* Initial dtv of the main thread, not allocated with normal malloc.  */
   EXTERN void *_dl_initial_dtv;
---8---

Best regards,
Andreas


1: https://bugzilla.redhat.com/show_bug.cgi?id=1124987


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org