Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Santiago Vila]

On Wed, Aug 24, 2016 at 11:43:54PM +0200, Bálint Réczey wrote:

> I don't like having this bug, but if a program fails to build 50% of
> the time but runs fine it can be released IMO since users are not
> affected.

You seem to imply that our "output" as package distributors is just
the set of binary packages. That's not the case. We provide source
packages and binary packages. If a user can't build a source
package that we provide then he/she is affected as well, because
taking the source package and building it (possibly with modifications)
is also a way of "using" it.


In either case, it is not the severity definitions what we have to
consider here, but release policy.

Release policy says "packages must autobuild".

If policy said "packages must autobuild most of the time", then yes,
we could maybe have packages which only build ok most of the time.

But that's not what release policy says, and that's why FTBFS bugs are
usually reported as serious regardless of the "frequency" of the
failure.


In practice this is not as difficult to achieve as one might think,
because it often happens that it's the tests who made the build to fail.

If a test fails very often but does not mean that the package is "bad",
the logical thing to do until somebody has the time to investigate it
is to just disable it (as you have just done with kodi, thanks a lot).

Thanks.

___
pkg-multimedia-maintainers mailing list
pkg-multimedia-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers

Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Bálint Réczey]

Hi Santiago,

2016-08-24 17:36 GMT+02:00 Santiago Vila :
> Hi.
>
> Any progress on this? This is still reproducible. I'm attaching a
> build log for the version which has just entered testing.

I have uploaded a new version to experimental with two related changes.

One is not failing the build when the test fails. This is not nice,
but the tests
were completely broken while kodi ran fine on my test system.

The second is running the tests with Valgrind when it is installed to have
better chance at finding allocation errors. I also proposed the change at
upstream listing an error which could mess up mutex handling:
https://github.com/xbmc/xbmc/pull/10334

>
> On Sun, 17 Apr 2016, Balint Reczey wrote:
>
>> It suggests there is something wrong with the mutex handling and TSAN seems 
>> to confirm that:
>> [...]
>> I need to test if the problem still exists in upstream master and verify if
>> it is not a false positive TSAN warning and the root cause is indeed here.
>
> Could you translate that into a language that a non-programmer
> could understand?
>
>> I'm setting the bug's severity to normal because while it is reproducible
>> the FTBFS does not happen on Debian's buildds [...]
>
> Hmm. I'm not going to enter into a severity war, but you should be
> aware that the theory that a FTBFS is not RC because "it does not
> happen" in Debian's buildds has many flaws.
>
> For example, for a package which FTBFS randomly, it is completely
> useless, as a successful build just means that we were lucky that
> time.

I'm trying to apply definitions from here:
https://www.debian.org/Bugs/Developer#severities

I don't like having this bug, but if a program fails to build 50% of
the time but runs fine it can be released IMO since users are not
affected.

>
> We should better not care too much about what happened in the
> autobuilders the last time it was tried, we should care more about
> what could happen the next time.
>
> I'm using sbuild and the autobuilders are using sbuild as well
> (or a variant of it).
>
> If you think this may not happen in official autobuilders, could you
> please explain why?

I don't think that it may not happen, but I say that it did not seem to
happen so far which makes me think it won't happen often and when
it happens autobuilders will try again anyway.

I understand that having this issue is painful for people like you doing the
hard work of rebuilding many packages and I'm trying to help.

Starting with 17.x the tests won't make the build fail until they run
reliably thus
you'll be able to rebuild kodi. Valgrind will also help upstream in catching
bugs earlier.

>
> (Because my machine does not have enough memory? How would I know that
> in advance when we don't have a Build-Memory control field?)
>
>> thus does not prevent rebuilding the package when needed.
>
> It does prevent me from rebuilding the package, which makes the
> software non-free for me and apparently anybody who does not have a
> machine with 32 GB of RAM (which is still a lot of people).

I guess you can build it with "DEB_BUILD_OPTIONS=nocheck", thus
this seems to be a bit of exaggeration.

>
> I would say that would deserve important severity at least.

I think the bug does not fit "important"'s definition [1] but feel free
to raise it.
I agree with you on this bug being valid and needs to be fixed
no matter what the severity is.

Cheers,
Balint

>
> (I said 32 GB just as an example, but if that's a more or less
> accurate summary of this bug, I'd like to know the exact figure).
>
> Thanks.


[1] "important: a bug which has a major effect on the usability of a
package, without rendering it completely unusable to everyone."

___
pkg-multimedia-maintainers mailing list
pkg-multimedia-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers

Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Santiago Vila]

Hi.

Any progress on this? This is still reproducible. I'm attaching a
build log for the version which has just entered testing.

On Sun, 17 Apr 2016, Balint Reczey wrote:

> It suggests there is something wrong with the mutex handling and TSAN seems 
> to confirm that:
> [...]
> I need to test if the problem still exists in upstream master and verify if
> it is not a false positive TSAN warning and the root cause is indeed here.

Could you translate that into a language that a non-programmer
could understand?

> I'm setting the bug's severity to normal because while it is reproducible
> the FTBFS does not happen on Debian's buildds [...]

Hmm. I'm not going to enter into a severity war, but you should be
aware that the theory that a FTBFS is not RC because "it does not
happen" in Debian's buildds has many flaws.

For example, for a package which FTBFS randomly, it is completely
useless, as a successful build just means that we were lucky that
time.

We should better not care too much about what happened in the
autobuilders the last time it was tried, we should care more about
what could happen the next time.

I'm using sbuild and the autobuilders are using sbuild as well
(or a variant of it).

If you think this may not happen in official autobuilders, could you
please explain why?

(Because my machine does not have enough memory? How would I know that
in advance when we don't have a Build-Memory control field?)

> thus does not prevent rebuilding the package when needed.

It does prevent me from rebuilding the package, which makes the
software non-free for me and apparently anybody who does not have a
machine with 32 GB of RAM (which is still a lot of people).

I would say that would deserve important severity at least.

(I said 32 GB just as an example, but if that's a more or less
accurate summary of this bug, I'd like to know the exact figure).

Thanks.

kodi_16.1+dfsg1-2_amd64-20160824T064358Z.gz
Description: application/gzip
___
pkg-multimedia-maintainers mailing list
pkg-multimedia-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers

Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Balint Reczey]

Control: severity -1 normal
Control: tags -1 upstream confirmed

Hi Santiago,

2016-04-16 18:43 GMT+02:00 Santiago Vila :
> I've put the program kodi-test here:
>
> https://people.debian.org/~sanvila/bug-820416/kodi-test.gz
>
> It was built on a stretch chroot today (for amd64).
>
> To reproduce the segfault, just install the build-dependencies for
> kodi, i.e. on a stretch chroot, please do:
>
> apt-get build-dep kodi
>
> then try to execute kodi-test:
>
> ./kodi-test
>
> This is all you need to reproduce the problem.

Thank you for the bug report and the additional information.

I have just created a chroot for reproducing the issue and even after I
set memory limits it did not crash.
I also built kodi-test myself which did not crash either.

root@stretch:/kodi-16.1~rc2+dfsg1# ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 30616
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) 2048000
open files  (-n) 65536
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 30616
virtual memory  (kbytes, -v) 4096000
file locks  (-x) unlimited

./kodi-test-dloaded
...
[--] Global test environment tear-down
[==] 574 tests from 89 test cases ran. (9800 ms total)
[  PASSED  ] 574 tests.
Clean shutdown of TestGlobalPattern1

Valgrind found a few issues in Google Test, but nothing extraordinary:
...
==21850== Use of uninitialised value of size 8
==21850==at 0xFAF6201: _itoa_word (in /lib/x86_64-linux-gnu/libc-2.22.so)
==21850==by 0xFAF998C: vfprintf (in /lib/x86_64-linux-gnu/libc-2.22.so)
==21850==by 0xFB23258: vsnprintf (in /lib/x86_64-linux-gnu/libc-2.22.so)
==21850==by 0xFB00D11: snprintf (in /lib/x86_64-linux-gnu/libc-2.22.so)
==21850==by 0x130A48A: testing::(anonymous 
namespace)::PrintByteSegmentInObjectTo(unsigned char const*, unsigned long, 
unsigned long, std::ostream*) [clone .constprop.392] (gtest-printers
.cc:72)
==21850==by 0x130E6EC: PrintBytesInObjectToImpl (gtest-printers.cc:90)
==21850==by 0x130E6EC: testing::internal2::PrintBytesInObjectTo(unsigned 
char const*, unsigned long, std::ostream*) (gtest-printers.cc:112)
==21850==by 0x134E4A2: PrintValue (gtest-printers.h:136)
==21850==by 0x134E4A2: operator<<  
(gtest-printers.h:201)
==21850==by 0x134E4A2: DefaultPrintNonContainerTo 
(gtest-printers.h:245)
==21850==by 0x134E4A2: DefaultPrintTo (gtest-printers.h:338)
==21850==by 0x134E4A2: PrintTo (gtest-printers.h:376)
==21850==by 0x134E4A2: Print (gtest-printers.h:600)
==21850==by 0x134E4A2: UniversalPrint (gtest-printers.h:756)
==21850==by 0x134E4A2: Print (gtest-printers.h:684)
==21850==by 0x134E4A2: PrintToString (gtest-printers.h:849)
==21850==by 0x134E4A2: 
testing::internal::ParameterizedTestCaseInfo::RegisterTests() 
(gtest-param-util.h:508)
==21850==by 0x130BC44: RegisterTests (gtest-param-util.h:602)
==21850==by 0x130BC44: 
testing::internal::UnitTestImpl::RegisterParameterizedTests() (gtest.cc:2290)
==21850==by 0x1322ED8: 
testing::internal::UnitTestImpl::PostFlagParsingInit() (gtest.cc:4125)
==21850==by 0x132C034: void 
testing::internal::InitGoogleTestImpl(int*, char**) (gtest.cc:4991)
==21850==by 0x8461B6: main (xbmc-test.cpp:40)
...

OTOH in a VirtualBox VM limited to 2GB of RAM i was able to
reproduce the crash and got a meaningful backtrace:
...
#0  __GI___pthread_mutex_lock (mutex=mutex@entry=0x140) at 
../nptl/pthread_mutex_lock.c:68
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = 
id = 
#1  0x014b08b4 in XbmcThreads::pthreads::RecursiveMutex::lock 
(this=0x140) at 
/home/vagrant/kodi-16.1~rc2+dfsg1/xbmc/threads/platform/pthreads/CriticalSection.h:49
No locals.
#2  XbmcThreads::CountingLockable::lock 
(this=0x140) at /home/vagrant/kodi-16.1~rc2+dfsg1/xbmc/threads/Lockables.h:60
No locals.
#3  XbmcThreads::UniqueLock::UniqueLock (lockable=..., 
this=) at 
/home/vagrant/kodi-16.1~rc2+dfsg1/xbmc/threads/Lockables.h:127
No locals.
#4  CSingleLock::CSingleLock (cs=..., this=) at 
/home/vagrant/kodi-16.1~rc2+dfsg1/xbmc/threads/SingleLock.h:38
No locals.
#5  XbmcThreads::CEventGroup::Set (child=0x7ffc9df57090, this=0xe0) at 
Event.h:122
No locals.
#6  CEvent::Set (this=this@entry=0x7ffc9df57090) at Event.cpp:80
iter =
l = { = 
{ = {}, mutex = @0x7ffc9df57098, owns 
= true}, }
#7  0x014b2d09 in CThread::staticThread (data=0x7ffc9df56fc0) at 
Thread.cpp:137
pThread = 0x7ffc9df56fc0
name = "AlarmClock"
id = 

Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Santiago Vila]

I've put the program kodi-test here:

https://people.debian.org/~sanvila/bug-820416/kodi-test.gz

It was built on a stretch chroot today (for amd64).

To reproduce the segfault, just install the build-dependencies for
kodi, i.e. on a stretch chroot, please do:

apt-get build-dep kodi

then try to execute kodi-test:

./kodi-test

This is all you need to reproduce the problem.

Thanks.

___
pkg-multimedia-maintainers mailing list
pkg-multimedia-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers

Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Santiago Vila]

tags 820416 - unreproducible
found 820416 16.1~rc2+dfsg1-1
thanks

I can reproduce it *every* time on several machines, so the problem is
indeed real and not imagined.

The program that segfaults is "kodi-test". Here is a backtrace:

#0  __gnu_cxx::__normal_iterator > >::__normal_iterator (__i=@0x201: 
, this=)
at /usr/include/c++/5/bits/stl_iterator.h:741
#1  std::vector >::begin (this=0x201)
at /usr/include/c++/5/bits/stl_vector.h:548
#2  CEvent::Set (this=this@entry=0x7ffdea4caca0) at Event.cpp:78
#3  0x014b2d09 in CThread::staticThread (data=0x7ffdea4cabd0) at 
Thread.cpp:137
#4  0x7f871f914454 in start_thread (arg=0x7f86f65b1700) at 
pthread_create.c:334
#5  0x7f8716dddecd in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


If you still believe there is not a problem anywhere, I can give you
access to a virtual machine where the problem may be reproduced easily
(please contact me privately for this).


I'm testing on a machine having 4GB of RAM and 4GB of swap.

According to my tests, kodi needs an average of 300 MB and a maximum
of 1200 MB to build. If the tests suddenly need more than 8 GB of
virtual memory (version 16.0+dfsg1-1 worked ok), I would say this is
disproportionate and a big step backwards.

Thanks.

___
pkg-multimedia-maintainers mailing list
pkg-multimedia-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers

Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Balint Reczey]

Control: tags -1 unreproducible
Control: notfound -1 16.1~rc2+dfsg1-1

Hi Santiago,

On Fri, 8 Apr 2016 00:21:34 +0200 (CEST) Santiago Vila 
wrote:
> 
> Dear maintainer: This package fails to build from source in stretch.
> 
> The error happens while doing the tests:
> 
> 
> [--] 1 test from TestAsyncFileCopy
> [ RUN  ] TestAsyncFileCopy.General
> /bin/bash: line 1: 21695 Segmentation fault  
> /<>/kodi-16.1~rc2+dfsg1/$check_program
> Makefile:608: recipe for target 'check' failed
> make[1]: *** [check] Error 139
> make[1]: Leaving directory '/<>/kodi-16.1~rc2+dfsg1'
> dh_auto_test: make -j1 check VERBOSE=1 returned exit code 2
> debian/rules:81: recipe for target 'build' failed
> make: *** [build] Error 2
> dpkg-buildpackage: error: debian/rules build gave error exit status 2
> 

I have just tested building the package in a clean Stretch chroot using
sbuild and it built fine.
The probloblem may not be reliably reproducible or got fixed in an other
package.

Cheers,
Balint

___
pkg-multimedia-maintainers mailing list
pkg-multimedia-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers

Bug#820416: kodi: FTBFS in testing (Segmentation fault)

[Santiago Vila]

Package: src:kodi
Version: 16.1~rc2+dfsg1-1
Severity: serious

Dear maintainer: This package fails to build from source in stretch.

The error happens while doing the tests:


[--] 1 test from TestAsyncFileCopy
[ RUN  ] TestAsyncFileCopy.General
/bin/bash: line 1: 21695 Segmentation fault  
/<>/kodi-16.1~rc2+dfsg1/$check_program
Makefile:608: recipe for target 'check' failed
make[1]: *** [check] Error 139
make[1]: Leaving directory '/<>/kodi-16.1~rc2+dfsg1'
dh_auto_test: make -j1 check VERBOSE=1 returned exit code 2
debian/rules:81: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2


The full build log is attached.

Thanks.

kodi_16.1~rc2+dfsg1-1_amd64-20160407-2325.gz
Description: application/gzip
___
pkg-multimedia-maintainers mailing list
pkg-multimedia-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-multimedia-maintainers