Bug#1037851: (no subject)

[Matthew Fernandez]

Btw, why does this bug not appear when searching for bugs against the 
package? https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=rumur

Bug#1002186: rumur: FTBFS: dh_auto_test: error: cd obj-x86_64-linux-gnu && make -j4 check ARGS\+=--verbose ARGS\+=-j4 returned exit code 2

[Matthew Fernandez]

Thanks for reporting!

It looks like there are now libc calls to `newfstatat` which is not in Rumur’s 
sandbox’s list of safe allowed syscalls. I’ll fix this upstream and then 
package a new release for Debian.

Bug#969156: rumur FTBFS on armel/armhf/mipsel/mips64el: test failures

[Matthew Fernandez]

Thanks for reporting this, Adrian. I noticed this some time after upload and 
fixed it upstream: 
https://github.com/Smattr/rumur/commit/68683c4742b380421936a703c4b9262dac1e68dc 


I hadn’t uploaded a fixed version to Debian because Rumur’s release cadence is 
so frequent I expected to release a new upstream version before the fix could 
get into Debian unstable. I hope this is an acceptable strategy.

By the way, is there any way to run the buildd tests prior to upload? I don’t 
have, e.g. armhf, hardware and bringing up a usable armhf qemu environment 
seems non-trivial. The buildd tests only seem to run after an upload is 
accepted, so I end up only being able to run this cross-platform testing once 
per release and only after the fact.

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

> One small issue... Valgrind recommends -O0 or -O1

TIL :) Thanks, Jeff!

> You can sometimes locate a bus error at build time with -Wcast-align.
> At runtime you can usually locate them with -fsanitize=undefined.

I had previously tried UBSan and, while it turned up a number of shifting and 
strict aliasing violations, it did not find anything that I believe could be 
the cause of the mipsel bus error.

> I can't track the trail any further. The source code is missing for
> pvm_pkdouble and pvm_upkdouble. I would be very suspect of
> pvm_pkdouble and pvm_upkdouble.

Unfortunately I think the reason you can’t find their source is that this code 
is unused. It is inside a ifdef SRE_ENABLE_PVM block and this macro is not 
defined during the build. You can confirm this by deleting these lines and 
recompiling.

AFAICT the code in src/squid is a library taken wholesale from somewhere else 
and then modified (see src/squid/clustalo.README). It contains a lot of code 
that goes unused in Clustal Omega.

Is the priority goal here to simply ship a non-crashing clustalo mipsel binary 
that BioPython can depend on? If so, maybe we can just disable compiler 
optimisation (-O0) and this may avoid provoking the bus error. Of course this 
doesn’t fix the underlying problem(s), but it looks as if debugging this to its 
root cause is going to result in the Debian package carrying a lot of invasive 
dquilt patches on top of upstream. OTOH I don’t know the performance 
requirements of BioPython and maybe an unoptimised clustalo is unacceptable to 
it.

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

> On Apr 30, 2020, at 00:31, Andreas Tille  wrote:
> 
> On Wed, Apr 29, 2020 at 05:51:26PM -0700, Matthew Fernandez wrote:
> 
>> The other option I suggested was Valgrind, but if you can’t run apt-file you 
>> probably can’t install Valgrind either.
> 
> Well, I guess apt-get is permitted for sudo but not apt-file.  So I can
> probably install valgrind inside the chroot environment.  I've never
> worked with valgrind.  What am I supposed to do?

Valgrind, in its default mode, checks for a variety of memory issues 
(use-after-free, write out-of bounds, …). You don’t need any special 
configure/build options, but you probably want to enable debug symbols (`export 
CFLAGS=-g; export CXXFLAGS=-g`). Then you can prefix the test you’re running 
with Valgrind: `valgrind ./src/clustalo -i debian/tests/biopython_testdata/f002 
…`.

Valgrind and ASan serve roughly the same purpose in this scenario, but I 
usually tend to prefer ASan because it is more efficient and tends to give you 
more accurate debugging clues.

> On the other hand:  Is valgrind possibly able to uncover issues also
> on any other architecture?

You mean if we were to use Valgrind to debug this on e.g. x86? In my own 
experiments on amd64, both ASan and Valgrind found multiple issues in both 
Clustal Omega and its dependency, argtable2. However I don’t believe any of the 
remaining ones I was seeing after the last patches I sent you could be causing 
the mipsel bus error; they were all memory leaks.

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

> On Apr 29, 2020, at 09:04, Andreas Tille  wrote:
> 
> On Wed, Apr 29, 2020 at 07:14:30AM -0700, Matthew Fernandez wrote:
> 
>> For those on this thread who have access to mipsel hardware or can shell in 
>> to one of the mipsel build machines, I would suggest running an 
>> ASan-instrumented test there (`export CFLAGS="-g -fsanitize=address"; export 
>> CXXFLAGS="-g -fsanitize=address"`) and see what we learn.
> 
> I tried on real hardware.  Unfortunately I'm running into
> 
> 
> 
> configure:3720: $? = 0
> configure:3709: gcc -v >&5
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/gcc/mipsel-linux-gnu/9/lto-wrapper
> Target: mipsel-linux-gnu
> Configured with: ../src/configure -v --with-pkgversion='Debian 9.3.0-10' 
> --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs 
> --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,gm2 --prefix=/usr 
> --with-gcc-major-version-only --program-suffix=-9 
> --program-prefix=mipsel-linux-gnu- --enable-shared --enable-linker-build-id 
> --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix 
> --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug 
> --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new 
> --enable-gnu-unique-object --disable-libitm --disable-libsanitizer 
> --disable-libquadmath --disable-libquadmath-support --enable-plugin 
> --enable-default-pie --with-system-zlib --with-target-system-zlib=auto 
> --enable-multiarch --disable-werror --enable-multilib --with-arch-32=mips32r2 
> --with-fp-32=xx --with-madd4=no --with-lxc1-sxc1=no --enable-targets=all 
> --with-arch-64=mips64r2 --enable-checking=release --build=mipsel-linux-gnu 
> --host=mipsel-linux-gnu --target=mipsel-linux-gnu
> Thread model: posix
> gcc version 9.3.0 (Debian 9.3.0-10) 
> configure:3720: $? = 0
> configure:3709: gcc -V >&5
> gcc: error: unrecognized command line option '-V'
> gcc: fatal error: no input files
> compilation terminated.
> configure:3720: $? = 1
> configure:3709: gcc -qversion >&5
> gcc: error: unrecognized command line option '-qversion'; did you mean 
> '--version'?
> gcc: fatal error: no input files
> compilation terminated.
> configure:3720: $? = 1
> configure:3740: checking whether the C compiler works
> configure:3762: gcc -g -O2 -fdebug-prefix-map=/home/tille/clustalo=. 
> -fstack-protector-strong -Wformat -Werror=format-security -fsanitize=address 
> -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wl,-z,now conftest.c  >&5
> /usr/bin/ld: cannot find libasan_preinit.o: No such file or directory
> /usr/bin/ld: cannot find -lasan
> collect2: error: ld returned 1 exit status
> configure:3766: $? = 1
> configure:3804: result: no
> configure: failed program was:
> 
> 
> 
> I have no idea why libasan_preinit.o and libasan.a are not installed.
> The package libgcc-9-dev is installed for sure and on amd64 both files
> are included here.  It seems the sudo command on eller does not permit
> me executing `apt-file update` in a chroot so I have no idea where these
> files are on mipsel (if they exist at all).
> 
> Any more help from debian-mipsel is really appreciated.

Hm yes, “--disable-libsanitizer” is rather ominous. I guess the mipsel GCC 
package has been built without ASan support. Surprising that it fails so 
messily (the front end seems to think -fsanitize=address is an accepted command 
line option), but libasan does indeed seem not available on mipsel [0]. The 
other option I suggested was Valgrind, but if you can’t run apt-file you 
probably can’t install Valgrind either. If anyone spectating has ideas, please 
chime in.

  [0]: 
https://packages.debian.org/search?searchon=contents=libasan.so=exactfilename=sid=mipsel

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

> On Apr 29, 2020, at 02:12, Andreas Tille  wrote:
> 
> Hi,
> 
> On Wed, Apr 29, 2020 at 10:30:35AM +0800, 黄佳文 wrote:
>> I am a developer from Loongson company (R & D CPU/mip64el), I've been
>> looking at this recently.
> 
> Very nice to see mips developers to care for biological software. :-)
> 
>> I did two experiments, and I found that when I used Python 3,7 to compile
>> python-biopython, Build successfully.
>> In the same environment, I just upgrade Python 3.7 to Python 3.8, and then
>> compile python-biopytho, Build fails, but not bus error.
>> I found through online query that some symbol tables were added and deleted
>> in upgrading Python 3.7 to 3.8. The following are symbol tables:
> 
> Sorry to insist here - I do not think that it is a Python version problem
> at all.  The issue can be reproduced in clustalo only which is pure C code.
> May be you have a look at
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956324#59
> 
> and the following discussion.  Despite Matthew has found some issues inside
> the C code it did not helped to prevent:
> 
> 
> Starting program: /home/tille/clustalo/src/clustalo -i 
> debian/tests/biopython_testdata/f002 --guidetree-out temp_test.dnd -o 
> temp_test.aln --outfmt clustal --force
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1".
> 
> Program received signal SIGBUS, Bus error.
> 0x5556a1b8 in PairDistances (distmat=0x7fff278c, mseq=0x55692a30, 
> pairdist_type=, bPercID=, istart=0, iend=3, 
> jstart=0, jend=3, fdist_in=0x0, 
>fdist_out=0x0) at pair_dist.c:346
> 346 NewProgress(, LogGetFP(, LOG_INFO),
> 
> 
> That's the issue we need to care about here.

To add another data point to this discussion, one other (fruitless) thing I 
tried previously was cross-compiling Clustal Omega. From an amd64 host, it’s 
possible to target mipsel using the GCC cross-compilers in the standard Debian 
repositories. You can then run the resulting binary using Qemu’s user mode. 
Using this technique, the f002 test runs to completion with no bus error. This 
is not really surprising as AFAIK unaligned accesses that would trigger a bus 
error on mipsel hardware would be silently allowed in this configuration (Qemu 
doesn’t faithfully emulate this hardware behaviour and amd64 allows unaligned 
access).

Unfortunately the repositories’ cross-compilers have been built without ASan 
enabled and you can’t attach to an emulated mipsel process with a native 
Valgrind. So debugging memory safety issues is not straightforward. To go 
further with this approach, you would have to build a mipsel-targeting 
cross-compiler with ASan enabled or cross-compile Valgrind to mipsel. For a 
true masochist, it may be possible to attach to the process with GDB or rr and 
reverse-step from the location Andreas has quoted, but I wouldn’t trust the 
debugger not to crash in this configuration. Even then the issue may not be 
reproducible because it may be dependent on transformations/optimizations only 
performed by the particular version of the native mipsel compiler called during 
packaging.

For those on this thread who have access to mipsel hardware or can shell in to 
one of the mipsel build machines, I would suggest running an ASan-instrumented 
test there (`export CFLAGS="-g -fsanitize=address"; export CXXFLAGS="-g 
-fsanitize=address"`) and see what we learn.

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

> On Apr 17, 2020, at 22:39, Andreas Tille  wrote:
> 
> Hi Matthew,
> 
> thanks a lot for your detailed investigation.
> 
> On Fri, Apr 17, 2020 at 04:28:23PM -0700, Matthew Fernandez wrote:
>>> Program received signal SIGBUS, Bus error.
>>> 0x5556a1b8 in PairDistances (distmat=0x7fff278c, mseq=0x55692a30, 
>>> pairdist_type=, bPercID=, istart=0, iend=3, 
>>> jstart=0, jend=3, fdist_in=0x0, 
>>>   fdist_out=0x0) at pair_dist.c:346
>>> 346 NewProgress(, LogGetFP(, LOG_INFO),
>> 
>> OK, let me try a little harder :)
>> 
>>$ # enable debugging symbols and Address Sanitizer
>>$ CFLAGS="-g -fsanitize=address" CXXFLAGS="-g -fsanitize=address" 
>> ./configure
>>…
>>$ make clean && make
>>…
>>$ ./src/clustalo -i debian/tests/biopython_testdata/f002 --guidetree-out 
>> temp_test.dnd -o temp_test.aln --outfmt clustal --force
>>=
>>==30264==ERROR: AddressSanitizer: dynamic-stack-buffer-overflow on 
>> address 0x7ffcfcbf5784 at pc 0x5620f0aa478c bp 0x7ffcfcbf56c0 sp 
>> 0x7ffcfcbf56b8
>>WRITE of size 4 at 0x7ffcfcbf5784 thread T0
>>#0 0x5620f0aa478b in PairDistances 
>> /home/matthew/clustal-omega-1.2.4/src/clustal/pair_dist.c:336
>>#1 0x5620f0a91d9f in AlignmentOrder 
>> /home/matthew/clustal-omega-1.2.4/src/clustal-omega.c:835
>>#2 0x5620f0a95c04 in Align 
>> /home/matthew/clustal-omega-1.2.4/src/clustal-omega.c:1221
>>#3 0x5620f0a90d76 in MyMain 
>> /home/matthew/clustal-omega-1.2.4/src/mymain.c:1192
>>#4 0x5620f0a88ca2 in main 
>> /home/matthew/clustal-omega-1.2.4/src/main.cpp:469
>>#5 0x7f3773d9009a in __libc_start_main ../csu/libc-start.c:308
>>#6 0x5620f0a89ad9 in _start 
>> (/home/matthew/clustal-omega-1.2.4/src/clustalo+0x2dad9)
>> 
>>Address 0x7ffcfcbf5784 is located in stack of thread T0
>>SUMMARY: AddressSanitizer: dynamic-stack-buffer-overflow 
>> /home/matthew/clustal-omega-1.2.4/src/clustal/pair_dist.c:336 in 
>> PairDistances
>>Shadow bytes around the buggy address:
>>  0x10001f976aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>  0x10001f976ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>  0x10001f976ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>  0x10001f976ad0: 00 00 00 00 00 00 00 00 00 00 00 00 ca ca ca ca
>>  0x10001f976ae0: 04 cb cb cb cb cb cb cb 00 00 00 00 ca ca ca ca
>>=>0x10001f976af0:[04]cb cb cb cb cb cb cb 00 00 00 00 00 00 00 00
>>  0x10001f976b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>  0x10001f976b10: f1 f1 f1 f1 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2
>>  0x10001f976b20: f2 f2 f2 f2 00 00 00 00 00 00 00 00 00 f2 f2 f2
>>  0x10001f976b30: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
>>  0x10001f976b40: 00 00 00 00 00 00 f1 f1 f1 f1 00 f2 f2 f2 f2 f2
>>Shadow byte legend (one shadow byte represents 8 application bytes):
>>  Addressable:   00
>>  Partially addressable: 01 02 03 04 05 06 07
>>  Heap left redzone:   fa
>>  Freed heap region:   fd
>>  Stack left redzone:  f1
>>  Stack mid redzone:   f2
>>  Stack right redzone: f3
>>  Stack after return:  f5
>>  Stack use after scope:   f8
>>  Global redzone:  f9
>>  Global init order:   f6
>>  Poisoned by user:f7
>>  Container overflow:  fc
>>  Array cookie:ac
>>  Intra object redzone:bb
>>  ASan internal:   fe
>>  Left alloca redzone: ca
>>  Right alloca redzone:cb
>>==30264==ABORTING
>> 
>> Looking at line 336 of pair_dist.c, it looks like the bound on the 
>> containing loop is wrong. So let’s try adjusting that:
>> 
>>$ vim src/clustal/pair_dist.c
>>$ git diff src/clustal/pair_dist.c
>>diff --git a/src/clustal/pair_dist.c b/src/clustal/pair_dist.c
>>index e6dbdc3..bb79e61 100644
>>--- a/src/clustal/pair_dist.c
>>+++ b/src/clustal/pair_dist.c
>>@@ -321,7 +321,7 @@ PairDistances(symmatrix_t **distmat, mseq_t *mseq, 
>> int pairdist_type, bool bPerc
>> 
>> /* FIXME: can get rid of iChunkStart, iChunkEnd now that we're 
>> using the arrays */
>> iChunkStart = iend;
>>-for(iChunk = 0; iChunk <= iNumberOfThreads; iChunk++)
>>+for(iChunk = 0;

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

> On Apr 17, 2020, at 13:18, Andreas Tille  wrote:
> 
> Hi Matthew,
> 
> On Fri, Apr 17, 2020 at 08:18:29AM -0700, Matthew Fernandez wrote:
>>> Thanks for the patch which I applied to packaging Git.  I assume you
>>> want to express that while these fixes are definitely good coding
>>> practice the bus error problem is not fixed by it, right?
>> 
>> Thanks, Andreas. It may fix the bus error, but I don’t have a MIPS machine
>> to test on. Some of those logging calls had the potential to leave you with
>> a misaligned stack pointer. IIUC unaligned loads on MIPS could cause such a
>> bus error.
> 
> I tried with hope ... but failed:
> 
> (sid_mipsel-dchroot)tille@eller:~/clustalo$ gdb --args src/clustalo -i 
> debian/tests/biopython_testdata/f002 --guidetree-out temp_test.dnd -o 
> temp_test.aln --outfmt clustal --force
> GNU gdb (Debian 9.1-3) 9.1
> ...
> Reading symbols from src/clustalo...
> (gdb) run
> Starting program: /home/tille/clustalo/src/clustalo -i 
> debian/tests/biopython_testdata/f002 --guidetree-out temp_test.dnd -o 
> temp_test.aln --outfmt clustal --force
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1".
> 
> Program received signal SIGBUS, Bus error.
> 0x5556a1b8 in PairDistances (distmat=0x7fff278c, mseq=0x55692a30, 
> pairdist_type=, bPercID=, istart=0, iend=3, 
> jstart=0, jend=3, fdist_in=0x0, 
>fdist_out=0x0) at pair_dist.c:346
> 346 NewProgress(, LogGetFP(, LOG_INFO),

OK, let me try a little harder :)

$ # enable debugging symbols and Address Sanitizer
$ CFLAGS="-g -fsanitize=address" CXXFLAGS="-g -fsanitize=address" 
./configure
…
$ make clean && make
…
$ ./src/clustalo -i debian/tests/biopython_testdata/f002 --guidetree-out 
temp_test.dnd -o temp_test.aln --outfmt clustal --force
=
==30264==ERROR: AddressSanitizer: dynamic-stack-buffer-overflow on address 
0x7ffcfcbf5784 at pc 0x5620f0aa478c bp 0x7ffcfcbf56c0 sp 0x7ffcfcbf56b8
WRITE of size 4 at 0x7ffcfcbf5784 thread T0
#0 0x5620f0aa478b in PairDistances 
/home/matthew/clustal-omega-1.2.4/src/clustal/pair_dist.c:336
#1 0x5620f0a91d9f in AlignmentOrder 
/home/matthew/clustal-omega-1.2.4/src/clustal-omega.c:835
#2 0x5620f0a95c04 in Align 
/home/matthew/clustal-omega-1.2.4/src/clustal-omega.c:1221
#3 0x5620f0a90d76 in MyMain 
/home/matthew/clustal-omega-1.2.4/src/mymain.c:1192
#4 0x5620f0a88ca2 in main 
/home/matthew/clustal-omega-1.2.4/src/main.cpp:469
#5 0x7f3773d9009a in __libc_start_main ../csu/libc-start.c:308
#6 0x5620f0a89ad9 in _start 
(/home/matthew/clustal-omega-1.2.4/src/clustalo+0x2dad9)

Address 0x7ffcfcbf5784 is located in stack of thread T0
SUMMARY: AddressSanitizer: dynamic-stack-buffer-overflow 
/home/matthew/clustal-omega-1.2.4/src/clustal/pair_dist.c:336 in PairDistances
Shadow bytes around the buggy address:
  0x10001f976aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001f976ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001f976ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001f976ad0: 00 00 00 00 00 00 00 00 00 00 00 00 ca ca ca ca
  0x10001f976ae0: 04 cb cb cb cb cb cb cb 00 00 00 00 ca ca ca ca
=>0x10001f976af0:[04]cb cb cb cb cb cb cb 00 00 00 00 00 00 00 00
  0x10001f976b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001f976b10: f1 f1 f1 f1 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2
  0x10001f976b20: f2 f2 f2 f2 00 00 00 00 00 00 00 00 00 f2 f2 f2
  0x10001f976b30: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
  0x10001f976b40: 00 00 00 00 00 00 f1 f1 f1 f1 00 f2 f2 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:   00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:   fa
  Freed heap region:   fd
  Stack left redzone:  f1
  Stack mid redzone:   f2
  Stack right redzone: f3
  Stack after return:  f5
  Stack use after scope:   f8
  Global redzone:  f9
  Global init order:   f6
  Poisoned by user:f7
  Container overflow:  fc
  Array cookie:ac
  Intra object redzone:bb
  ASan internal:   fe
  Left alloca redzone: ca
  Right alloca redzone:cb
==30264==ABORTING

Looking at line 336 of pair_dist.c, it looks like the bound on the containing 
loop is wrong. So let’s try adjusting that:

$ vim src/clustal/pair_dist.c
$ git diff src/clustal/pair_dist.c
diff --git a/src/clustal/pair_dist.c b/src/clustal/pair_dist.c
index e6dbdc3..bb79e61 1

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

On Fri, 17 Apr 2020 at 08:09, Andreas Tille  wrote:

> Hi Matthew,
>
> On Fri, Apr 17, 2020 at 07:40:54AM -0700, Matthew Fernandez wrote:
> >
> > As a jumping off point, the attached patch fixes some issues with
> logging calls in the upstream 1.2.4 source release.
>
> Thanks for the patch which I applied to packaging Git.  I assume you
> want to express that while these fixes are definitely good coding
> practice the bus error problem is not fixed by it, right?


Thanks, Andreas. It may fix the bus error, but I don’t have a MIPS machine
to test on. Some of those logging calls had the potential to leave you with
a misaligned stack pointer. IIUC unaligned loads on MIPS could cause such a
bus error.

Bug#956324: Clustalo bus error on mipsel (Was: Bug#956324: python-biopython: FTBFS on mipsel)

[Matthew Fernandez]

> On Apr 17, 2020, at 04:20, Andreas Tille  wrote:
> 
> Control: tags -1 help
> 
> Hi,
> 
> as it can be seen on the recent build log of clustalo on mips[1] the
> build fails with
> 
> 
> # Run additional test from python-biopython package to verify that
> # this will work as well
> src/clustalo -i debian/tests/biopython_testdata/f002 --guidetree-out 
> temp_test.dnd -o temp_test.aln --outfmt clustal --force
> make[1]: *** [debian/rules:25: override_dh_auto_test-arch] Bus error
> 
> 
> I injected that extra test since this very test failed inside the
> python-biopython package.  To track it down I logged in to
> eller.debian.org tried to build the package and was able to reproduce
> the error.  Thus I fired up gdb in this session and got:
> 
> 
> (sid_mipsel-dchroot)tille@eller:~/clustalo-1.2.4$ gdb --args src/clustalo -i 
> debian/tests/biopython_testdata/f002 --guidetree-out temp_test.dnd -o 
> temp_test.aln --outfmt clustal --force
> GNU gdb (Debian 9.1-3) 9.1
> ...
> Reading symbols from src/clustalo...
> (gdb) run
> Starting program: /home/tille/clustalo-1.2.4/src/clustalo -i 
> debian/tests/biopython_testdata/f002 --guidetree-out temp_test.dnd -o 
> temp_test.aln --outfmt clustal --force
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1".
> 
> Program received signal SIGBUS, Bus error.
> 0x5556a188 in PairDistances (distmat=0x7fff278c, mseq=0x55692a38, 
> pairdist_type=, bPercID=, istart=0, iend=3, 
> jstart=0, jend=3, fdist_in=0x0, fdist_out=0x0) at pair_dist.c:346
> 346 NewProgress(, LogGetFP(, LOG_INFO),
> (gdb)
> 
> 
> So it seems the bus error occures somehow here:
> 
>   
> https://salsa.debian.org/med-team/clustalo/-/blob/master/src/clustal/pair_dist.c#L346
> 
> 
> I have no idea how to continue from here.  I'm hoping that someone with
> some more detailed knowledge might have a clue how to fix this issue on
> mipsel.  Otherwise I personally do not see any better solution than to
> remove clustalo from mipsel architecture.

As a jumping off point, the attached patch fixes some issues with logging calls 
in the upstream 1.2.4 source release.



clustalo-log-format-calls.patch
Description: Binary data

Bug#949542: C++ help needed for pbdagcon

[Matthew Fernandez]

> On Jan 22, 2020, at 11:56, Andreas Tille  wrote:
> 
> Control: tags -1 help
> 
> I have fixed Python2->Python3 migration as well as the FTBFS with
> pbseqlib 5.3.3+dfsg-1 issue in Git[1].  Unfortunately there is another
> build issue in the C++ code which I have no idea how to fix:
> 
> 
> ...
> g++ -g -O2 -fdebug-prefix-map=/build/pbdagcon-0.3+git20180411.c14c422+ds=. 
> -fstack-protector-strong -Wformat -Werror=format-security -O3 -std=c++14 
> -Wall -Wuninitialized -pedantic -Wdate-time -D_FORTIFY_SOURCE=2 -MMD -MP 
> -I/build/pbdagcon-0.3+git20180411.c14c422+ds/DAZZ_DB 
> -I/build/pbdagcon-0.3+git20180411.c14c422+ds/DALIGNER 
> -I/usr/include/pbseq/alignment -I/usr/include/pbseq/pbdata 
> -I/usr/include/pbseq/hdf -I/usr/include/pbseq -isystem.//third-party   -c -o 
> DazAlnProvider.o DazAlnProvider.cpp
> In file included from DazAlnProvider.cpp:10:
> DazAlnProvider.hpp:97:19: error: expected ')' before '&' token
>   97 | Target(DAZZ_DB& db, int tspace, int small);
>  |   ~   ^
>  |   )
> DazAlnProvider.hpp:122:5: error: 'DAZZ_DB' does not name a type
>  122 | DAZZ_DB db_;
>  | ^~~
> DazAlnProvider.hpp:161:5: error: 'DAZZ_DB' does not name a type
>  161 | DAZZ_DB db_;
>  | ^~~
> DazAlnProvider.cpp: In constructor 'DazAlnProvider::DazAlnProvider(const 
> ProgramOpts&)':
> DazAlnProvider.cpp:34:33: error: 'db_' was not declared in this scope
>   34 | int status = Open_DB(path, _);
>  | ^~~
> DazAlnProvider.cpp: In destructor 'virtual DazAlnProvider::~DazAlnProvider()':
> DazAlnProvider.cpp:74:15: error: 'db_' was not declared in this scope
>   74 | Close_DB(_);
>  |   ^~~
> DazAlnProvider.cpp: In member function 'virtual bool 
> DazAlnProvider::nextTarget(std::string&, std::vector&)':
> DazAlnProvider.cpp:125:25: error: 'db_' was not declared in this scope
>  125 | seq = Load_Subread(_, trg_->id, 0, trg_->length, targSeqBuf_, 
> 0);
>  | ^~~
> DazAlnProvider.cpp: At global scope:
> DazAlnProvider.cpp:225:15: error: expected constructor, destructor, or type 
> conversion before '(' token
>  225 | Target::Target(DAZZ_DB& db, int tspace, int small) :
>  |   ^
> DazAlnProvider.cpp: In member function 'void Target::firstRecord(Record&, 
> bool)':
> DazAlnProvider.cpp:246:14: error: 'db_' was not declared in this scope
>  246 | length = db_.reads[id].rlen;
>  |  ^~~

Just taking some educated guesses here, but the way this variable is used 
indicates to me that the code expects it to be a HITS_DB, declared in 
DAZZ_DB/DB.h. The DAZZ_DB subdirectory looks like a copy of upstream project 
https://github.com/thegenemyers/DAZZ_DB 
. The latest revision of this calls 
this type DAZZ_DB, not HITS_DB. So maybe the solution is to pull in the latest 
revision of this?

Bug#939506: expected primary-expression before ‘{’ token (Was: Bug#939506: unanimity ftbfs in unstable)

[Matthew Fernandez]

> On Jan 15, 2020, at 02:13, Andreas Tille  wrote:
> 
> Hi again,
> 
> I forgot to mention that I bounced your mail to the bug log of #939506
> and I also CC this one to make sure there is some publicly visible
> record of the discussion.

Good idea for a future audience.

> On Wed, Jan 15, 2020 at 09:27:16AM +0100, Andreas Tille wrote:
>> Hi Matthew,
>> 
>> On Tue, Jan 14, 2020 at 07:00:02PM -0800, Matthew Fernandez wrote:
>>> 
>>> Offhand I don’t know the fix to the error message you quoted, but I just 
>>> tried to reproduce the build error on Debian 10.2. This repository has 
>>> multiple build systems in the root directory and no build instructions in 
>>> the README, so I guessed CMake. However, this doesn’t work:
>>> 
>>>$ mkdir build
>>>$ cd build
>>>$ cmake ..
>>>-- The CXX compiler identification is GNU 8.3.0
>>>-- The C compiler identification is GNU 8.3.0
>>>-- Performing Test HAS_NO_UNUSED_LOCAL_TYPEDEFS - Success
>>>-- Configuring incomplete, errors occurred!
>>>See also "/home/matthew/unanimity/build/CMakeFiles/CMakeOutput.log".
>>>See also "/home/matthew/unanimity/build/CMakeFiles/CMakeError.log”.
>>> 
>>> Unfortunately without further context I don’t know how to build this 
>>> program.
>> 
>> Ahhh, I assumed you would know how to build a Debian package from Git.
>> Well, here is some short introduction.  The best idea is to use
>> git-buildpackage.  If you have installed it you can do
>> 
>>   gbp clone https://salsa.debian.org/med-team/unanimity
>>   cd unanimity
>>   gbp buildpackage
>> 
>> This will call the build system that is used in Debian.  BTW, gbp needs
>> some configuration like:
>> 
>> ~> cat ~/.gbp.conf
>> [DEFAULT]
>> builder = ~/bin/git-pbuilder
>> 
>> # Might lead to problems because it tries to use non-patched makefiles
>> # cleaner = fakeroot debian/rules clean
>> cleaner = /bin/true
>> pristine-tar = True
>> export=WC
>> 
>> [buildpackage]
>> # use this for more svn-buildpackage like behaviour:
>> export-dir = ../build-area/
>> tarball-dir = ../tarballs/
>> pbuilder = True
>> pbuilder-options=--source-only-changes
>> 
>> 
>> The builder script can be used to control that the build is done using
>> cowbuilder which is a clean chroot system.  My bin/git-pbuilder has
>> basically this line:
>> 
>>  /usr/bin/pdebuild --pbuilder cowbuilder --buildresult `dirname \$PWD` 
>> --debbuildopts "-i\.git -I.git $*"
>> 
>> Before you can use cowbuilder you need to do
>> 
>>  sudo cowbuilder --create
>> 
>> 
>> Sorry for writing instructions bottom up - but I don't know what you
>> know about Debian package building.
>> 
>> Hope that you might find some time to reproduce any may be suggest a
>> patch.  If not you might have learned at least something about Debian
>> packaging. ;-)

Despite using git-buildpackage in my own packaging attempts, I am basically a 
GBP noob. All of the above was new information to me, so thank you for 
educating me :) Following your steps, I can now reproduce this error. I’ll try 
to  investigate but given the complexity of this build system and dependencies 
I would not hold my breath. If someone else reading can offer more help, please 
do.

Bug#939506: expected primary-expression before ‘{’ token (Was: Bug#939506: unanimity ftbfs in unstable)

[Matthew Fernandez]

Hi Andreas,

This is not related to aghermann, correct? Offhand I don’t know the fix to the 
error message you quoted, but I just tried to reproduce the build error on 
Debian 10.2. This repository has multiple build systems in the root directory 
and no build instructions in the README, so I guessed CMake. However, this 
doesn’t work:

$ mkdir build
$ cd build
$ cmake ..
-- The CXX compiler identification is GNU 8.3.0
-- The C compiler identification is GNU 8.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Boost version: 1.67.0
CMake Error at cmake/uny-dependencies.cmake:21 (add_subdirectory):
  add_subdirectory given source
  "/home/matthew/unanimity/third-party/pbcopper" which is not an existing
  directory.
Call Stack (most recent call first):
  CMakeLists.txt:38 (include)


-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29")
-- Checking for module 'zlib'
--   Found zlib, version 1.2.11
CMake Error at cmake/uny-dependencies.cmake:45 (add_subdirectory):
  add_subdirectory given source "/home/matthew/unanimity/third-party/pbbam"
  which is not an existing directory.
Call Stack (most recent call first):
  CMakeLists.txt:38 (include)


-- Performing Test HAS_NO_UNUSED_LOCAL_TYPEDEFS
-- Performing Test HAS_NO_UNUSED_LOCAL_TYPEDEFS - Success
-- Configuring incomplete, errors occurred!
See also "/home/matthew/unanimity/build/CMakeFiles/CMakeOutput.log".
See also "/home/matthew/unanimity/build/CMakeFiles/CMakeError.log”.

Unfortunately without further context I don’t know how to build this program.

Thanks,
Matthew

> On Jan 14, 2020, at 12:13, Andreas Tille  wrote:
> 
> Hi Matthew,
> 
> I take the freedom to ping you personally about a bug I was asking
> for help on Debian Mentors before.  My guess is its also not such
> a hard C++ problem.
> 
> (Unfortunately when I compile unanimity from Git[1] right now I'm
> running into a different issue even before:
> 
> ...
> [ 56%] Building CXX object 
> src/CMakeFiles/unanimity.dir/genomicconsensus/experimental/ConsensusModelFactory.cpp.o
> cd /build/unanimity-3.4.1+git20180307.02aa264+dfsg/build/src && /usr/bin/c++  
>  -I/build/unanimity-3.4.1+git20180307.02aa264+dfsg/include 
> -I/build/unanimity-3.4.1+git20180307.02aa264+dfsg/build/generated 
> -I/build/unanimity-3.4.1+git20180307.02aa264+dfsg/src  -g -O2 
> -fdebug-prefix-map=/build/unanimity-3.4.1+git20180307.02aa264+dfsg=. 
> -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time 
> -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wno-unused-parameter -Wno-unused-variable 
> -Wno-unused-local-typedefs   -std=c++14 -o 
> CMakeFiles/unanimity.dir/genomicconsensus/experimental/ConsensusModelFactory.cpp.o
>  -c 
> /build/unanimity-3.4.1+git20180307.02aa264+dfsg/src/genomicconsensus/experimental/ConsensusModelFactory.cpp
> In file included from /usr/include/pbcopper/data/ReadId.h:13,
> from /usr/include/pbcopper/data/Read.h:19,
> from /usr/include/pbcopper/data/MappedRead.h:10,
> from /usr/include/pbbam/BamRecord.h:19,
> from 
> /build/unanimity-3.4.1+git20180307.02aa264+dfsg/include/pacbio/genomicconsensus/experimental/IPoaModel.h:8,
> from 
> /build/unanimity-3.4.1+git20180307.02aa264+dfsg/include/pacbio/genomicconsensus/experimental/arrow/ArrowModel.h:5,
> from 
> /build/unanimity-3.4.1+git20180307.02aa264+dfsg/src/genomicconsensus/experimental/ConsensusModelFactory.cpp:8:
> /usr/include/pbcopper/data/Interval.h:24:7: error: redefinition of ‘class 
> PacBio::Data::Interval’
>   24 | class Interval
>  |   ^~~~
> In file included from 
> /build/unanimity-3.4.1+git20180307.02aa264+dfsg/include/pacbio/genomicconsensus/experimental/ReferenceWindow.h:10,
> from 
> /build/unanimity-3.4.1+git20180307.02aa264+dfsg/include/pacbio/genomicconsensus/experimental/Consensus.h:10,
> from 
> /build/unanimity-3.4.1+git20180307.02aa264+dfsg/include/pacbio/genomicconsensus/experimental/WindowResult.h:7,
>   

Bug#925629: aghermann: ftbfs with GCC-9

[Matthew Fernandez]

Thanks for being clear, Andrei. And thanks for your work building and 
maintaining this package.

I jumped into this thread knowing nothing about this package but merely 
recognizing a compiler error. Now that I look up what it does, I don’t have the 
necessary background to fully understand its functionality. It does look like 
it’s still seeing some usage in Debian though [0]. I’m comfortable with C++, so 
Andreas, if you still want to proceed with the packaging work, I’m happy to 
review your changes.

By the way, what is the canonical upstream source for this project? All the 
repo links on the homepage [1] except SourceForge are broken and I assume no 
one is using SourceForge these days.

  [0]: https://qa.debian.org/popcon.php?package=aghermann 

  [1]: http://johnhommer.com/academic/code/aghermann/

Bug#925629: aghermann: ftbfs with GCC-9

[Matthew Fernandez]

Sorry, I didn’t have much context for the original issue and was probably too 
terse in my responses. Some more elaboration inline below.

> On Jan 13, 2020, at 01:14, Andreas Tille  wrote:
> 
> On Sun, Jan 12, 2020 at 02:08:56PM -0800, Matthew Fernandez wrote:
>> 
>>> On Jan 12, 2020, at 12:49, Andreas Tille  wrote:
>>> 
>>> Hi,
>>> 
>>> I'm wondering how this bug
>>> 
>>> 
>>> rk1968/rk1968.cc: In lambda function:
>>> rk1968/rk1968.cc:237:103: error: expected '{' before '->' token
>>> 237 | auto make_error_return = [] ( const char* fmt, ...) 
>>> __attribute__ ((format (printf, 2, 3))) -> int
>>> |   
>>> ^~
>>> 
>>> 
>>> with gcc 9 can be fixed in aghermann.  Any help would be appreciated.
>> 
>> Turning this into a C++11 attribute ([[gnu::format(printf, 2, 3)]]) makes 
>> the parse error go away, but -Wattributes still indicates GCC is ignoring it.
> 
> I admit I do not understand your "but -Wattributes ...".  I can confirm
> that this patch[1] builds the package successfully.

-Wattributes is a flag to GCC to enable warnings about attributes. What I did 
to experiment was extract the code you’d quoted into an isolated context:

$ cat - >test.cc <http://test.cc/> < int
  {
return 0;
  }
}
EOT
$ g++ -std=c++11 -c -Wattributes test.cc

>> You might need to bump that GCC issue with some evidence that the bug still 
>> exists and see what the maintainers say.
> 
> I need to admit that I understand so less from all the gcc issues you
> tried to explain - I do not even have any idea what C++ attributes are.
> I simply cared for that Debian bug since nobody else did. :-(
> So I feel really incompetent to discuss this with gcc maintainers but
> I'd welcome if you bring it up there.

GCC attributes like the __attribute__ example here are a mechanism for 
annotating C/C++ code with things not covered by the ISO standards, but 
supported by compiler extensions. Attributes can be applied to many things 
including variables and functions, and the function attributes are documented 
at https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html 
<https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html>. These days Clang 
also understands many of the GCC attributes.

The particular one in question here is telling the compiler that the lambda 
function being defined has similar behavior to the libc printf function. The 
integer parameters say a printf format string appears as the second argument 
and the first varargs parameter is the third argument. This looks off by one, 
but I think the captured local () counts as the first parameter. The only 
thing the compiler uses this attribute for is to detect calls to this lambda 
with incorrect arguments and emit warnings for them.

The C++11 standard brought in a more commonly supported way of declaring 
attributes using square brackets. Unfortunately the standard does not support 
many common attributes and you still need to use a “gnu::” prefix to access the 
GCC-specific attributes. Hence, “[[gnu::format(printf, 2, 3)]]” being the C++11 
equivalent of this.

>> If you need to bypass this urgently, I would recommend deleting the 
>> attribute as that particular one is only used to aid the compiler in 
>> creating warnings.
> 
> Hmmm, as I said my patch[1] works and I just have the gut feeling (as I
> said I have no competence!) that this might be better than to delete the
> attribute.  If not would you mind sending an alternative patch which is
> better in your opinion?

Lambda functions, which are already being used in this code, are a C++11 
feature as are this style of attributes. So I imagine it would be acceptable to 
upstream to take your patch. Having said that, when I did this in my experiment 
code above the compiler warned that it was ignoring this attribute as it 
thought it was being applied to the trailing “int”.

I re-read the GCC issue and realized I’d misread Jonathan Wakely’s comments. 
The issue is actually still open and Jonathan was just noting that r265787 
introduced the bug, not fixed it. So I think what we’re seeing is consistent 
with the GCC maintainers’ view.

I would suggest proposing your patch upstream. Although its primary purpose is 
working around a compiler bug, it’s also an objective improvement in 
modernizing the code base.

> Thanks again
> 
>   Andreas.
> 
> 
>>  [0]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90333 
>> <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90333>
> 
> [1] 
> https://salsa.debian.org/med-team/aghermann/blob/master/debian/patches/workaround_gcc-9_issue.patch
>  
> <https://salsa.debian.org/med-team/aghermann/blob/master/debian/patches/workaround_gcc-9_issue.patch>
> 
> -- 
> http://fam-tille.de <http://fam-tille.de/>

Bug#925629: aghermann: ftbfs with GCC-9

[Matthew Fernandez]

> On Jan 12, 2020, at 12:49, Andreas Tille  wrote:
> 
> Hi,
> 
> I'm wondering how this bug
> 
> 
> rk1968/rk1968.cc: In lambda function:
> rk1968/rk1968.cc:237:103: error: expected '{' before '->' token
>  237 | auto make_error_return = [] ( const char* fmt, ...) 
> __attribute__ ((format (printf, 2, 3))) -> int
>  |
>^~
> 
> 
> with gcc 9 can be fixed in aghermann.  Any help would be appreciated.

I think you’re hitting GCC bug #90333 [0]. This claims to have been fixed in 
r265787, but I can still reproduce this issue with GCC 9.2.1 that includes that 
commit. Turning this into a C++11 attribute ([[gnu::format(printf, 2, 3)]]) 
makes the parse error go away, but -Wattributes still indicates GCC is ignoring 
it. You might need to bump that GCC issue with some evidence that the bug still 
exists and see what the maintainers say. If you need to bypass this urgently, I 
would recommend deleting the attribute as that particular one is only used to 
aid the compiler in creating warnings.

  [0]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90333