Bug#814183: openmpi 1.10.2 is broken on powerpc

[Alastair McKinstry]

On 04/09/2016 15:43, Emilio Pozuelo Monfort wrote:
> On Fri, 12 Feb 2016 00:17:28 +0100 Emilio Pozuelo Monfort  
> wrote:
>> On Tue, 9 Feb 2016 21:49:29 +0200 Graham Inggs  wrote:
>>> Petsc rebuilt successfully [1] a couple of hours ago on poulenc.d.o. [2].
>>> My previous tests were done on partch.d.o. [3].  Partch has 2GB of RAM
>>> vs Poulenc's 5GB, I don't know if this is significant.
>> aces3 failed on powerpc-osuosl-01.
>>
>> poulenc is a PPC970FX
>> patch is a POWER7
>> powerpc-osuosl-01 is a POWER8
> Any progress on this? Has this been forwarded upstream?
Yes, reported upstream.
I'm testing out a new version 2.0.1 that may have a fix.

>
> Emilio
Alastair

-- 
Alastair McKinstry, , , 
https://diaspora.sceal.ie/u/amckinstry
Misentropy: doubting that the Universe is becoming more disordered. 

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Emilio Pozuelo Monfort]

On Fri, 12 Feb 2016 00:17:28 +0100 Emilio Pozuelo Monfort  
wrote:
> On Tue, 9 Feb 2016 21:49:29 +0200 Graham Inggs  wrote:
> > Petsc rebuilt successfully [1] a couple of hours ago on poulenc.d.o. [2].
> > My previous tests were done on partch.d.o. [3].  Partch has 2GB of RAM
> > vs Poulenc's 5GB, I don't know if this is significant.
> 
> aces3 failed on powerpc-osuosl-01.
> 
> poulenc is a PPC970FX
> patch is a POWER7
> powerpc-osuosl-01 is a POWER8

Any progress on this? Has this been forwarded upstream?

Emilio

Bug#814183: [Debichem-devel] Bug#816590: Bug#814183: openmpi 1.10.2 is broken on powerpc

[Graham Inggs]

Please see #816101 [1].  It seems the powerpc and mipsel issues are
closely related.
The PETSc package maintainer conditionally disabled the 2 process MPI
tests on powerpc and mipsel in order to work around the problem.


[1] https://bugs.debian.org/816101

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Graham Inggs]

On 3 March 2016 at 13:47, Emilio Pozuelo Monfort  wrote:
> Might be related to #813722 / #814183.

Definitely.

ELPA built on poulenc and praetorius, but failed on powerpc-osuosl-01:

https://buildd.debian.org/status/logs.php?pkg=elpa=powerpc

Only looking at elpa >= 2015.05.001-1 since openmpi 1.10, and ignoring
failures quicker than 2.5 hours due to bugs in packaging.

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Graham Inggs]

I filed LP: #1550863 [1] to track the powerpc build failures in Ubuntu.


[1] https://bugs.launchpad.net/bugs/1550863

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Emilio Pozuelo Monfort]

On Fri, 12 Feb 2016 09:25:56 +0200 Graham Inggs  wrote:
> On 12 February 2016 at 01:17, Emilio Pozuelo Monfort  wrote:
> > On Tue, 9 Feb 2016 21:49:29 +0200 Graham Inggs  wrote:
> >> Petsc rebuilt successfully [1] a couple of hours ago on poulenc.d.o. [2].
> >> My previous tests were done on partch.d.o. [3].  Partch has 2GB of RAM
> >> vs Poulenc's 5GB, I don't know if this is significant.
> >
> > aces3 failed on powerpc-osuosl-01.
> >
> > poulenc is a PPC970FX
> > patch is a POWER7
> > powerpc-osuosl-01 is a POWER8
> >
> > Dunno if that is relevant.
> 
> It might be, thanks!  Is there any way to arrange for aces3 to be
> rebuilt on poulenc?  That should tell us something.

It built on poulenc and failed on powerpc-osuosl-01:

https://buildd.debian.org/status/logs.php?pkg=aces3=3.0.8-5%2Bb1=powerpc

Emilio

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Graham Inggs]

On 12 February 2016 at 01:17, Emilio Pozuelo Monfort  wrote:
> On Tue, 9 Feb 2016 21:49:29 +0200 Graham Inggs  wrote:
>> Petsc rebuilt successfully [1] a couple of hours ago on poulenc.d.o. [2].
>> My previous tests were done on partch.d.o. [3].  Partch has 2GB of RAM
>> vs Poulenc's 5GB, I don't know if this is significant.
>
> aces3 failed on powerpc-osuosl-01.
>
> poulenc is a PPC970FX
> patch is a POWER7
> powerpc-osuosl-01 is a POWER8
>
> Dunno if that is relevant.

It might be, thanks!  Is there any way to arrange for aces3 to be
rebuilt on poulenc?  That should tell us something.

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Emilio Pozuelo Monfort]

On Tue, 9 Feb 2016 21:49:29 +0200 Graham Inggs  wrote:
> Petsc rebuilt successfully [1] a couple of hours ago on poulenc.d.o. [2].
> My previous tests were done on partch.d.o. [3].  Partch has 2GB of RAM
> vs Poulenc's 5GB, I don't know if this is significant.

aces3 failed on powerpc-osuosl-01.

poulenc is a PPC970FX
patch is a POWER7
powerpc-osuosl-01 is a POWER8

Dunno if that is relevant.

Cheers,
Emilio

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Graham Inggs]

Petsc rebuilt successfully [1] a couple of hours ago on poulenc.d.o. [2].
My previous tests were done on partch.d.o. [3].  Partch has 2GB of RAM
vs Poulenc's 5GB, I don't know if this is significant.


[1] 
https://buildd.debian.org/status/fetch.php?pkg=petsc=powerpc=3.6.2.dfsg1-3%2Bb3=1455016089
[2] https://db.debian.org/machines.cgi?host=poulenc
[3] https://db.debian.org/machines.cgi?host=partch

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Graham Inggs]

I don't believe the warning below is related to the problem.

> A deprecated MCA variable value was specified in the environment or
> on the command line.  Deprecated MCA variables should be avoided;
> they may disappear in future releases.

It can be avoided by changing the following line in petsc's debian/rules

export OMPI_MCA_orte_rsh_agent=/bin/false

to

export OMPI_MCA_plm_rsh_agent=/bin/false

Unfortunately this does not prevent the building ending with (as does aces3):

Build killed with signal TERM after 150 minutes of inactivity

On powerpc, running one of petsc's tests on one processor gets a
result (instantly):

$ mpiexec -n 1 ./ex19 -da_refine 3 -snes_monitor_short -pc_type mg
-ksp_type fgmres -pc_mg_type full
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
  0 SNES Function norm 0.0406612
  1 SNES Function norm 3.33636e-06
  2 SNES Function norm 1.653e-11
Number of SNES iterations = 2

Running it on two processors never completes:

$ mpiexec -n 2 ./ex19 -da_refine 3 -snes_monitor_short -pc_type mg
-ksp_type fgmres -pc_mg_type full
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
  0 SNES Function norm 0.0406612

Bug#814183: openmpi 1.10.2 is broken on powerpc

[Matthias Klose]

Package: src:openmpi
Version: 1.10.2-5
Severity: serious
Tags: sid stretch

openmpi 1.10.2 is broken on powerpc.

Graham Inggs confirmed that at least aces3 and petsc fail in the same way in 
Debian unstable, as soon the mpi test program is launched.


[...]
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
--
A deprecated MCA variable value was specified in the environment or
on the command line.  Deprecated MCA variables should be avoided;
they may disappear in future releases.

  Deprecated variable: orte_rsh_agent
  New variable:plm_rsh_agent
--
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
Number of SNES iterations = 2

Session terminated, terminating shell... ...terminated.
make: *** [build-arch] Terminated

build logs:
https://launchpad.net/ubuntu/+source/aces3/3.0.8-5build2/+build/8974836
https://launchpad.net/ubuntu/+source/petsc/3.6.2.dfsg1-3build2/+build/8975053