[OMPI devel] RFC: Rename nightly snapshot tarballs

[Jeff Squyres (jsquyres)]

SHORT VERSION
=

Our nightly snapshot tarball filenames have become confusing.  Here's the most 
recent on each branch:

- Master: openmpi-v2.x-dev-3019-gbd1b6fe.tar.bz2
- v2.0.x: openmpi-v2.0.1-93-g97511a1.tar.bz2
- v2.x:   openmpi-v2.0.1-203-g56991c6.tar.bz2

I propose changing them to the following format:

openmpi-${BRANCHNAME}-${MMDD}-${SHORTHASH}.tar.bz2

Here's what last night's tarballs would have looked like with that scheme:

openmpi-master-20161017-bd1b6fe.tar.bz2
openmpi-v2.0.x-20161017-a66f3e2.tar.bz2
    openmpi-v2.x-20161017-56991c6.tar.bz2

Optionally, we could put a HHMM timestamp in there for finer granularity.

MORE DETAIL
===

We use "git describe" to come up with the version strings for our nightly 
snapshot tarballs.  "git describe" shows you three things:

- the first tag that it finds going back along the git commit history
- the number of commits it had to traverse to find that tag
- the short hast of the HEAD commit

For lack of a longer explanation, the names of the current snapshots *do* make 
sense given our commit and tag history; they're just... weird.

After talking this through with a colleague today, I think we have two choices:

1. Continue to use the "git describe" output
2. Come up with a different scheme

Using git describe output means that we are reliant on git tags.  Funky 
placement of git tags -- including the moving of branches back from 
ompi-release to ompi -- is how we ended up in this situation.  This might 
constitute empirical evidence that "git describe" is not a good solution for 
this community.

Instead, perhaps we should make a new scheme.  We need two properties in the 
snapshot filenames:

1. Name of the branch that the tarball came from.
2. Easily sortable by a human (i.e., know that tarball X came before or after 
tarball Y).

Property #1 is self-evident (the branch name is in the filename); property #2 
comes from the timestamp.  

NOTE: It may be desirable to add HHMM in there; it's not common, but 
*sometimes* we do make more than one snapshot in a day (e.g., if one snapshot 
is borked, so we fix it and then generate another snapshot).

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] RFC: Rename nightly snapshot tarballs

[Christopher Samuel]

On 18/10/16 07:17, Jeff Squyres (jsquyres) wrote:

> NOTE: It may be desirable to add HHMM in there; it's not common, but
> *sometimes* we do make more than one snapshot in a day (e.g., if one
> snapshot is borked, so we fix it and then generate another
> snapshot).

If it's been happened before then I'd suggest allow for it to happen
again by adding HHMM.  Otherwise looks sensible to me (YMMV).

-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] Fwd: Problem with double shared library

[Gilles Gouaillardet]

Folks,


this is a follow up on a question from the users ML


is there any reason why plugins do not depend on the main openmpi libs 
(libopen-pal.so and libopen-rte.so, libompi.so and liboshmem.so if needed) ?


i guess that would solve the issue here without having to use RTLD_GLOBAL.


Cheers,


Gilles



 Forwarded Message 
Subject:Re: [OMPI users] Problem with double shared library
Date:   Tue, 18 Oct 2016 10:45:42 +0900
From:   Gilles Gouaillardet 
To: Open MPI Users 



Sean,


if i understand correctly, your built a libtransport_mpi.so library that 
depends on Open MPI, and your main program dlopen libtransport_mpi.so.


in this case, and at least for the time being,  you need to use 
RTLD_GLOBAL in your dlopen flags.



Cheers,


Gilles


On 10/18/2016 4:53 AM, Sean Ahern wrote:

Folks,

For our code, we have a communication layer that abstracts the code 
that does the actual transfer of data. We call these "transports", and 
we link them as shared libraries. We have created an MPI transport 
that compiles/links against OpenMPI 2.0.1 using the compiler wrappers. 
When I compile OpenMPI with the--disable-dlopenoption (thus cramming 
all of OpenMPI's plugins into the MPI library directly), things work 
great with our transport shared library. But when I have a "normal" 
OpenMPI (without --disable-dlopen) and create the same transport 
shared library, things fail. Upon launch, it appears that OpenMPI is 
unable to find the appropriate plugins:


[hyperion.ceintl.com:25595 ]
mca_base_component_repository_open: unable to open
mca_patcher_overwrite:

/home/sean/work/ceisvn/apex/branches/OpenMPI/apex32/machines/linux_2.6_64/openmpi-2.0.1/lib/openmpi/mca_patcher_overwrite.so:
undefined symbol: *mca_patcher_base_patch_t_class* (ignored)
[hyperion.ceintl.com:25595 ]
mca_base_component_repository_open: unable to open mca_shmem_mmap:

/home/sean/work/ceisvn/apex/branches/OpenMPI/apex32/machines/linux_2.6_64/openmpi-2.0.1/lib/openmpi/mca_shmem_mmap.so:
undefined symbol: *opal_show_help* (ignored)
[hyperion.ceintl.com:25595 ]
mca_base_component_repository_open: unable to open
mca_shmem_posix:

/home/sean/work/ceisvn/apex/branches/OpenMPI/apex32/machines/linux_2.6_64/openmpi-2.0.1/lib/openmpi/mca_shmem_posix.so:
undefined symbol: *opal_show_help* (ignored)
[hyperion.ceintl.com:25595 ]
mca_base_component_repository_open: unable to open mca_shmem_sysv:

/home/sean/work/ceisvn/apex/branches/OpenMPI/apex32/machines/linux_2.6_64/openmpi-2.0.1/lib/openmpi/mca_shmem_sysv.so:
undefined symbol: *opal_show_help* (ignored)
--
It looks like opal_init failed for some reason; your parallel
process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS
--
--
It looks like orte_init failed for some reason; your parallel
process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_init failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--
--
It looks like MPI_INIT failed for some reason; your parallel
process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or
environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "Error" (-1) instead of "Success" (0)


If I skip our shared libraries and instead write a standard MPI-based 
"hello, world" program that links against MPI directly (without 
--disable-dlopen), everything is again fine.


It seems that having the double dlopenis causing problems for OpenMPI 
finding its own shared libraries.


Note: I do have LD_LIBRARY_PATHpointing to …"openmpi-2.0.1/

Re: [OMPI devel] RFC: Rename nightly snapshot tarballs

[Gilles Gouaillardet]

Jeff,


i have no strong objections against the new scheme.

that being said, i think i like the current one better ...

imho, the issue is the tag was put on a commit that is common to master 
and branch.



i guess we did :

git checkout master

git tag v2.x

git branch v2.x


what if we had done instead :
git checkout master
git checkout -b v2.x
git commit --allow-empty -m "forking the v2.x branch"
git tag v2.x


pro: i guess that would have produced the "expected" behavior.
cons: v2.x would be only on the v2.x branch, so the branch point is 
really the parent of the tag


regardless we want to change the way we tag, could we simply sed the git 
describe output instead ?

master : git describe | sed -e 's/v2.x-dev/master/g'
v2.x: git describe | sed -e 's/v2.0.1/v2.x/g'
v2.0.x: git describe | sed -e 's/v2.0.1/v2.0.x/g'
or
v2.x: git describe | sed -e 's/v2.0.1/v2.1.0a/g'
v2.0.x: git describe | sed -e 's/v2.0.1/v2.0.2a/g'


Cheers,

Gilles

On 10/18/2016 5:17 AM, Jeff Squyres (jsquyres) wrote:

SHORT VERSION
=

Our nightly snapshot tarball filenames have become confusing.  Here's the most 
recent on each branch:

- Master: openmpi-v2.x-dev-3019-gbd1b6fe.tar.bz2
- v2.0.x: openmpi-v2.0.1-93-g97511a1.tar.bz2
- v2.x:   openmpi-v2.0.1-203-g56991c6.tar.bz2

I propose changing them to the following format:

 openmpi-${BRANCHNAME}-${MMDD}-${SHORTHASH}.tar.bz2

Here's what last night's tarballs would have looked like with that scheme:

 openmpi-master-20161017-bd1b6fe.tar.bz2
 openmpi-v2.0.x-20161017-a66f3e2.tar.bz2
 openmpi-v2.x-20161017-56991c6.tar.bz2

Optionally, we could put a HHMM timestamp in there for finer granularity.

MORE DETAIL
===

We use "git describe" to come up with the version strings for our nightly snapshot 
tarballs.  "git describe" shows you three things:

- the first tag that it finds going back along the git commit history
- the number of commits it had to traverse to find that tag
- the short hast of the HEAD commit

For lack of a longer explanation, the names of the current snapshots *do* make 
sense given our commit and tag history; they're just... weird.

After talking this through with a colleague today, I think we have two choices:

1. Continue to use the "git describe" output
2. Come up with a different scheme

Using git describe output means that we are reliant on git tags.  Funky placement of git 
tags -- including the moving of branches back from ompi-release to ompi -- is how we 
ended up in this situation.  This might constitute empirical evidence that "git 
describe" is not a good solution for this community.

Instead, perhaps we should make a new scheme.  We need two properties in the 
snapshot filenames:

1. Name of the branch that the tarball came from.
2. Easily sortable by a human (i.e., know that tarball X came before or after 
tarball Y).

Property #1 is self-evident (the branch name is in the filename); property #2 
comes from the timestamp.

NOTE: It may be desirable to add HHMM in there; it's not common, but 
*sometimes* we do make more than one snapshot in a day (e.g., if one snapshot 
is borked, so we fix it and then generate another snapshot).



___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] RFC: Rename nightly snapshot tarballs

[r...@open-mpi.org]

You make a valid point - I too prefer simplicity to a 50-character-long name. 
There should be some simple way of making it clear which branch the tarball 
came from...your suggestions seem reasonable and easy to do. I’m sure we’ll be 
talking about this on the telecon in the morning.

> On Oct 17, 2016, at 8:56 PM, Gilles Gouaillardet  wrote:
> 
> Jeff,
> 
> 
> i have no strong objections against the new scheme.
> 
> that being said, i think i like the current one better ...
> 
> imho, the issue is the tag was put on a commit that is common to master and 
> branch.
> 
> 
> i guess we did :
> 
> git checkout master
> 
> git tag v2.x
> 
> git branch v2.x
> 
> 
> what if we had done instead :
> git checkout master
> git checkout -b v2.x
> git commit --allow-empty -m "forking the v2.x branch"
> git tag v2.x
> 
> 
> pro: i guess that would have produced the "expected" behavior.
> cons: v2.x would be only on the v2.x branch, so the branch point is really 
> the parent of the tag
> 
> regardless we want to change the way we tag, could we simply sed the git 
> describe output instead ?
> master : git describe | sed -e 's/v2.x-dev/master/g'
> v2.x: git describe | sed -e 's/v2.0.1/v2.x/g'
> v2.0.x: git describe | sed -e 's/v2.0.1/v2.0.x/g'
> or
> v2.x: git describe | sed -e 's/v2.0.1/v2.1.0a/g'
> v2.0.x: git describe | sed -e 's/v2.0.1/v2.0.2a/g'
> 
> 
> Cheers,
> 
> Gilles
> 
> On 10/18/2016 5:17 AM, Jeff Squyres (jsquyres) wrote:
>> SHORT VERSION
>> =
>> 
>> Our nightly snapshot tarball filenames have become confusing.  Here's the 
>> most recent on each branch:
>> 
>> - Master: openmpi-v2.x-dev-3019-gbd1b6fe.tar.bz2
>> - v2.0.x: openmpi-v2.0.1-93-g97511a1.tar.bz2
>> - v2.x:   openmpi-v2.0.1-203-g56991c6.tar.bz2
>> 
>> I propose changing them to the following format:
>> 
>> openmpi-${BRANCHNAME}-${MMDD}-${SHORTHASH}.tar.bz2
>> 
>> Here's what last night's tarballs would have looked like with that scheme:
>> 
>> openmpi-master-20161017-bd1b6fe.tar.bz2
>> openmpi-v2.0.x-20161017-a66f3e2.tar.bz2
>> openmpi-v2.x-20161017-56991c6.tar.bz2
>> 
>> Optionally, we could put a HHMM timestamp in there for finer granularity.
>> 
>> MORE DETAIL
>> ===
>> 
>> We use "git describe" to come up with the version strings for our nightly 
>> snapshot tarballs.  "git describe" shows you three things:
>> 
>> - the first tag that it finds going back along the git commit history
>> - the number of commits it had to traverse to find that tag
>> - the short hast of the HEAD commit
>> 
>> For lack of a longer explanation, the names of the current snapshots *do* 
>> make sense given our commit and tag history; they're just... weird.
>> 
>> After talking this through with a colleague today, I think we have two 
>> choices:
>> 
>> 1. Continue to use the "git describe" output
>> 2. Come up with a different scheme
>> 
>> Using git describe output means that we are reliant on git tags.  Funky 
>> placement of git tags -- including the moving of branches back from 
>> ompi-release to ompi -- is how we ended up in this situation.  This might 
>> constitute empirical evidence that "git describe" is not a good solution for 
>> this community.
>> 
>> Instead, perhaps we should make a new scheme.  We need two properties in the 
>> snapshot filenames:
>> 
>> 1. Name of the branch that the tarball came from.
>> 2. Easily sortable by a human (i.e., know that tarball X came before or 
>> after tarball Y).
>> 
>> Property #1 is self-evident (the branch name is in the filename); property 
>> #2 comes from the timestamp.
>> 
>> NOTE: It may be desirable to add HHMM in there; it's not common, but 
>> *sometimes* we do make more than one snapshot in a day (e.g., if one 
>> snapshot is borked, so we fix it and then generate another snapshot).
>> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel