Re: [Freesurfer] Building from source #2 -- dynamic binding

[Yaroslav Halchenko]

On Sun, 21 Feb 2016, R Edgar wrote:

> On 21 February 2016 at 22:33,   wrote:
> > Intolerable differences have not been found. They are always minor <%5 and
> > their source can come from several places. For example, different
> > compilers (or even different versions of the same compiler) can have
> > different sorting behavior when presented with objects that have
> > equivalent comparators. Others occur with data structures being
> > written/read to disk (Im still trying to fully understand those). As well
> > as a host of other potential differences as you mention below.

> > However, no cases have been found to fall outside expected and accepted
> > tolerance levels.

> That's a relief, but it makes me wonder about another point (and
> please understand that I'm making the following statement from a
> position of ignorance as to how Freesurfer is used in practice):

> Shouldn't differences like this be embraced? They are giving a measure
> of the error bars which should be applied to any results.

> I'd be interested in hearing people's thoughts about this.

> Going back to smaller and concrete things, I'm also curious as to the
> differences you say are creeping in when IO occurs. Is this happening
> for formatted or binary I/O?

Before commenting I want to state that this sub-thread is IMHO
off-topic, since I don't think it shouldn't (theoretically) have
to do anything with dynamic linking of internal code ...

IO issues also Z K mentioned also sounded interesting to my ear ;)
smells like alignment issues or uninitialized memory somewhere --
valgrind could be of great help here...  eventually, whenever a proper
debian package of freesurfer would see a daylight and there would be
some unit-tests, building/testing on exotic platforms such as
big-endians (powerpc, sparc, s390x) and the ones with stricter alignment
requirements (iirc armhf) can help to pin down locations of such bugs
since then e.g. 0x1 becomes 0x1 and thus sticking into your face ;)
(we have encounter such ones from time to time with e.g. numpy and
scipy)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Building from source #2 -- dynamic binding

[Yaroslav Halchenko]

On Sun, 21 Feb 2016, zkauf...@nmr.mgh.harvard.edu wrote:

> This topic gets brought up occasionally and their are valid arguments
> to both sides. One reason we have hesitated to use dynamic libs is the
> partly due to freesurfers long release cycle (all subjects that are
> part of a study need to be performed all on the same version). This
> long release cycle sometimes necessitates fixes to a particular binary
> in the stream, which users are then free to use. Although this is a
> less than ideal release strategy, it is the reality of the situation.
> And if we linked against dynamic libs than any time a binary was
> updated, ALL those libs, would need to be updated,
> which in turn would affect all binaries which link against them. 

AFAIK it is exactly the other way around ;) Please correct me if I
am wrong.

With static inclusion of code, if the fix is in the code which is shared
among binaries, you will need  to provide new copies for all those
effected binaries so they come with new copies of that code.   Only if
the fix is within code specific to the binary -- only that binary
indeed.

In case if that 'fixed' code being a part of an internal dynamic library
[*], you would need to provide only a copy of that library, and binaries
linked against it can stay original since they just dynamically load
that code from the library and do not carry broken code.   If it
is a fix to the code specific to the binary -- situation is just the
same as with static linking.

[*] under assumption that the fix doesn't entail changing  API/ABI.  In
case if those change -- indeed adjustment/rebuild of dependent binaries
would be necessary.  BUT such situations come much less often than just
fixes of the code without changing data structures and function
interfaces. And in your case, even if that happens, it is just a matter
again of uploading all those affected binaries (as you would do with
static linking) + the dynamic library.  And again, I think, even if a
binary uses some functionality of the library, but not a 'fixed'
function, it could as well stay without 'update' while linking to
the new dyn library.

That is the primary reason why Linux distributions rely on dynamic
libraries and reusing system-wide installed artifacts (e.g. java script
"libraries", Python modules, etc) as often as possible -- to fix a
vulnerability in the code of a core library/artifact requires just
upload of the fixed library/artifact without rebuilding all binaries (or
replacing all copies of artifacts) which could have potentially absorbed
that code via static "linking".  Could you imagine the chaos if
libc was statically included in every binary and then security fix
was needed to propagate to all 30,000 packages? ;)


> I suppose only newly released binaries could be static,
> but their may be unintended consequences that Im not thinking of at the
> moment.

> Im open to conversing about this, and appreciate any constructive feedback
> on improving our release model. 

Well -- I can't recommend anything new really, and just repeat my
whining: modularize just a bit (e.g. separate package for heavy data
pieces which rarely change + dynamic libraries, you already have that
bundle of externals you could deploy and reuse), and then you can
easily release not just "binary patches" but entire bugfix releases of
the codebase since they will probably be a fraction of the current size.
That is AFAIK how anything is released these days really ;)

NB. somewhat crazier scheme could be -- release binaries via git-annex,
with upgrades constituting "git pull; git annex get . " ;) It
would become especially efficient in case of dynamic internal
libraries, since fetching a bugfix release would be to fetch only
affected dynamic libraries and not all affected binaries linking against
them.

> But I would prefer to take the
> conversation offline as it gets overly technical rather quick.

ah (I was replying inline without reading all the way till the
end, sorry ;)), then this will be the last one then on this mailing list
in this series.  Could there may be a freesurfer-devel mailing list to
discuss such topics? ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose 

Re: [Freesurfer] Building from source #2 -- dynamic binding

[R Edgar]

On 21 February 2016 at 22:33,   wrote:
> Intolerable differences have not been found. They are always minor <%5 and
> their source can come from several places. For example, different
> compilers (or even different versions of the same compiler) can have
> different sorting behavior when presented with objects that have
> equivalent comparators. Others occur with data structures being
> written/read to disk (Im still trying to fully understand those). As well
> as a host of other potential differences as you mention below.
>
> However, no cases have been found to fall outside expected and accepted
> tolerance levels.

That's a relief, but it makes me wonder about another point (and
please understand that I'm making the following statement from a
position of ignorance as to how Freesurfer is used in practice):

Shouldn't differences like this be embraced? They are giving a measure
of the error bars which should be applied to any results.

I'd be interested in hearing people's thoughts about this.


Going back to smaller and concrete things, I'm also curious as to the
differences you say are creeping in when IO occurs. Is this happening
for formatted or binary I/O?

Thanks,

Richard
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Building from source #2 -- dynamic binding

[zkaufman]

Intolerable differences have not been found. They are always minor <%5 and
their source can come from several places. For example, different
compilers (or even different versions of the same compiler) can have
different sorting behavior when presented with objects that have
equivalent comparators. Others occur with data structures being
written/read to disk (Im still trying to fully understand those). As well
as a host of other potential differences as you mention below.

However, no cases have been found to fall outside expected and accepted
tolerance levels.

-Zeke




> On 20 February 2016 at 17:28, Bruce Fischl 
> wrote:
>
>> hmmm, the only thing that worries me about dynamic linking is that it
>> will add variability to the outputs. Zeke has spent endless amounts of
>> time
>> tracking down e.g. mac vs. pc differences in math libs and such.
>
> What sort of differences were found? Differences in transcendental
> functions are to be expected (so long as they are small) and lots of
> other things can cause slightly different results to be generated from
> floating point calculations (for instance, when summing an array, the
> result might vary with the number of OpenMP threads used). Were the
> segmentations produced intolerably different?
>
> Regards,
>
> Richard
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
>
>

___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Building from source #2 -- dynamic binding

[zkaufman]

This topic gets brought up occasionally and their are valid arguments to
both sides. One reason we have hesitated to use dynamic libs is the partly
due to freesurfers long release cycle (all subjects that are part of a
study need to be performed all on the same version). This long release
cycle sometimes necessitates fixes to a particular binary in the stream,
which users are then free to use. Although this is a less than ideal
release strategy, it is the reality of the situation. And if we linked
against dynamic libs than any time a binary was updated, ALL those libs
would need to be updated, which in turn would affect all binaries which
link against them. I suppose only newly released binaries could be static,
but their may be unintended consequences that Im not thinking of at the
moment.

Im open to conversing about this, and appreciate any constructive feedback
on improving our release model. But I would prefer to take the
conversation offline as it gets overly technical rather quick.

-Zeke

>
> On Sat, 20 Feb 2016, Bruce Fischl wrote:
>
>> hmmm, the only thing that worries me about dynamic linking is that it
>> will add variability to the outputs. Zeke has spent endless amounts of
>> time
>> tracking down e.g. mac vs. pc differences in math libs and such. Won't
>> dynamic linking just make that a much more prevalent problem?
>
> nope -- I am not talking about dynamically linking external libraries
> ATM, but first about the internal ones, where the common code where
> instead of absorbing the same .a or .o blobs into multiple binaries,
> there would be a set of internal dynamic libraries which those
> binaries would be linked against, so there will no copies of
> binary code among different binaries.  I guess
> https://www.gnu.org/software/automake/manual/html_node/Libtool-Convenience-Libraries.html
> is a quick intro into those.
>
> As for linking against external libraries -- those could be bundled
> along with rpath pointing to their location or LD_LIBRARY_PATH
> override assuring they are picked up (instead of possibly available
> identically named system-wide ones).
>
> So overall it is possible to achieve absent variability while using
> dynamic linking and allowing for possibility of the flexibility ;)
>
> Cheers
> --
> Yaroslav O. Halchenko
> Center for Open Neuroscience http://centerforopenneuroscience.org
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
> ___
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
>
>

___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Building from source #2 -- dynamic binding

[R Edgar]

On 20 February 2016 at 17:28, Bruce Fischl  wrote:

> hmmm, the only thing that worries me about dynamic linking is that it
> will add variability to the outputs. Zeke has spent endless amounts of time
> tracking down e.g. mac vs. pc differences in math libs and such.

What sort of differences were found? Differences in transcendental
functions are to be expected (so long as they are small) and lots of
other things can cause slightly different results to be generated from
floating point calculations (for instance, when summing an array, the
result might vary with the number of OpenMP threads used). Were the
segmentations produced intolerably different?

Regards,

Richard
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Building from source #2 -- dynamic binding

[Yaroslav Halchenko]

On Sat, 20 Feb 2016, Bruce Fischl wrote:

> hmmm, the only thing that worries me about dynamic linking is that it 
> will add variability to the outputs. Zeke has spent endless amounts of time 
> tracking down e.g. mac vs. pc differences in math libs and such. Won't 
> dynamic linking just make that a much more prevalent problem?

nope -- I am not talking about dynamically linking external libraries
ATM, but first about the internal ones, where the common code where
instead of absorbing the same .a or .o blobs into multiple binaries,
there would be a set of internal dynamic libraries which those
binaries would be linked against, so there will no copies of
binary code among different binaries.  I guess
https://www.gnu.org/software/automake/manual/html_node/Libtool-Convenience-Libraries.html
is a quick intro into those.

As for linking against external libraries -- those could be bundled
along with rpath pointing to their location or LD_LIBRARY_PATH
override assuring they are picked up (instead of possibly available
identically named system-wide ones). 

So overall it is possible to achieve absent variability while using
dynamic linking and allowing for possibility of the flexibility ;)

Cheers
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Re: [Freesurfer] Building from source #2 -- dynamic binding

[Bruce Fischl]

hmmm, the only thing that worries me about dynamic linking is that it 
will add variability to the outputs. Zeke has spent endless amounts of time 
tracking down e.g. mac vs. pc differences in math libs and such. Won't 
dynamic linking just make that a much more prevalent problem?
Bruce



On Sat, 20 Feb 2016, Yaroslav Halchenko 
wrote:

> Hi again,
>
> While our "building from source" discussion is "hot" I wondered to ask
> if there was an attempt or may be plans for to switch to using dynamic
> linking (and collect functionality within a few internal
> libraries) instead of static duplication of binary code across
> binaries.
>
> At some point Michael Hanke and me have tried to achieve such a goal:
> http://anonscm.debian.org/cgit/pkg-exppsy/freesurfer.git/tree/debian/patches/internal_shared_libs
> (although a bit also mixing up with linking against system wide
> libraries) so overall it is possible and should shrink currently sized
> at almost 4GB bin (stripped of debug symbols) into probably (forgot the
> size we got then) 50-100MB.  I expect similar or may be even more
> drastic effect on debug symbols files (which are useful to provide as an
> option).  I hope I don't need to outline why 10-fold cut down in
> binaries size would be a good thing ;)
>
> So I wondered, if there are any plans, and if not (yet) -- may be we
> could proceed together to achieve that goal?  above experiment with
> creating and maintaining that huge patch ourselves obviously has failed,
> but taking incremental steps we might succeed eventually, IFF such
> changes would be accepted upstream (i.e. into your code).  Or do
> you see any possible reason why assembling few of internal dynamic
> libraries and linking against your libraries bundle would be a no go?
>
> With best regards,
>
___
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.