Re: [fis-gtm] "action needed" items
On Thu, Mar 31, 2016 at 10:32:24PM +0200, Andreas Tille wrote: > Just uploaded. Will be in unstable after the next mirror sync. I forgot that due to the name change of the binary package it needs to pass the new queue. To my experience processing of the new queue is faster these days than for instance last year. Kind regards Andreas. -- http://fam-tille.de
Re: [fis-gtm] "action needed" items
Hi Amul,
On Thu, Mar 31, 2016 at 10:59:59AM -0400, Amul Shah wrote:
> Hi Andreas,
>
> On 03/31/16 06:05, Andreas Tille wrote:
> >On Wed, Mar 30, 2016 at 05:17:52PM -0400, Amul Shah wrote:
> >>FIS released GT.M V6.3-000 yesterday and I am in the process of updating the
> >>Debian package. Since I have the spare cycles, I want to address a few of
> >>the "action needed" items listed on https://tracker.debian.org/pkg/fis-gtm
> >Thanks for keeping the packages up to date.
> [amul:2] After the last round of updates, we instituted a few changes
> internally to ensure that we can ship the Debian package ASAP. Can you look
> over my recent commit and make any necessary changes? Also, when do can we
> push the new version into unstable?
Just uploaded. Will be in unstable after the next mirror sync.
The only thing I'd recommend to test is the following patch:
$ git diff
diff --git a/debian/rules b/debian/rules
index cef9eff..5415aa0 100755
--- a/debian/rules
+++ b/debian/rules
@@ -15,6 +15,8 @@ BINPKG := $(shell awk '/Package:.*[0-9]/{print $$2;}'
debian/control)
GTM_INSTALL_DIR := lib/$(MARCH)/fis-gtm/$(UAPIDIR)
LOCAL_GTM_INSTALL_DIR := $(CURDIR)/debian/$(BINPKG)/usr/$(GTM_INSTALL_DIR)
+export DEB_BUILD_MAINT_OPTIONS = hardening=+all
+
%:
dh $@ --parallel
This should reduce the number of lintian issues about hardening (when
using lintian with info severity).
> >>The above files are FIS GT.M database files generated during the build.
> >>These databases hold the online help for FIS GT.M executables. Database
> >>files won't be the same due to time related information in the block
> >>headers. So I need to exclude these files from being checked.
> >I wonder whether there would be any sensible chance to determine the
> >time stamp - may be for instance to the time stamp of the changelog.
> >Does GT.M provide any such functionality?
>
> [amul:2] I don't know of such a functionality. :(
Could you contact upstream about adding such a feature. If I would
design such a system I think this is kind of an essential feature for
instance if you want to design a test suite to be able to rely on a
defined state. BTW, what about creating a test suite that could be run
at build time as well as autopkgtest.
> >I do not think that you can exclude any files from beeing checked. I'd
> >recommend talking with upstream whether any fixed time setting would be
> >possible or the reproducible builds team whether they know any way to
> >create a fake-system-time.
>
> [amul:2] By upstream, do you mean FIS GT.M developers?
Yes, that's the usual Debian slang.
> There is a library
> libfaketime (https://github.com/wolfcw/libfaketime) that might help in this.
> I'll let you know.
That would be nice.
> >>-- build log check warning --
> >I admit I never cared about this and thus can't comment on this.
> [amul:2] Ok, I'll happily ignore this for now. :)
>
> As always, thanks for you help!
You are welcome
Andreas.
--
http://fam-tille.de
Re: [fis-gtm] "action needed" items
Hi Andreas, On 03/31/16 06:05, Andreas Tille wrote: On Wed, Mar 30, 2016 at 05:17:52PM -0400, Amul Shah wrote: FIS released GT.M V6.3-000 yesterday and I am in the process of updating the Debian package. Since I have the spare cycles, I want to address a few of the "action needed" items listed on https://tracker.debian.org/pkg/fis-gtm Thanks for keeping the packages up to date. [amul:2] After the last round of updates, we instituted a few changes internally to ensure that we can ship the Debian package ASAP. Can you look over my recent commit and make any necessary changes? Also, when do can we push the new version into unstable? report if they consider it an issue of packaging. Previously, when I looked at the non-reproducible build warnings, I saw a warning complaining about the following list of files: dsehelp.dat gdehelp.dat gtmhelp.dat lkehelp.dat mupiphelp.dat The above files are FIS GT.M database files generated during the build. These databases hold the online help for FIS GT.M executables. Database files won't be the same due to time related information in the block headers. So I need to exclude these files from being checked. I wonder whether there would be any sensible chance to determine the time stamp - may be for instance to the time stamp of the changelog. Does GT.M provide any such functionality? [amul:2] I don't know of such a functionality. :( I readhttps://wiki.debian.org/ReproducibleBuilds andhttps://reproducible-builds.org/docs to learn how to exclude these files from being checked, but could not find any mechanism. Most of the docs merely glorified the greatness* reproducible builds. Does anyone know a way to exclude these files? * I agree with it the principle, but I have an exception that I cannot work around. I do not think that you can exclude any files from beeing checked. I'd recommend talking with upstream whether any fixed time setting would be possible or the reproducible builds team whether they know any way to create a fake-system-time. [amul:2] By upstream, do you mean FIS GT.M developers? There is a library libfaketime (https://github.com/wolfcw/libfaketime) that might help in this. I'll let you know. -- build log check warning -- I admit I never cared about this and thus can't comment on this. [amul:2] Ok, I'll happily ignore this for now. :) As always, thanks for you help! Amul _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Parallel software performance analysis
Hello Jonathan, The Debian-Med team is considering plans for optimising large parts of the scientific software in Debian /en masse/. Many of the participants work for EU organisation. I'm sure they would appreciate your participation: https://lists.debian.org/debian-med/2016/03/msg00152.html Cheers, On Thu, Mar 31, 2016 at 3:58 PM, Jonathan Boyle wrote: > Dear All > > > > I’m currently involved in a EU funded project to help EU organisations > (e.g. academia and industry) improve performance of parallel software by > analysing performance and making recommendations on how to improve. These > services are free of charge to EU organisations. > > > > It’s important to realise we don’t offer to refactor code, and the code > analysis tools are designed for typical HPC software, e.g. MPI & OpenMP > running on Linux/Unix, which will undoubtedly place limits on who we can > help. > > > > This could be useful to someone developing or using parallel software so I > thought I’d send details to the list. There’s more information on the > website https://pop-coe.eu/services including how to apply. > > > > Please feel free to send on the information to anyone who might be > interested, or to get in touch with myself to discuss further. > > > > Best wishes > > Jonathan > > > > Jonathan Boyle > > HPC Application Analyst > > Numerical Algorithms Group (NAG) > > Peter House, Oxford Street, Manchester, M1 5AN > > +44 (0)161 602 3821 > > http://nag.co.uk/ > > > > > > > -- Michael R. Crusoe CWL Community Engineer cru...@ucdavis.edu Common Workflow Language projectUniversity of California, Davis https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe
Re: Proposal to start link time optimisation for Debian Med packages
[Steffen Möller] > @Petter, Holger: For packages featuring LTO, would you consider it > reasonable to run those twice in the CI, i.e. with and without the LTO > optimisation? I doubt ci.debian.net is the right tool for this, as it is supposed to run tests on the installed packages, not during build. Or did you suggest to provide two different packages, one with LTO and one without it? If there are two different packages or one packages with two different binaries, I do not see any problem with testing them using ci.debian.net. Perhaps jenkins.debian.net or the autobuilders are a better tool for such testing? -- Happy hacking Petter Reinholdtsen
Re: Proposal to start link time optimisation for Debian Med packages
On 31/03/16 02:34, Charles Plessy wrote: > Le Wed, Mar 30, 2016 at 01:36:58PM +0200, Steffen Möller a écrit : >> I had started this friendly and constructive thread on Debian Devel on >> link time optimisation >> >> https://lists.debian.org/debian-devel/2016/03/msg00399.html >> >> and my personal consensus is that we should possibly start with the most >> rewarding scientific packages of ours to see how it goes. What do you >> think? > Hi Steffen and everybody, > > here is a quick side comment, sorry for not having the time to participate > to that thread. > > Related to optimisation, we systematically override -O3 to -O2. Probably in > most of the cases the upstream authors never benchmarked the difference > anyway, > but in remaining cases, aren't we making the programs in our packages slower > only for the sake of building everything with the same flags and avoiding > compilation errors on architectures where nobody ever reported that these > programs in particular have been used ? > Hi Charles, I presume we kind of inadvertently do so by using the debhelper tools that may set other options than upstream had in mind. Er, yes, somehow this feels unfortunate, indeed. I would prefer to give this some extra thought in another thread, though. Many thanks and greetings Steffen
Re: [fis-gtm] "action needed" items
Hi Amul,
On Wed, Mar 30, 2016 at 05:17:52PM -0400, Amul Shah wrote:
> FIS released GT.M V6.3-000 yesterday and I am in the process of updating the
> Debian package. Since I have the spare cycles, I want to address a few of
> the "action needed" items listed on https://tracker.debian.org/pkg/fis-gtm.
Thanks for keeping the packages up to date.
> I made some changes to address the uscan error and lintian warnings, but I
> have some questions about two other items.
>
> -- non-reproducible builds --
> The link for this,
> https://tests.reproducible-builds.org/rb-pkg/testing/amd64/fis-gtm.html, is
> marked with a FTBFS for the second build. The problem with the second build
> seems to be a configuration problem on the build server. Notice the
> complaints (below) from PERL about LC_ALL. The recurring setlocale warnings
> seem to have caused problems for CMake resulting in a build failure.
> >I: using fakeroot in build.
> >I: pbuilder: network access will be disabled during build
> >I: Current time: Mon May 1 02:34:11 GMT-14 2017
> >I: pbuilder-time-stamp: 1493555651
> >I: Building the build Environment
> >I: extracting base tarball
> >[/var/cache/pbuilder/testing-reproducible-base.tgz]
> >I: copying local configuration
> >perl: warning: Setting locale failed.
> >perl: warning: Please check that your locale settings:
> > LANGUAGE = (unset),
> > LC_ALL = "fr_CH.UTF-8",
> > LANG = "fr_CH.UTF-8"
> > are supported and installed on your system.
> >perl: warning: Falling back to the standard locale ("C").
> See
> https://tests.reproducible-builds.org/logs/unstable/amd64/fis-gtm_6.2-002A-3.build2.log.gz
> for the full build log
Seems as if the Build server is located in French speaking part of
Swiss. :-) I'd also not really concerned about this. Sometimes I
needed to force a certain locale for some packages but I do not think
this is needed here.
> Do I need to take any action to address the above?
I do not think so. I guess the reproducible builds team would file a
bug report if they consider it an issue of packaging.
> Previously, when I looked at the non-reproducible build warnings, I saw a
> warning complaining about the following list of files:
> dsehelp.dat
> gdehelp.dat
> gtmhelp.dat
> lkehelp.dat
> mupiphelp.dat
>
> The above files are FIS GT.M database files generated during the build.
> These databases hold the online help for FIS GT.M executables. Database
> files won't be the same due to time related information in the block
> headers. So I need to exclude these files from being checked.
I wonder whether there would be any sensible chance to determine the
time stamp - may be for instance to the time stamp of the changelog.
Does GT.M provide any such functionality?
> I read https://wiki.debian.org/ReproducibleBuilds and
> https://reproducible-builds.org/docs to learn how to exclude these files
> from being checked, but could not find any mechanism. Most of the docs
> merely glorified the greatness* reproducible builds. Does anyone know a way
> to exclude these files? * I agree with it the principle, but I have an
> exception that I cannot work around.
I do not think that you can exclude any files from beeing checked. I'd
recommend talking with upstream whether any fixed time setting would be
possible or the reproducible builds team whether they know any way to
create a fake-system-time.
> -- build log check warning --
> The fis-gtm build was tagged with "W-compiler-flags-hidden". If I understood
> https://wiki.debian.org/Hardening#Notes_for_packages_using_CMake correctly,
> I should get dpkg-buildflags for free. Am I correct?
>
> The hardening options are in force.
> >shaha:~/debmed/fis-gtm> hardening-check
> >/usr/lib/x86_64-linux-gnu/fis-gtm/V6.3-000_x86_64/mumps
> >/usr/lib/x86_64-linux-gnu/fis-gtm/V6.3-000_x86_64/libgtmshr.so
> >/usr/lib/x86_64-linux-gnu/fis-gtm/V6.3-000_x86_64/mumps:
> > Position Independent Executable: no, normal executable!
> > Stack protected: yes
> > Fortify Source functions: yes
> > Read-only relocations: yes
> > Immediate binding: no, not found!
> >/usr/lib/x86_64-linux-gnu/fis-gtm/V6.3-000_x86_64/libgtmshr.so:
> > Position Independent Executable: no, regular shared library (ignored)
> > Stack protected: yes
> > Fortify Source functions: yes (some protected functions found)
> > Read-only relocations: yes
> > Immediate binding: no, not found!
>
>
> On a second read of
> https://qa.debian.org/bls/bytag/W-compiler-flags-hidden.html, I think I
> understand the complaint better.
> >buildd log scanner tag W-compiler-flags-hidden
> >
> >description
> >
> >The package contains build commands which hide the real compiler flags
> >(non-verbose builds). This prevents automatic checks for missing
> >(hardening) flags.
> >
> >False positives are possible, especially when building in parallel. In this
> >case this warning can be ignored.
> The complaint is that the build flags are not present in the build log file.
> Would the fix be to
Re: Proposal to start link time optimisation for Debian Med packages
Hi all, On 30.03.2016 22:28, Sascha Steinbiss wrote: >> The promotion of this enhancement I consider to be exceptionally >> important, especially so if we can tie this up with the continuous >> integration testing and some benchmarking. For all the folks that wait >> over some NGS data set to be aligned etc, a 20% reduction of compute >> time may mean that they see results a working day earlier. > > Indeed I am really curious about the level of improvement one can get. > In GenomeTools we did some experiments compiling the whole code base in > one big concatenated source file (an 'amalgamation') to allow the compiler > to optimise, inline etc. as much as possible. I'd consider this an approach > with even more optimisation potential than LTO. Looking at the run time of > demanding tasks (ESA generation, etc) we were IIRC only able to achieve ~10% > of run time improvement using this level of optimisation. YMMV. > A ~10% speed up is a huge improvement! Especially, as ESA generation is a task bound by memory accesses. Compilers usually have a hard time reasoning about those. If LTO or amalgation can help with that, a lot of bioinformatics applications would benefit from it. I'll try to come up with some hard numbers for andi and see it they are anywhere near your 10%. Best, Fabian
