Re: GATK

[Pierre Gruet]

Hi Andreas,

Le 14/07/2023 à 08:14, Andreas Tille a écrit :

Hi Pierre,


[...]
  

Right now I am preparing the move of my family some hundreds of kilometers
away, I expect to get more time to look at it closer afterwards.


Sure, Real life should have preference over volunteer work.  Is the
place where you move to a good location to have a developer sprint?
While I like the location in Berlin (with the chance to get even more
comfortable room) we can for sure rotate the sprint location a bit again
to possibly attract other people.


It's Lille, North of France. While the location is not the most 
difficult to reach, I still have to see if we could meet in some 
comfortable places, especially during weekends for the sprints.
Possibly something could be done in Brussels, which is not far, but 
still I guess we would have to know someone to get in.


In any case, I like the idea and think about it.

  

[...]


Kind regards
Andreas.



Have a great Sunday,

--
Pierre


OpenPGP_signature
Description: OpenPGP digital signature

Re: GATK

[Andreas Tille]

Hi Pierre,

Am Thu, Jul 13, 2023 at 10:23:10PM +0200 schrieb Pierre Gruet:
> Yeah, GATK is one of my main goals, with Nextflow. If I remember correctly,
> scala was a blocker but there might be a way to deal with it...

Thanks a lot!
 
> Right now I am preparing the move of my family some hundreds of kilometers
> away, I expect to get more time to look at it closer afterwards.

Sure, Real life should have preference over volunteer work.  Is the
place where you move to a good location to have a developer sprint?
While I like the location in Berlin (with the chance to get even more
comfortable room) we can for sure rotate the sprint location a bit again
to possibly attract other people.
 
> Still, as I said, GATK and Nextflow are my packaging goals in the team, and
> I am happy to say many dependencies have been added in the last year :)

Yes, and we have some time for the next release cycle.

Kind regards
   Andreas.

-- 
http://fam-tille.de

Re: GATK

[Pierre Gruet]

Hi Andreas,

Le 12/07/2023 à 15:01, Andreas Tille a écrit :

Hi Pierre,

I was wondering whether we might be able to package GATK for the next
release.  I've just fixed the watch file in Git.  It would be great if
you could have a look once you might find some spare cycles.


Yeah, GATK is one of my main goals, with Nextflow. If I remember 
correctly, scala was a blocker but there might be a way to deal with it...


Right now I am preparing the move of my family some hundreds of 
kilometers away, I expect to get more time to look at it closer afterwards.


Still, as I said, GATK and Nextflow are my packaging goals in the team, 
and I am happy to say many dependencies have been added in the last year :)




Kind regards
Andreas.



Best,

--
Pierre


OpenPGP_signature
Description: OpenPGP digital signature

GATK

[Andreas Tille]

Hi Pierre,

I was wondering whether we might be able to package GATK for the next
release.  I've just fixed the watch file in Git.  It would be great if
you could have a look once you might find some spare cycles.

Kind regards
   Andreas.

-- 
http://fam-tille.de

Bug#1014344: marked as done (ITP: gatk-bwamem -- interface to call Heng Li's bwa mem aligner from Java code)

[Debian Bug Tracking System]

Your message dated Fri, 22 Jul 2022 19:10:10 +
with message-id 
and subject line Bug#1014344: fixed in gatk-bwamem 1.0.4+dfsg2-1
has caused the Debian Bug report #1014344,
regarding ITP: gatk-bwamem -- interface to call Heng Li's bwa mem aligner from 
Java code
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)


-- 
1014344: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1014344
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: wnpp
Severity: wishlist
Owner: Debian-med team 
X-Debbugs-Cc: debian-de...@lists.debian.org, debian-med@lists.debian.org

* Package name: gatk-bwamem
  Version : 1.0.4
  Upstream Author : Broad Institute
* URL : https://github.com/broadinstitute/gatk-bwamem-jni/
* License : BSD-3-Clause
  Programming Lang: Java
  Description : interface to call Heng Li's bwa mem aligner from Java code

BWA (Burrows-Wheeler Aligner) is a software package for mapping low-divergent
sequences against a large reference genome, such as the human genome. It is
written in C.

gatk-bwamem provides a Java library and a shared library to allow one to use
BWA from Java code.
--- End Message ---
--- Begin Message ---
Source: gatk-bwamem
Source-Version: 1.0.4+dfsg2-1
Done: Pierre Gruet 

We believe that the bug you reported is fixed in the latest version of
gatk-bwamem, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1014...@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Pierre Gruet  (supplier of updated gatk-bwamem package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmas...@ftp-master.debian.org)


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Format: 1.8
Date: Wed, 06 Jul 2022 12:28:41 +0200
Source: gatk-bwamem
Binary: libgatk-bwamem-java libgatk-bwamem-jni libgatk-bwamem-jni-dbgsym
Architecture: source all amd64
Version: 1.0.4+dfsg2-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Med Packaging Team 

Changed-By: Pierre Gruet 
Description:
 libgatk-bwamem-java - interface to call Heng Li's bwa mem aligner from Java 
code
 libgatk-bwamem-jni - interface to call Heng Li's bwa mem aligner from Java 
code (jni)
Closes: 1014344
Changes:
 gatk-bwamem (1.0.4+dfsg2-1) unstable; urgency=medium
 .
   * Initial release (Closes: #1014344)
Checksums-Sha1:
 ff1bd4b1140d6a06e280668bc71dc52f1cbf3984 2498 gatk-bwamem_1.0.4+dfsg2-1.dsc
 514dfb2510d7c8d5d5f90707dc3f47dfeae7287b 144424 
gatk-bwamem_1.0.4+dfsg2.orig-bwa-apache2.tar.xz
 c27d5ad7ee506407448891b75f3d155bd795dd47 21360 
gatk-bwamem_1.0.4+dfsg2.orig.tar.xz
 f244f57a24e9d496013c9970c161e17b22967174 10152 
gatk-bwamem_1.0.4+dfsg2-1.debian.tar.xz
 7b8ba12cb821a417b4c02422908742f7f0b09674 11767 
gatk-bwamem_1.0.4+dfsg2-1_amd64.buildinfo
 8d37594fe2658466fea181366c6b3ac75b55c8b1 23312 
libgatk-bwamem-java_1.0.4+dfsg2-1_all.deb
 b5733304134dbcd5eb4b3688961692d720ed3227 212976 
libgatk-bwamem-jni-dbgsym_1.0.4+dfsg2-1_amd64.deb
 2fa6e551d5f3f955dda61b73b5d36338bd5da0b4 91064 
libgatk-bwamem-jni_1.0.4+dfsg2-1_amd64.deb
Checksums-Sha256:
 8f777c6858d9e618f9254bf03b35402fc4375115b37bb9631bbe1886a1c0c074 2498 
gatk-bwamem_1.0.4+dfsg2-1.dsc
 954d5d5aae624ce0faa4c9c3d30e91490dd93ce41b2205d84042c3391d35093a 144424 
gatk-bwamem_1.0.4+dfsg2.orig-bwa-apache2.tar.xz
 767c56d136ab24f1fc0511b4a5dcf927ba0028fa140c61e97da10dd57291d798 21360 
gatk-bwamem_1.0.4+dfsg2.orig.tar.xz
 4edb11ffa809fb686bf049cfee2f99870e8f375b82592abee8b29ce4bfb318fc 10152 
gatk-bwamem_1.0.4+dfsg2-1.debian.tar.xz
 b31cf3dd3ea7798109383d75b419fad5f6a283331dd9a9c2f2eb30b8472a00f9 11767 
gatk-bwamem_1.0.4+dfsg2-1_amd64.buildinfo
 7e80c2da0b09880014a044dea85114340f15459726990a7a1e84e3786399b7bf 23312 
libgatk-bwamem-java_1.0.4+dfsg2-1_all.deb
 5c648400edb5a773fe4c50fe11ca4773dea3e09a2b7ba0421b146a2863200a41 212976 
libgatk-bwamem-jni-dbgsym_1.0.4+dfsg2-1_amd64.deb
 8ca64e1ac61fad40624a44d1586a491370a3f634b5acdef7dacc55eea8d3ea51 91064 
libgatk-bwamem-jni_1.0.4+dfsg2-1_amd64.deb
Files:
 f09f03abc7ae4599c553b4aece527743 2498 java optional 
gatk-bwamem_1.0.4+dfsg2-1.dsc
 d0b90d3400ed4ccc469367c4dfbd9bca 144424 java optional 
gatk-bwamem_1.0.4+dfsg2.orig-bwa-apache2.tar.xz
 ed2b395e5416b1bf2b6902acedd61c80 21360 java optional 
gatk-bwamem_1.0.4+dfsg2

Bug#1014344: ITP: gatk-bwamem -- interface to call Heng Li's bwa mem aligner from Java code

[Pierre Gruet]

Package: wnpp
Severity: wishlist
Owner: Debian-med team 
X-Debbugs-Cc: debian-de...@lists.debian.org, debian-med@lists.debian.org

* Package name: gatk-bwamem
  Version : 1.0.4
  Upstream Author : Broad Institute
* URL : https://github.com/broadinstitute/gatk-bwamem-jni/
* License : BSD-3-Clause
  Programming Lang: Java
  Description : interface to call Heng Li's bwa mem aligner from Java code

BWA (Burrows-Wheeler Aligner) is a software package for mapping low-divergent
sequences against a large reference genome, such as the human genome. It is
written in C.

gatk-bwamem provides a Java library and a shared library to allow one to use
BWA from Java code.

Bug#1008128: marked as done (ITP: gatk-fermilite -- interface to call Heng Li's fermi-lite assembler from Java code)

[Debian Bug Tracking System]

Your message dated Tue, 05 Apr 2022 18:00:13 +
with message-id 
and subject line Bug#1008128: fixed in gatk-fermilite 1.2.1+dfsg-1
has caused the Debian Bug report #1008128,
regarding ITP: gatk-fermilite -- interface to call Heng Li's fermi-lite 
assembler from Java code
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)


-- 
1008128: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008128
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: wnpp
Severity: wishlist
Owner: Debian-med team 
X-Debbugs-Cc: debian-de...@lists.debian.org, debian-med@lists.debian.org

* Package name: gatk-fermilite
  Version : 1.2.1
  Upstream Author : Broad Institute
* URL : https://github.com/broadinstitute/gatk-fermilite-jni
* License : BSD-3-clause
  Programming Lang: Java, C
  Description : interface to call Heng Li's fermi-lite assembler from Java 
code

Fml-asm (fermi-lite assembler) is a command-line tool for assembling Illumina
short reads in regions from 100bp to 10 million bp in size, based on the
fermi-lite library.

gatk-fermilite provides a Java library and a shared library to allow one to use
fermilite from Java code.
 
The package will be team-maintained inside Debian-med team. It is needed as a
dependency of the packaging target gatk.
--- End Message ---
--- Begin Message ---
Source: gatk-fermilite
Source-Version: 1.2.1+dfsg-1
Done: Pierre Gruet 

We believe that the bug you reported is fixed in the latest version of
gatk-fermilite, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1008...@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Pierre Gruet  (supplier of updated gatk-fermilite package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmas...@ftp-master.debian.org)


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Format: 1.8
Date: Fri, 25 Mar 2022 21:43:37 +0100
Source: gatk-fermilite
Binary: libgatk-fermilite-java libgatk-fermilite-jni 
libgatk-fermilite-jni-dbgsym
Architecture: source all amd64
Version: 1.2.1+dfsg-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Med Packaging Team 

Changed-By: Pierre Gruet 
Description:
 libgatk-fermilite-java - interface to call Heng Li's fermi-lite assembler from 
Java code
 libgatk-fermilite-jni - interface to call Heng Li's fermi-lite assembler from 
Java code (
Closes: 1008128
Changes:
 gatk-fermilite (1.2.1+dfsg-1) unstable; urgency=medium
 .
   * Initial release (Closes: #1008128)
Checksums-Sha1:
 6567afc6ca03af777ed743e633f22ef9d4cc28a2 2209 gatk-fermilite_1.2.1+dfsg-1.dsc
 23608342ed1796d5e329645e6dcd8dcadbe4a3cd 12080 
gatk-fermilite_1.2.1+dfsg.orig.tar.xz
 e0824a4359577685210fc9c4d93830cc345d9ef2 7432 
gatk-fermilite_1.2.1+dfsg-1.debian.tar.xz
 7a0938d235be49838952f30a096df2da2152898f 11706 
gatk-fermilite_1.2.1+dfsg-1_amd64.buildinfo
 f9756d6383b7d9f87133858cd66785faa2aa71cc 11996 
libgatk-fermilite-java_1.2.1+dfsg-1_all.deb
 818c65da9d893440238f262180c72f5b6536e80d 12876 
libgatk-fermilite-jni-dbgsym_1.2.1+dfsg-1_amd64.deb
 5b8d95b80b66587e4bddec99336b15fbeef4e825 4704 
libgatk-fermilite-jni_1.2.1+dfsg-1_amd64.deb
Checksums-Sha256:
 65442ef1a9755ee84ba435871825fd832cb6a2c4ef854c83bbeacf84d2dc37fa 2209 
gatk-fermilite_1.2.1+dfsg-1.dsc
 c451f5e2a09a0472ec44d6707974086e6bbf8e5c02f35b619d2fc7bf5a503ab2 12080 
gatk-fermilite_1.2.1+dfsg.orig.tar.xz
 b419e1f12fae3086c3a511a2c2e186f1d99bba6c73352f13fc07929575b77541 7432 
gatk-fermilite_1.2.1+dfsg-1.debian.tar.xz
 7691cc4947e6165870d0261ce3b066d1a6703094418b10ff2a0fbb58ce842d79 11706 
gatk-fermilite_1.2.1+dfsg-1_amd64.buildinfo
 cd3555283cefdea1dd6806800aaf848da9e1d02a97ad0444a2e4f76a3b019775 11996 
libgatk-fermilite-java_1.2.1+dfsg-1_all.deb
 6fa9e5b0e926f232d55967d94086f3cabfeb9e1d7ac8f2a5597caaa12f94d43e 12876 
libgatk-fermilite-jni-dbgsym_1.2.1+dfsg-1_amd64.deb
 0acbe4cbdecd6d075db26d6d325f09d1ff1b2fa54969bb2c9ffe801546644dbd 4704 
libgatk-fermilite-jni_1.2.1+dfsg-1_amd64.deb
Files:
 9f3c6936c9c3269b2bd8c25b03faf405 2209 java optional 
gatk-fermilite_1.2.1+dfsg-1.dsc
 ccac31ec944a7bb326f0dcbd5e667cf7 12080 java optional 
gatk-fermilite_1.2.1+dfsg.orig.tar.xz
 aa9ad090abfc68501acdc2f1efb341ed 7432 java optional 
gatk-fermilite_1.2.1+dfsg-1.debian.tar.xz

Bug#1008128: ITP: gatk-fermilite -- interface to call Heng Li's fermi-lite assembler from Java code

[Pierre Gruet]

Package: wnpp
Severity: wishlist
Owner: Debian-med team 
X-Debbugs-Cc: debian-de...@lists.debian.org, debian-med@lists.debian.org

* Package name: gatk-fermilite
  Version : 1.2.1
  Upstream Author : Broad Institute
* URL : https://github.com/broadinstitute/gatk-fermilite-jni
* License : BSD-3-clause
  Programming Lang: Java, C
  Description : interface to call Heng Li's fermi-lite assembler from Java 
code

Fml-asm (fermi-lite assembler) is a command-line tool for assembling Illumina
short reads in regions from 100bp to 10 million bp in size, based on the
fermi-lite library.

gatk-fermilite provides a Java library and a shared library to allow one to use
fermilite from Java code.
 
The package will be team-maintained inside Debian-med team. It is needed as a
dependency of the packaging target gatk.

Re: Advice needed: building new gatk-bwamem-jni against another version of bwa

[Pierre Gruet]

Hi Andrius,

Thanks for looking at my issue.

Le 14/03/2022 à 06:03, Andrius Merkys a écrit :

Hi Pierre,

On 2022-03-13 16:35, Pierre Gruet wrote:

My proposal would be to design a multiple upstream tarball for
gatk-bwamem-jni: original one + the sources at the tip of the Apache2
branch of bwa. It would build a libbwa.a lib which would not be
installed in /usr/lib, but in a private directory of gatk-bwamem-jni.


Quick question. AFAIU, libbwa.a would be statically linked to
gatk-bwamem-jni binaries. So would there still be a need to install
libbwa.a anywhere?


Yes this is totally right, my mistake. In fact there would be no 
(alternative) libbwa.a to install, the link to the JNI shared library 
would indeed be static.


Thanks for pointing this out.



Best,
Andrius



Best,

--
Pierre

Re: Advice needed: building new gatk-bwamem-jni against another version of bwa

[Andrius Merkys]

Hi Pierre,

On 2022-03-13 16:35, Pierre Gruet wrote:
> My proposal would be to design a multiple upstream tarball for
> gatk-bwamem-jni: original one + the sources at the tip of the Apache2
> branch of bwa. It would build a libbwa.a lib which would not be
> installed in /usr/lib, but in a private directory of gatk-bwamem-jni.

Quick question. AFAIU, libbwa.a would be statically linked to
gatk-bwamem-jni binaries. So would there still be a need to install
libbwa.a anywhere?

Best,
Andrius

Re: Advice needed: building new gatk-bwamem-jni against another version of bwa

[Pierre Gruet]

Hi Nilesh,

Thanks for the quick answer. Your advice is much appreciated, as always.

Le 13/03/2022 à 15:46, Nilesh Patra a écrit :

On 3/13/22 8:05 PM, Pierre Gruet wrote:
What should I do? My proposal would be to design a multiple upstream 
tarball for gatk-bwamem-jni: original one + the sources at the tip of 
the Apache2 branch of bwa.> It would build a libbwa.a lib which would 
not be installed in /usr/lib, but in a private directory of 
gatk-bwamem-jni.
By doing so, I would not interfere with the currently Debian-packaged 
bwa and I would also be able to build, run and ship gatk-bwamem-jni... 
which would, still, be independent of the bwa that is shipped in Debian.

Does this seem sensible?


Introducing code copies in packages are bad for several reasons - for 
instance, possible security issues in the embedded copy that would

go into next stable release; and hence this should be the last resort.
Probably the better thing to do would be to instead talk to upstream 
about it and ask them to port the code to latest bwa versions.
If the ETA is sensible, it makes sense to wait; however if nothing else 
works, vendoring should work as you proposed.


Sure, security is a big issue and also the reason I asked here.
You're right, anyway upstream should be asked about it first. My 
ultimate packaging target is gatk, so I guess we have a bit of work 
before getting it: plenty of time to exchange with upstream.




If you want a multi-orig solution, please do it programmatically with 
d/watch and d/gbp.conf as for instance done in JS team[1]


[1]: https://wiki.debian.org/Javascript/GroupSourcesTutorial#Manual_way


Incidentally I have already worked on some MUT packages, so yes, I am 
used to do so!




Regards,
Nilesh



Best,

--
Pierre

Re: Advice needed: building new gatk-bwamem-jni against another version of bwa

[Nilesh Patra]

On 3/13/22 8:05 PM, Pierre Gruet wrote:

What should I do? My proposal would be to design a multiple upstream tarball for 
gatk-bwamem-jni: original one + the sources at the tip of the Apache2 branch of 
bwa.> It would build a libbwa.a lib which would not be installed in /usr/lib, 
but in a private directory of gatk-bwamem-jni.
By doing so, I would not interfere with the currently Debian-packaged bwa and I 
would also be able to build, run and ship gatk-bwamem-jni... which would, 
still, be independent of the bwa that is shipped in Debian.
Does this seem sensible?


Introducing code copies in packages are bad for several reasons - for instance, 
possible security issues in the embedded copy that would
go into next stable release; and hence this should be the last resort.
Probably the better thing to do would be to instead talk to upstream about it 
and ask them to port the code to latest bwa versions.
If the ETA is sensible, it makes sense to wait; however if nothing else works, 
vendoring should work as you proposed.

If you want a multi-orig solution, please do it programmatically with d/watch 
and d/gbp.conf as for instance done in JS team[1]

[1]: https://wiki.debian.org/Javascript/GroupSourcesTutorial#Manual_way

Regards,
Nilesh



OpenPGP_signature
Description: OpenPGP digital signature

Advice needed: building new gatk-bwamem-jni against another version of bwa

[Pierre Gruet]

Hello everyone,

I need your advice on a packaging issue I am meeting right now.
I am in the course of packaging gatk-bwamem-jni [0], providing a native 
interface which allows one to use bwa (Debian packaged [1]) from Java 
code. Fine.


But the upstream of gatk-bwamem-jni relies on a version of bwa which is 
the tip of a -- now stale -- branch called "Apache2" in the Github 
repository of bwa. It cannot build against the Debian-packaged 
libbwa-dev as differences between the bwa branches "Apache2" and 
"master" are now too important.


What should I do? My proposal would be to design a multiple upstream 
tarball for gatk-bwamem-jni: original one + the sources at the tip of 
the Apache2 branch of bwa. It would build a libbwa.a lib which would not 
be installed in /usr/lib, but in a private directory of gatk-bwamem-jni.
By doing so, I would not interfere with the currently Debian-packaged 
bwa and I would also be able to build, run and ship gatk-bwamem-jni... 
which would, still, be independent of the bwa that is shipped in Debian.

Does this seem sensible?

Thanks a lot for your attention and help.

Best regards,

--
Pierre

[0] https://github.com/broadinstitute/gatk-bwamem-jni
[1] https://tracker.debian.org/pkg/bwa


OpenPGP_signature
Description: OpenPGP digital signature

About gatk -- git-lfs and dependencies to be packaged

[Pierre Gruet]

Hi everyone,

Recently there has been quite a lot of movement around the packaging of 
gatk:


- I listed the dependencies to be packaged in our common spreadsheet, in 
the Java sheet, on lines 61 to 78. I think some of them should reveal 
optional, but I also see at some point we will be blocked by the need 
for Scala 2.12, whereas we only have 2.11 in Debian. This is a blocker 
we have already met on other packages;


- Nilesh and Yadd could find a new formulation for debian/watch which 
seems to enable us to use the git-lfs features to get the whole source 
tarball:


cat debian/watch

version=4
 opts="mode=git, gitmode=full, pgpmode=none" \
 https://github.com/broadinstitute/gatk.git \
 refs/tags/v?([\d\.]+) debian

Yet I gave it a try and the whole source is approx. 9.5 GB large and the 
(not repacked) orig tarball is 3.7 GB large. Thus we should do some work 
to strip a lot of big files, which I think would amount to deactivating 
some tests.


- Steffen and I looked at the dependencies list I discussed above and we 
wondered if getting gatk packaged now was worth the effort...
In any case, the Scala issue should be fixed, maybe we should have a 
discussion with the Scala team to see how we could stop being stuck with 
our old Debian-packaged Scala.



Best,

--
Pierre

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Andreas Tille]

Hi Steffen,

On Sat, Nov 14, 2020 at 10:44:36PM +0100, Steffen Möller wrote:
> 
> https://www.bioconductor.org/packages/release/bioc/html/TitanCNA.html

Uploaded to new.

> https://bioconductor.org/packages/release/bioc/html/PureCN.html

Pushed to Git but 3 remaining errors in autopkgtest.

> How are we with more R packages while the transition is happening?
> Please ping me when you think I should address their packaging.

May be you can check the autopkgtest and exclude the responsible
tests.  I will probably not do much over the weekend.

Kind regards

  Andreas. 

-- 
http://fam-tille.de

Re: Progress of Snpeff packaging (Was: Re: /usr/bin/picard Re: bcbio will need another while - needs gatk)

[Andreas Tille]

Hi Pierre,

thanks a lot for your continuous work on snpeff!

On Thu, Nov 19, 2020 at 08:34:10PM +0100, Pierre Gruet wrote:
> 
> It's good we talk about it. There were 4 identified missing packages to
> be able to complete the packaging of Snpeff; 3 of them are now in
> Debian, only akka-actor misses.
> 
> akka-actor is part of akka, this is a framework for concurrent
> programming written in Scala. And here is the problem: we have Scala
> 2.11 in Debian, current upstream version is 2.13 and akka would need at
> least 2.12. Also, Scala 2.12 and 2.13 would need Scala Build Tool (sbt)
> in order to be built, and sbt in turns requires Scala 2.12 or 2.13.
> 
> I guess at some point, this loop did not exist and there might be an old
> version of either sbt or Scala which could be built with what we
> currently have in Debian. But this is quite a lot of work, and I feel no
> one is willing to do it now -- perhaps that Scala thing is quite
> peculiar and we would need someone with time and high skills.

I remember I had some issues with scala when trying to package pilon.
There was some workaround which relaxed the dependency of sbt - but
I remember sbt was a can of worms.  But I guess someone should open
it sooner or later. :-(
 
> In September I exchanged a few emails with a colleague of Steffen, who
> knows Scala. While he helped a lot on understanding some aspects of the
> language, he does not know about the Debian workflow -- which is plainly
> understandable -- and thus we are, so far, left with the current issue.

It would be great if we could teach some Scala expert about Debian
packaging.  I'd love to do this.

> Maybe we should see if there is another framework for concurrent
> programming in pure Java that we could fit into Snpeff by writing patches...

That's beyond my knowledge - but you've found always some practical
solution and I fully trust you. :-)

Kind regards

   Andreas.

-- 
http://fam-tille.de

Progress of Snpeff packaging (Was: Re: /usr/bin/picard Re: bcbio will need another while - needs gatk)

[Pierre Gruet]

Hi,

Le 15/11/2020 à 18:16, Andreas Tille a écrit :
> 
>>
>> This may be worth a sidenote - bcbio to me is something metaphorical. We
>> have it in the distribution already. And now we work to get all the
>> runtime-dependencies in to make it functional. And snpEff is pretty high
>> up (it interprets the importance of nucleotide variations, you need to
>> have identified these variants for that, first). So, my comment on
>> "snpeff" being invoked was to be interpreted as "see, here we are already".
> 
> Snpeff is wanted from several sided.  I really hope we get it soon.
>

It's good we talk about it. There were 4 identified missing packages to
be able to complete the packaging of Snpeff; 3 of them are now in
Debian, only akka-actor misses.

akka-actor is part of akka, this is a framework for concurrent
programming written in Scala. And here is the problem: we have Scala
2.11 in Debian, current upstream version is 2.13 and akka would need at
least 2.12. Also, Scala 2.12 and 2.13 would need Scala Build Tool (sbt)
in order to be built, and sbt in turns requires Scala 2.12 or 2.13.

I guess at some point, this loop did not exist and there might be an old
version of either sbt or Scala which could be built with what we
currently have in Debian. But this is quite a lot of work, and I feel no
one is willing to do it now -- perhaps that Scala thing is quite
peculiar and we would need someone with time and high skills.

In September I exchanged a few emails with a colleague of Steffen, who
knows Scala. While he helped a lot on understanding some aspects of the
language, he does not know about the Debian workflow -- which is plainly
understandable -- and thus we are, so far, left with the current issue.

Maybe we should see if there is another framework for concurrent
programming in pure Java that we could fit into Snpeff by writing patches...

>  
> Kind regards
> 
>Andreas. 
> 

Bye,
Pierre

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Andreas Tille]

Hi Steffen,

On Sun, Nov 15, 2020 at 02:39:03PM +0100, Steffen Möller wrote:
> >> I just checked what my installation has in this directory ... and it
> >> seems like I spotted a typo (or a creative fix of a typo elsewhere)
> > ?
> tranlate misses the "s"

Fixed in new upload.  BTW, it would be perfectly fine if you simply
fix such things in a team upload.

> >> I presume you would be happy for me to put there a link from cnvkit.py
> >> (like bcbio expects it) to /usr/bin/cnvkit, right?
> > Well, I think its orthogonal to my own happyness.  If it works for
> > you just do the same as in the given example package.
> I added that symlink and then patched bcbio to not expect the .py since
> bcbio ignored the $PATH on this one.

Fine for me.  Regarding the .py extension I became quite relaxed.  See
our package template:

   
https://salsa.debian.org/med-team/community/package_template/-/blob/master/debian/lintian-overrides

> > I think we are pretty close to snpEff due to Pierre's efforts.
> 
> Nice!
> 
> This may be worth a sidenote - bcbio to me is something metaphorical. We
> have it in the distribution already. And now we work to get all the
> runtime-dependencies in to make it functional. And snpEff is pretty high
> up (it interprets the importance of nucleotide variations, you need to
> have identified these variants for that, first). So, my comment on
> "snpeff" being invoked was to be interpreted as "see, here we are already".

Snpeff is wanted from several sided.  I really hope we get it soon.
 
Kind regards

   Andreas. 

-- 
http://fam-tille.de

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Steffen Möller]

Hi Andreas,

On 14.11.20 21:50, Andreas Tille wrote:
> Hi Steffen,
>
> On Sat, Nov 14, 2020 at 08:18:48PM +0100, Steffen Möller wrote:
>>> We have the workaround
>>>
>>>  /usr/lib/debian-med/bin/
>>>
>>> see for instance in eigensoft[1] and lots of other packages.  Simply
>>> provide this for picard-tools and make sure gatk users set the PATH
>>> accordingly.
>> Ah - that is good. Seems like I should read our policy document again.
> Not sure whether it is mentioned in policy.  I introduced this 2 or 3
> sprints ago.  Finally you are Uploader of at least one package using
> this technique. ;-P
;)
>> I just checked what my installation has in this directory ... and it
>> seems like I spotted a typo (or a creative fix of a typo elsewhere)
> ?
tranlate misses the "s"
>
>> $ ls -l /usr/lib/debian-med/bin/
>> total 0
>> lrwxrwxrwx 1 root root 26 Nov 12 16:57 tranlate ->
>> ../../../bin/fsa-translate
>>
>> $ apt-file search tranlate
>> fsa: /usr/lib/debian-med/bin/tranlate
>> $ apt-file search /usr/bin/translate
>> drslib: /usr/bin/translate_cmip3
>> openafs-client: /usr/bin/translate_et
>> translate: /usr/bin/translate
>>
>> I presume you would be happy for me to put there a link from cnvkit.py
>> (like bcbio expects it) to /usr/bin/cnvkit, right?
> Well, I think its orthogonal to my own happyness.  If it works for
> you just do the same as in the given example package.
I added that symlink and then patched bcbio to not expect the .py since
bcbio ignored the $PATH on this one.
>>> Simply ping the authors about this.  As far as I remember the last
>>> status was that sources are lost (?) and a backup needs to be found.
>> And if we just go for a non-free binary package for those jars? Just to
>> get somewhere?
> I personally consider maintaining non-free packages a pure nuisance and
> I'm not motivated to maintain such a package.  I think we should do
> *way* more effort to free *all* our non-free packages.
>
>> bcbio is in contrib anyway because of vienna-rna.
> Same here.  I do not see any record of attempts to free vienna-rna.
https://github.com/ViennaRNA/ViennaRNA/issues/97
>> And I do not think
>> these .jar files
>> will find much future adoption without a source backing, so this problem
>> will eradicate
>> itself.
> I will not try to stop anybody to maintain non-free packages - just do
> not trust on me in doing so.
>
>> There is also
>> bcbio.pipeline.config_utils.CmdNotFound: '_get_program_cmd' 'snpEff'
>> {'dir': '/usr/local/share/java/snpeff', 'jvm_opts': ['-Xms750m',
>> '-Xmx3g']} None
> I think we are pretty close to snpEff due to Pierre's efforts.

Nice!

This may be worth a sidenote - bcbio to me is something metaphorical. We
have it in the distribution already. And now we work to get all the
runtime-dependencies in to make it functional. And snpEff is pretty high
up (it interprets the importance of nucleotide variations, you need to
have identified these variants for that, first). So, my comment on
"snpeff" being invoked was to be interpreted as "see, here we are already".

>> ... picard-tools stuff snipped - I have nothing to say here ...
Works now.
>> Somehow Debian does barely complete any of these tests. There is always
>> something missing. And if it is tophat that upstream had asked us not to
>> support any more.
> I asked *several* times whether we need tophat and consequently will do
> a Python3 port.  Its perfectly fine for me to re-introduce tophat once
> it works with Python3.

Do other things first. I would not be completely surprised if this is
gone in one of the next updates to bcbio.

Steffen

Re: [RFS] gsort and dependencies (Was: /usr/bin/picard Re: bcbio will need another while - needs gatk)

[Steffen Möller]

Hi Nilesh,

Impressive as usual.

On 15.11.20 12:17, Nilesh Patra wrote:
> Hi,
>
> I packaged gsort and the rest of the dependency chain as needed by
> bcbio. Everything seems to build+pass its autopkgtests.
>
> The dependency chain is as follows:
>
> 1. golang-github-alexflint-go-scalar [1]
> 2. golang-github-alexflint-go-arg [2]    (depends on [1])
> 3. ggd-utils [3]   (depends on [2])
> 4. gsort [4]   (depends on [3] and [2])
>
> Please consider to review + upload to NEW
>
> PS:
> Simple gbp-buildpackage will not work on [1] since this follows new
> go-style guideline which drops pristine-tar[5]
> Hence the way here would be to run origtargz and run pdebuild w/o gbp
> You probably knew about this, but I re-iterated just in case you're
> not used to the new workflow for go-team and this saves your time
> while building this.
> Please consider my best of intentions here.
>
> [1]:
> https://salsa.debian.org/go-team/packages/golang-github-alexflint-go-scalar
> 
Just uploaded [1].
> [2]:
> https://salsa.debian.org/go-team/packages/golang-github-alexflint-go-arg
> 
> [3]: https://salsa.debian.org/med-team/ggd-utils
> 
> [4]: https://salsa.debian.org/med-team/gsort/
> 
> [5]:
> https://go-team.pages.debian.net/workflow-changes.html#wf-2017-11-pristine-tar
> 

With our short new queue I think I go back to waiting for a package's
acceptance prior to uploading a dependency.

Well done!

Steffen

[RFS] gsort and dependencies (Was: /usr/bin/picard Re: bcbio will need another while - needs gatk)

[Nilesh Patra]

Hi,

I packaged gsort and the rest of the dependency chain as needed by bcbio.
Everything seems to build+pass its autopkgtests.

The dependency chain is as follows:

1. golang-github-alexflint-go-scalar [1]
2. golang-github-alexflint-go-arg [2](depends on [1])
3. ggd-utils [3]   (depends on [2])
4. gsort [4]   (depends on [3] and [2])

Please consider to review + upload to NEW

PS:
Simple gbp-buildpackage will not work on [1] since this follows new
go-style guideline which drops pristine-tar[5]
Hence the way here would be to run origtargz and run pdebuild w/o gbp
You probably knew about this, but I re-iterated just in case you're not
used to the new workflow for go-team and this saves your time while
building this.
Please consider my best of intentions here.

[1]:
https://salsa.debian.org/go-team/packages/golang-github-alexflint-go-scalar
[2]:
https://salsa.debian.org/go-team/packages/golang-github-alexflint-go-arg
[3]: https://salsa.debian.org/med-team/ggd-utils
[4]: https://salsa.debian.org/med-team/gsort/
[5]:
https://go-team.pages.debian.net/workflow-changes.html#wf-2017-11-pristine-tar

Nilesh

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Nilesh Patra]

On Sun, 15 Nov 2020 at 02:34, Andreas Tille  wrote:

> On Sun, Nov 15, 2020 at 01:21:05AM +0530, Nilesh Patra wrote:
> >
> > I did have a look earlier today.
> > gsort has dependency on unpackaged packages - "
> > https://github.com/alexflint/go-arg; and "
> > https://github.com/gogetdata/ggd-utils;
> >
> > and "https://github.com/alexflint/go-arg; has dep on "
> > https://github.com/alexflint/go-scalar; - which is again, not in the
> > archive.
> >
> > I however see alexflint/go-scalar packaged in salsa[1], and even uploaded
> > but not in the archive and it even has an open ITP[2].
> > I've no idea if we should go about uploading this or not.
> > I'd need your ACK/advice + sponsorship on this.
>
> I usually seek contact - for instance write to the ITP - and announce
> what you will do (for instance that you will ask for sponsored upload in
> 24 hours) and assume agreement if you do not hear anything.
>

I replied to the ITP and CC'd both you and Steffen. Please consider
uploading this when you have time.
I'm then going ahead with packaging the rest of chain - I hope this is OK?

PS: Simple gbp-buildpackage will not work on this package since this
follows new go-style guideline which drops pristine-tar[1]
Hence the way here would be to run origtargz and run pdebuild w/o gbp
You probably knew about this, but I re-iterated just in case you're not
used to the new workflow for go-team and this saves your time while
building this.
Please consider my best of intentions here.

[1]:
https://go-team.pages.debian.net/workflow-changes.html#wf-2017-11-pristine-tar

Nilesh

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Steffen Möller]

On 14.11.20 20:18, Steffen Möller wrote:
> I presume you would be happy for me to put there a link from cnvkit.py
> (like bcbio expects it) to /usr/bin/cnvkit, right?

Update: I patched bcbio to find it. While looking at the source tree of
the cnvkit wrapper of bcbio I found that it references two BioConductor
packages that I hence put on the spreadsheet:

https://www.bioconductor.org/packages/release/bioc/html/TitanCNA.html
https://bioconductor.org/packages/release/bioc/html/PureCN.html

How are we with more R packages while the transition is happening?
Please ping me when you think I should address their packaging.

Best,

Steffen

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Steffen Möller]

Hi Nilesh,

On 14.11.20 20:51, Nilesh Patra wrote:
>
> On Sun, 15 Nov 2020 at 00:49, Steffen Möller  > wrote:
>
> >> Should we have someone among us for who packaging go-based software
> >> feels like easy then the newly surfaced
> https://github.com/brentp/gsort 
> >> would be nice.
> > Nilesh did so several times.
> Wow! @Nilesh, please have a look.
>
>
> I did have a look earlier today.
> gsort has dependency on unpackaged packages -
> "https://github.com/alexflint/go-arg
> " and
> "https://github.com/gogetdata/ggd-utils
> "
>
> and "https://github.com/alexflint/go-arg
> " has dep on
> "https://github.com/alexflint/go-scalar
> " - which is again, not in the
> archive.
>
> I however see alexflint/go-scalar packaged in salsa[1], and even
> uploaded but not in the archive and it even has an open ITP[2].
> I've no idea if we should go about uploading this or not.
> I'd need your ACK/advice + sponsorship on this.
>
> PS: I tried contacting its original Uploader Balasankar C. on IRC,
> since he's kind of active around a few rooms we have in common but I
> didn't get a response from them yet.
>
> [1]:
> https://salsa.debian.org/go-team/packages/golang-github-alexflint-go-scalar
> 
> [2]: http://bugs.debian.org/926079 

This is all good news.

I suggest you go for ggd-utils and write an eMail to the aspiring
maintainer of go-scalar, explaining what you are after.

And then let's see how this develops. On a weekend ... not unlikely to
get a late reply.

Thank you tons!

Steffen

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Andreas Tille]

On Sun, Nov 15, 2020 at 01:21:05AM +0530, Nilesh Patra wrote:
> 
> I did have a look earlier today.
> gsort has dependency on unpackaged packages - "
> https://github.com/alexflint/go-arg; and "
> https://github.com/gogetdata/ggd-utils;
> 
> and "https://github.com/alexflint/go-arg; has dep on "
> https://github.com/alexflint/go-scalar; - which is again, not in the
> archive.
> 
> I however see alexflint/go-scalar packaged in salsa[1], and even uploaded
> but not in the archive and it even has an open ITP[2].
> I've no idea if we should go about uploading this or not.
> I'd need your ACK/advice + sponsorship on this.

I usually seek contact - for instance write to the ITP - and announce
what you will do (for instance that you will ask for sponsored upload in
24 hours) and assume agreement if you do not hear anything.
 
> PS: I tried contacting its original Uploader Balasankar C. on IRC, since
> he's kind of active around a few rooms we have in common but I didn't get a
> response from them yet.

Getting no response might happen but this should not stop progress.

Kind regards

   Andreas.
 
> [1]:
> https://salsa.debian.org/go-team/packages/golang-github-alexflint-go-scalar
> [2]: http://bugs.debian.org/926079
> 
> Nilesh

-- 
http://fam-tille.de

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Andreas Tille]

Hi Steffen,

On Sat, Nov 14, 2020 at 08:18:48PM +0100, Steffen Möller wrote:
> > We have the workaround
> >
> >  /usr/lib/debian-med/bin/
> >
> > see for instance in eigensoft[1] and lots of other packages.  Simply
> > provide this for picard-tools and make sure gatk users set the PATH
> > accordingly.
> 
> Ah - that is good. Seems like I should read our policy document again.

Not sure whether it is mentioned in policy.  I introduced this 2 or 3
sprints ago.  Finally you are Uploader of at least one package using
this technique. ;-P
 
> I just checked what my installation has in this directory ... and it
> seems like I spotted a typo (or a creative fix of a typo elsewhere)

?
 
> $ ls -l /usr/lib/debian-med/bin/
> total 0
> lrwxrwxrwx 1 root root 26 Nov 12 16:57 tranlate ->
> ../../../bin/fsa-translate
> 
> $ apt-file search tranlate
> fsa: /usr/lib/debian-med/bin/tranlate
> $ apt-file search /usr/bin/translate
> drslib: /usr/bin/translate_cmip3
> openafs-client: /usr/bin/translate_et
> translate: /usr/bin/translate
> 
> I presume you would be happy for me to put there a link from cnvkit.py
> (like bcbio expects it) to /usr/bin/cnvkit, right?

Well, I think its orthogonal to my own happyness.  If it works for
you just do the same as in the given example package.
 
> > Simply ping the authors about this.  As far as I remember the last
> > status was that sources are lost (?) and a backup needs to be found.
> 
> And if we just go for a non-free binary package for those jars? Just to
> get somewhere?

I personally consider maintaining non-free packages a pure nuisance and
I'm not motivated to maintain such a package.  I think we should do
*way* more effort to free *all* our non-free packages.
 
> bcbio is in contrib anyway because of vienna-rna.

Same here.  I do not see any record of attempts to free vienna-rna.

> And I do not think
> these .jar files
> will find much future adoption without a source backing, so this problem
> will eradicate
> itself.

I will not try to stop anybody to maintain non-free packages - just do
not trust on me in doing so.

> There is also
> bcbio.pipeline.config_utils.CmdNotFound: '_get_program_cmd' 'snpEff'
> {'dir': '/usr/local/share/java/snpeff', 'jvm_opts': ['-Xms750m',
> '-Xmx3g']} None

I think we are pretty close to snpEff due to Pierre's efforts.

> ... picard-tools stuff snipped - I have nothing to say here ...

> Somehow Debian does barely complete any of these tests. There is always
> something missing. And if it is tophat that upstream had asked us not to
> support any more.

I asked *several* times whether we need tophat and consequently will do
a Python3 port.  Its perfectly fine for me to re-introduce tophat once
it works with Python3.
 
Kind regards

 Andreas.

-- 
http://fam-tille.de

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Nilesh Patra]

Hi Andreas and Steffen

On Sun, 15 Nov 2020 at 00:49, Steffen Möller  wrote:

> >> Should we have someone among us for who packaging go-based software
> >> feels like easy then the newly surfaced https://github.com/brentp/gsort
> >> would be nice.
> > Nilesh did so several times.
> Wow! @Nilesh, please have a look.
>

I did have a look earlier today.
gsort has dependency on unpackaged packages - "
https://github.com/alexflint/go-arg; and "
https://github.com/gogetdata/ggd-utils;

and "https://github.com/alexflint/go-arg; has dep on "
https://github.com/alexflint/go-scalar; - which is again, not in the
archive.

I however see alexflint/go-scalar packaged in salsa[1], and even uploaded
but not in the archive and it even has an open ITP[2].
I've no idea if we should go about uploading this or not.
I'd need your ACK/advice + sponsorship on this.

PS: I tried contacting its original Uploader Balasankar C. on IRC, since
he's kind of active around a few rooms we have in common but I didn't get a
response from them yet.

[1]:
https://salsa.debian.org/go-team/packages/golang-github-alexflint-go-scalar
[2]: http://bugs.debian.org/926079

Nilesh

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Steffen Möller]

Hi Andreas,

On 14.11.20 09:43, Andreas Tille wrote:
> On Fri, Nov 13, 2020 at 10:25:39PM +0100, Steffen Möller wrote:
>> I installed a binary distribution of gatk and then ran a bit into
>> trouble over
>>
>> $ apt-file search /usr/bin/picard
>> picard: /usr/bin/picard
>> picard-tools: /usr/bin/picard-tools
>>
>> I tend to think that this exposes a weakness of Debian - it misses
>> namespaces. Others may say that it is a strength. Don't think so. You
>> want to use the same scripts that your scientific peer uses. Everything
>> else goes under "the distribution has patches, the work may not be
>> completely reproducible" - because a log file has a different hash value
>> since executions of /usr/bin/picard were changed to /usr/bin/picard-tools.
> We have the workaround
>
>  /usr/lib/debian-med/bin/
>
> see for instance in eigensoft[1] and lots of other packages.  Simply
> provide this for picard-tools and make sure gatk users set the PATH
> accordingly.

Ah - that is good. Seems like I should read our policy document again.

I just checked what my installation has in this directory ... and it
seems like I spotted a typo (or a creative fix of a typo elsewhere)

$ ls -l /usr/lib/debian-med/bin/
total 0
lrwxrwxrwx 1 root root 26 Nov 12 16:57 tranlate ->
../../../bin/fsa-translate

$ apt-file search tranlate
fsa: /usr/lib/debian-med/bin/tranlate
$ apt-file search /usr/bin/translate
drslib: /usr/bin/translate_cmip3
openafs-client: /usr/bin/translate_et
translate: /usr/bin/translate

I presume you would be happy for me to put there a link from cnvkit.py
(like bcbio expects it) to /usr/bin/cnvkit, right?


>
>> Should we have someone among us for who packaging go-based software
>> feels like easy then the newly surfaced https://github.com/brentp/gsort
>> would be nice.
> Nilesh did so several times.
Wow! @Nilesh, please have a look.
>> And - qualimap. The jar files are still missing that we need to build that.
> Simply ping the authors about this.  As far as I remember the last
> status was that sources are lost (?) and a backup needs to be found.

And if we just go for a non-free binary package for those jars? Just to
get somewhere?

bcbio is in contrib anyway because of vienna-rna. And I do not think
these .jar files
will find much future adoption without a source backing, so this problem
will eradicate
itself.

There is also
bcbio.pipeline.config_utils.CmdNotFound: '_get_program_cmd' 'snpEff'
{'dir': '/usr/local/share/java/snpeff', 'jvm_opts': ['-Xms750m',
'-Xmx3g']} None

>> Also need to look at how picard is invoked since somehow this does not
>> seem compatible - or it is just because of version differences.
> See above for picard-tools.  You might like to try this first.

I needed to educate myself a bit on picard. There is also a new upstream
version, which kind of fits. As it is now, I got the error message
"'picard' is not a valid command. See PicardCommandLine -h for more
information"
which is coming from picard even though bcbio is executing just that.
This may be another twist that conda may have introduced?
I have just edited the wrapper and installed the link, but now run into
a confusion with Java options that bcbio mixes in
E   '-Xms750m' is not a valid command. See
PicardCommandLine -h for more information.

as in
picard -Xms750m -Xmx2000m -XX:+UseSerialGC
MergeV...cbio/tests/data/variants/S1-variants-snp.vcf.gz
I=bcbio/tests/data/variants/S1-variants-indel.vcf.gz

Update: I fixed that for the new upstream version 2.23.8 but would not
mind an extra pair of eyeballs prior to an upload over what I did to
debian/bin/PicardCommandLine, even though this now passed bcbio's test.


Should anyone feel like it - you see different tests of bcbio listed like
$ cat tests/pytest.ini |head -n 10
# content of pytest.ini
[pytest]
markers =
    cancer: cancer variant calling pipeline
    cancermulti: cancer variant calling pipeline with multiple callers
    cancerpanel: cancer variant calling pipeline on panels
    cancerprecall: cancer pipeline with pre-called variants
    combo: not sure what this test is for
    devel: test for unsupported features still in development
    ensemble: test variant calling with multiple callers

$ wc -l tests/pytest.ini
53 tests/pytest.ini

bcbio downloads extra data for testing, so this all needs some extra
work, still.

Somehow Debian does barely complete any of these tests. There is always
something missing. And if it is tophat that upstream had asked us not to
support any more.

It is a bit discouraging that we are not covering more of bcbio, yet.
But then again, it is a real-world usability for us, too. And we do not
need to cover _everything_ of bcbio to make some noise about it and
allow it to proceed to testing. Just some well-defined workflow would
suffice.

Best,

Steffen

Re: /usr/bin/picard Re: bcbio will need another while - needs gatk

[Andreas Tille]

Hi Steffen,

On Fri, Nov 13, 2020 at 10:25:39PM +0100, Steffen Möller wrote:
> I installed a binary distribution of gatk and then ran a bit into
> trouble over
> 
> $ apt-file search /usr/bin/picard
> picard: /usr/bin/picard
> picard-tools: /usr/bin/picard-tools
> 
> I tend to think that this exposes a weakness of Debian - it misses
> namespaces. Others may say that it is a strength. Don't think so. You
> want to use the same scripts that your scientific peer uses. Everything
> else goes under "the distribution has patches, the work may not be
> completely reproducible" - because a log file has a different hash value
> since executions of /usr/bin/picard were changed to /usr/bin/picard-tools.

We have the workaround

 /usr/lib/debian-med/bin/

see for instance in eigensoft[1] and lots of other packages.  Simply
provide this for picard-tools and make sure gatk users set the PATH
accordingly.
 
> Should we have someone among us for who packaging go-based software
> feels like easy then the newly surfaced https://github.com/brentp/gsort
> would be nice.

Nilesh did so several times.
 
> And - qualimap. The jar files are still missing that we need to build that.

Simply ping the authors about this.  As far as I remember the last
status was that sources are lost (?) and a backup needs to be found.

> Also need to look at how picard is invoked since somehow this does not
> seem compatible - or it is just because of version differences.

See above for picard-tools.  You might like to try this first.

Kind regards

 Andreas.


[1] 
https://salsa.debian.org/med-team/eigensoft/-/blob/master/debian/README.Debian 

-- 
http://fam-tille.de

/usr/bin/picard Re: bcbio will need another while - needs gatk

[Steffen Möller]

Hello,

I installed a binary distribution of gatk and then ran a bit into
trouble over

$ apt-file search /usr/bin/picard
picard: /usr/bin/picard
picard-tools: /usr/bin/picard-tools

I tend to think that this exposes a weakness of Debian - it misses
namespaces. Others may say that it is a strength. Don't think so. You
want to use the same scripts that your scientific peer uses. Everything
else goes under "the distribution has patches, the work may not be
completely reproducible" - because a log file has a different hash value
since executions of /usr/bin/picard were changed to /usr/bin/picard-tools.

At the moment the tests are a bit weird in that it fails over not
finding its configuration files

ValueError: Could not find input system configuration file
/tmp/bcbio_cl6jd4o1/test_automated_output/bcbio_system.yaml, including
inside standard directory /galaxy
steffen@wurzel:~/med/bcbio$ find . -name bcbio_system.yaml
./debian/bcbio/usr/share/bcbio/config/bcbio_system.yaml
./config/bcbio_system.yaml

but we are at

4 failed, 8 passed, 71 deselected, 47 warnings in 272.41 seconds

Should we have someone among us for who packaging go-based software
feels like easy then the newly surfaced https://github.com/brentp/gsort
would be nice.

And - qualimap. The jar files are still missing that we need to build that.

Also need to look at how picard is invoked since somehow this does not
seem compatible - or it is just because of version differences.

Steffen


On 12.11.20 22:02, Steffen Möller wrote:
> [2020-11-12T20:56Z] /usr/bin/bash: line 1: gatk: command not found
>
> There are also other issues, like use using a newer version of STAR
> (rna-star) and the file testing against is not prepared for that. Kind
> of neat: The hts-nim-tools should have an executable with hyphens, not
> underscores. Conda did this wrong but now bcbio adopted this. Upstream
> offered to change to underscore but I just added a symlink and would
> like to learn from this trivial case about the difficulties this makes
> and if this is also something that our link to bio.tools can help with.
>
> Let's see.
>
> Best,
>
> Steffen
>

bcbio will need another while - needs gatk

[Steffen Möller]

[2020-11-12T20:56Z] /usr/bin/bash: line 1: gatk: command not found

There are also other issues, like use using a newer version of STAR
(rna-star) and the file testing against is not prepared for that. Kind
of neat: The hts-nim-tools should have an executable with hyphens, not
underscores. Conda did this wrong but now bcbio adopted this. Upstream
offered to change to underscore but I just added a symlink and would
like to learn from this trivial case about the difficulties this makes
and if this is also something that our link to bio.tools can help with.

Let's see.

Best,

Steffen

gatk: Any hint for Java (Maven) packaging

[Andreas Tille]

Hi,

I just commited

   Vcs-Git: git://anonscm.debian.org/debian-med/gatk.git

I had no real luck with mh_make which only created debian/maven.* files
but no d/contol and d/rules file due to some error in the last step.

I also noticed the following output from mh_make process:

In pom.xml: This dependency cannot be found in the Debian Maven repository. 
Ignore this dependency?  samtools:htsjdk:jar:null
In pom.xml: This dependency cannot be found in the Debian Maven repository. 
Ignore this dependency?  picard:picard:jar:null
In pom.xml: This dependency cannot be found in the Debian Maven repository. 
Ignore this dependency?  colt:colt:jar:null
In pom.xml: This dependency cannot be found in the Debian Maven repository. 
Ignore this dependency?  it.unimi.dsi:fastutil:jar:null
In pom.xml: This dependency cannot be found in the Debian Maven repository. 
Ignore this dependency?  com.google.code.cofoja:cofoja:jar:null
In pom.xml: This dependency cannot be found in the Debian Maven repository. 
Ignore this dependency?  net.sf.jgrapht:jgrapht:jar:null

So there are definitely some missing Build-Dependencies in the current
state.  However, my problem is even before these Build-Depends are
needed since the Java build is not started at all.  Any idea how to
change d/rules to get some build process triggered?

Thanks in advance

   Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-med-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150327122754.gd10...@an3as.eu