Re: canu
On 02/11/2023 10:54, George Marselis wrote: > I'm just undecided because slurm itself needs a bit more than just install the client package and you have immediate advantage without more administration. if I may: maybe gnu parallel would be a good suggestion, as well? Hi, George. GNU Parallel is a way of running 'embarrasingly' parallel jobs (i.e. running jobs that don't communicate with each other) concurrently. It's an improvement on "xargs" that also allows you to run jobs across a cluster of computers. This is useful when jobs consume only a small proportion of the available resources, but there are many jobs to run. However, "canu" jobs typically require very large resources so "Slurm" is used to queue them up for execution. The problem I encountered was that the saved Slurm queue state was stale after an upgrade, resulting in a failure of "canu" to submit a job to the "Slurm" job-queue. I didn't realise until later that "canu" will automatically use "Slurm" (or other resource managers like GE) if it detects that it is installed and, in principle, available. I agree that GNU Parallel is good, but I wouldn't use it for "canu". Andreas is right that "Slurm" requires non-trivial configuration before it can be used and, for that reason, I don't recommend adding it as a "suggests" to the Debian-Med "canu" package. Bye, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk
Re: canu
> I'm just undecided because slurm itself needs a bit more than just install the client package and you have immediate advantage without more administration. if I may: maybe gnu parallel would be a good suggestion, as well? On Thu, Nov 2, 2023 at 7:21 AM Andreas Tille wrote: > Hi Tony, > > Am Wed, Nov 01, 2023 at 05:11:41PM + schrieb Tony Travis: > > > > Sorry, the attached screenshot with your picture on it is what I read, > but I > > didn't notice that it was the upstream README.md. > > O, dear, if I would have written all the stuff where Github > instances put my picture on it, I would probably need a couple of life > times. ;-P > > > > BTW, should we take the message that it makes sense to suggest > > > slurm-client for canu from your observation? > > > > No, because "canu" works without "Slurm": > > Yes, it works without slurm. If it would really need slurm we would use > Recommends or even Depends. Suggests means that it might work better > under some circumstances. I'm just undecided because slurm itself needs > a bit more than just install the client package and you have immediate > advantage without more administration. > > > Our problem was that the saved > > state of the Slurm controller was incompatible with the upgraded version > of > > Slurm. I'm not sure how that happened, but this is not my server and I > was > > simply helping to diagnose why "canu" crashed. > > > > Unfortunately, I didn't save the error messages, but I saw that "canu" > was > > reporting problems submitting a job to "Slurm". I installed "canu" in a > > Bioconda env so that my colleague could continue his work, then I removed > > the stale job-state information and restarted "Slurm". > > > > At some point, I'll reinstall "canu" from the Med-Bio package, but at > > present I don't want to disrupt the work on someone else's server. I'll > use > > the Ubuntu Bug-Tracker to report any issues after reinstalling. > > Thanks a lot for the precise report > Andreas. > > -- > http://fam-tille.de > >
Re: canu
Hi Tony, Am Wed, Nov 01, 2023 at 05:11:41PM + schrieb Tony Travis: > > Sorry, the attached screenshot with your picture on it is what I read, but I > didn't notice that it was the upstream README.md. O, dear, if I would have written all the stuff where Github instances put my picture on it, I would probably need a couple of life times. ;-P > > BTW, should we take the message that it makes sense to suggest > > slurm-client for canu from your observation? > > No, because "canu" works without "Slurm": Yes, it works without slurm. If it would really need slurm we would use Recommends or even Depends. Suggests means that it might work better under some circumstances. I'm just undecided because slurm itself needs a bit more than just install the client package and you have immediate advantage without more administration. > Our problem was that the saved > state of the Slurm controller was incompatible with the upgraded version of > Slurm. I'm not sure how that happened, but this is not my server and I was > simply helping to diagnose why "canu" crashed. > > Unfortunately, I didn't save the error messages, but I saw that "canu" was > reporting problems submitting a job to "Slurm". I installed "canu" in a > Bioconda env so that my colleague could continue his work, then I removed > the stale job-state information and restarted "Slurm". > > At some point, I'll reinstall "canu" from the Med-Bio package, but at > present I don't want to disrupt the work on someone else's server. I'll use > the Ubuntu Bug-Tracker to report any issues after reinstalling. Thanks a lot for the precise report Andreas. -- http://fam-tille.de
Re: canu
Am Wed, Nov 01, 2023 at 10:02:24AM + schrieb Tony Travis: > I didn't say that you gave advice "to use bioconda": I said that I had > followed your advice 'about installing "canu" under "bioconda3"'. > > I needed to fix the problem we encountered with "canu" installed from the > Debian-Med package quickly and adopted a pragmatic solution. This is what I > read in your notes about packaging "canu" for Debian-Med: > > > Installing with a 'package manager' is not encouraged, but if you have no > > other choice: > > > > Conda: conda install -c conda-forge -c bioconda -c defaults canu > > > > Homebrew: brew install brewsci/bio/canu I keep on insisting that these are *not* my notes. Its Upstream README.md what you are quoting. I'm pretty sure I will not write such advise since I'm literally uneducated about conda and have no idea about such command lines. > At the time, I believed that had no other choice... > > > > However, I would be interested to know if you have any plans to package it > > > again for Debian-Med? > > > > Canu is and remains packaged for Debian Med. Since you are using Ubuntu > > which derives from Unstable you will not even miss it even if the package > > might be removed from testing (which the bug you read is about). > > > > In short: Please be more verbose about your problem. > > Upon further investigation, it seems that the problems were caused by a > broken "Slurm" instance on the system in question due to an upgrade (the > saved state of the job queue was inconsistent with the new version of > "Slurm"). I didn't know that "canu" automatically detects the presence of a > job scheduler and will try to use it by default. In any case I'd recommend to add command line and output to give some context for your readers. > I keep forgetting that this is a Debian forum and you prefer me to send bug > reports to the Ubuntu Bug-Tracker. I will refrain from posting about > Debian-Med issues that I encounter under Ubuntu here in future. Please understand me correctly: I'd love to help you - but you need to provide context. Writing something in the sense of "canu is broken under Debian thus I had to use conda" is neither sufficient information to help you nor helpful for our project. Our bug tracker gathers some context (installed packages) and asks you for details which is helpful. If you can't use `reportbug` for whatever reason using this mailing list is a fallback we like to provide since Debian Med people like to care about their users. BTW, should we take the message that it makes sense to suggest slurm-client for canu from your observation? Kind regards Andreas. -- http://fam-tille.de
Re: canu
On 01/11/2023 06:28, Andreas Tille wrote: [...] @Andreas, I've now removed the "canu" package and followed your advice about installing "canu" under "bioconda3": I *never ever* gave any advise to use bioconda. https://salsa.debian.org/med-team/canu Hi, Andreas. I didn't say that you gave advice "to use bioconda": I said that I had followed your advice 'about installing "canu" under "bioconda3"'. I needed to fix the problem we encountered with "canu" installed from the Debian-Med package quickly and adopted a pragmatic solution. This is what I read in your notes about packaging "canu" for Debian-Med: Installing with a 'package manager' is not encouraged, but if you have no other choice: Conda: conda install -c conda-forge -c bioconda -c defaults canu Homebrew: brew install brewsci/bio/canu At the time, I believed that had no other choice... However, I would be interested to know if you have any plans to package it again for Debian-Med? Canu is and remains packaged for Debian Med. Since you are using Ubuntu which derives from Unstable you will not even miss it even if the package might be removed from testing (which the bug you read is about). In short: Please be more verbose about your problem. Upon further investigation, it seems that the problems were caused by a broken "Slurm" instance on the system in question due to an upgrade (the saved state of the job queue was inconsistent with the new version of "Slurm"). I didn't know that "canu" automatically detects the presence of a job scheduler and will try to use it by default. I keep forgetting that this is a Debian forum and you prefer me to send bug reports to the Ubuntu Bug-Tracker. I will refrain from posting about Debian-Med issues that I encounter under Ubuntu here in future. Bye, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk
Re: canu
Hi Tony, Am Tue, Oct 31, 2023 at 11:13:13AM + schrieb Tony Travis: > What's the status of "canu"? Since canu has a test suite issue on arm64 a bug was filed[1]. Since we have no idea about this issue the forwarded the problem upstream[2]. > The "canu" package in Debian-Med appears to be broken under Ubuntu 22.04 and Sorry "broken under Ubuntu 22.04" is not a helpful information. While on several Debian forums you get the answer "this is Debian if you have a problem with Ubuntu ... blabla" you know that you are welcome here with questions. However, we do not have a magic ball to guess what exactly is broken. Please provide as minimum some command line and the output to enable us reproducing what exactly is broken. Under Debian on amd64 architecture its definitely not "obviously" broken since it is passing its CI test on this architecture which you are most probably using. > I see that it has been marked for removal in Nov 2023: This is due to a CI test issue on arm64 (see also discussion with upstream[2] which explains, that this issue seems to occure only under specific CI circumstances). This again raises the question: What is broken for you and do you possibly want to report this in an upstream issue? > > https://tracker.debian.org/pkg/canu > > @Andreas, I've now removed the "canu" package and followed your advice about > installing "canu" under "bioconda3": I *never ever* gave any advise to use bioconda. > > https://salsa.debian.org/med-team/canu > > However, I would be interested to know if you have any plans to package it > again for Debian-Med? Canu is and remains packaged for Debian Med. Since you are using Ubuntu which derives from Unstable you will not even miss it even if the package might be removed from testing (which the bug you read is about). In short: Please be more verbose about your problem. Kind regards Andreas. [1] https://bugs.debian.org/1052347 [2] https://github.com/marbl/canu/issues/2271 -- http://fam-tille.de
canu
Hi, What's the status of "canu"? The "canu" package in Debian-Med appears to be broken under Ubuntu 22.04 and I see that it has been marked for removal in Nov 2023: https://tracker.debian.org/pkg/canu @Andreas, I've now removed the "canu" package and followed your advice about installing "canu" under "bioconda3": https://salsa.debian.org/med-team/canu However, I would be interested to know if you have any plans to package it again for Debian-Med? Thanks, Tony. -- Minke Informatics Limited, Registered in Scotland - Company No. SC419028 Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK) tel. +44(0)19755 63548http://minke-informatics.co.uk mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk
