Dear Marius, the slurm typo is now fixed and the job crash without any error reported. When I look at the pulsar log(attached), I see this:
Jun 30 19:20:24 nic5-login1 pulsar[210900]: 2021-06-30 19:20:24,122 WARNI
[pulsar.managers.util.cli.job.slurm][[manager=_default_]-[action=preprocess]-[job=11]]
Unrecognized long argument passed to Slurm CLI plugin: directory_mode
Do you have an idea ?
Thanks
Luc
------------
Luc Cornet, PhD
Bio-informatician
Mycology and Aerobiology
Sciensano
----- Mail original -----
De: "Marius van den Beek" <m.vandenb...@gmail.com>
À: "Luc Cornet" <luc.cor...@uliege.be>
Cc: "HelpGalaxy" <galaxy-dev@lists.galaxyproject.org>, "Baurain Denis"
<denis.baur...@uliege.be>, "Pierre Becker" <pierre.bec...@sciensano.be>,
"Colignon David" <david.colig...@uliege.be>
Envoyé: Mercredi 30 Juin 2021 18:38:24
Objet: [galaxy-dev] Re: Galaxy install problems
Hi Luc,
you need to capitalize the first letter of slurm in your config:
pulsar_yaml_config:
conda_auto_init: True
conda_auto_install: True
staging_directory: "{ { pulsar_staging_dir }}"
persistence_directory: "{ { pulsar_persistence_dir }}"
tool_dependency_dir: "{ { pulsar_dependencies_dir }}"
# The following are the settings for the pulsar server to contact the message
queue with related timeouts etc.
message_queue_url: "pyamqp://galaxy_au:{ { rabbitmq_password_galaxy_au }}@{ {
galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1"
managers:
_default_:
type: queued_cli
job_plugin: Slurm
native_specification: "-p batch --tasks=1 --cpus-per-task=2 --mem-per-cpu=1000
-t 10:00"
min_polling_interval: 0.5
amqp_publish_retry: True
amqp_publish_retry_max_retries: 5
amqp_publish_retry_interval_start: 10
amqp_publish_retry_interval_step: 10
amqp_publish_retry_interval_max: 60
Best,
Marius
On Wed, 30 Jun 2021 at 18:35, Luc Cornet < [ mailto:luc.cor...@uliege.be |
luc.cor...@uliege.be ] > wrote:
Re,
We shot down pulsar, run the playbook again, restart pulsar:
We I try to launch an analysis, I have this error for the job (still CLI):
Failed to find job_plugin of type slurm, available types include ['LSF',
'Slurm', 'SlurmTorque', 'Torque']
Luc,
------------
Luc Cornet, PhD
Bio-informatician
Mycology and Aerobiology
Sciensano
----- Mail original -----
De: "Marius van den Beek" < [ mailto:m.vandenb...@gmail.com |
m.vandenb...@gmail.com ] >
À: "Luc Cornet" < [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] >
Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org |
galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [
mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] >, "Pierre Becker" <
[ mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] >, "Colignon
David" < [ mailto:david.colig...@uliege.be | david.colig...@uliege.be ] >
Envoyé: Mercredi 30 Juin 2021 17:24:13
Objet: [galaxy-dev] Re: Galaxy install problems
That should be `systemctl restart pulsar.service`
Marius
On Wed, 30 Jun 2021 at 17:05, Luc Cornet < [ mailto: [
mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] | [
mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] ] > wrote:
Re,
This is the log of pulsar.
As pointed by Keith Superman, the pulsar don't restart since the step is
skipped.
How can I force pulsar to restart ?
Thanks,
Luc
------------
Luc Cornet, PhD
Bio-informatician
Mycology and Aerobiology
Sciensano
----- Mail original -----
De: "Marius van den Beek" < [ mailto: [ mailto:m.vandenb...@gmail.com |
m.vandenb...@gmail.com ] | [ mailto:m.vandenb...@gmail.com |
m.vandenb...@gmail.com ] ] >
À: "Luc Cornet" < [ mailto: [ mailto:luc.cor...@uliege.be |
luc.cor...@uliege.be ] | [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ]
] >
Cc: "HelpGalaxy" < [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org |
galaxy-dev@lists.galaxyproject.org ] | [
mailto:galaxy-dev@lists.galaxyproject.org | galaxy-dev@lists.galaxyproject.org
] ] >, "Baurain Denis" < [ mailto: [ mailto:denis.baur...@uliege.be |
denis.baur...@uliege.be ] | [ mailto:denis.baur...@uliege.be |
denis.baur...@uliege.be ] ] >, "Pierre Becker" < [ mailto: [
mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] | [
mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] ] >, "Colignon
David" < [ mailto: [ mailto:david.colig...@uliege.be | david.colig...@uliege.be
] | [ mailto:david.colig...@uliege.be | david.colig...@uliege.be ] ] >
Envoyé: Mercredi 30 Juin 2021 16:44:50
Objet: Re: [galaxy-dev] Re: Galaxy install problems
If you've set up pulsar to start as a systemd service you should get the
logs with journalctl.
I assume that's `journalctl -u pulsar.service`.
For a quickstart in journalctl check out
[ [ https://www.linode.com/docs/guides/how-to-use-journalctl/ |
https://www.linode.com/docs/guides/how-to-use-journalctl/ ] | [
https://www.linode.com/docs/guides/how-to-use-journalctl/ |
https://www.linode.com/docs/guides/how-to-use-journalctl/ ] ]
On Wed, 30 Jun 2021 at 16:38, Luc Cornet < [ mailto: [
mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] | [
mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] ] > wrote:
> OK,
>
> Sure but can you please tell me how to get these pulsar application logs ?
>
> Thanks
> Luc
>
> ------------
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> ----- Mail original -----
> De: "Marius van den Beek" < [ mailto: [ mailto:m.vandenb...@gmail.com |
> m.vandenb...@gmail.com ] | [ mailto:m.vandenb...@gmail.com |
> m.vandenb...@gmail.com ] ] >
> À: "Luc Cornet" < [ mailto: [ mailto:luc.cor...@uliege.be |
> luc.cor...@uliege.be ] | [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be
> ] ] >
> Cc: "HelpGalaxy" < [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] | [
> mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] ] >, "Baurain Denis" <
> [ mailto: [ mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] | [
> mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] ] >, "Pierre
> Becker" < [ mailto: [ mailto:pierre.bec...@sciensano.be |
> pierre.bec...@sciensano.be ] | [ mailto:pierre.bec...@sciensano.be |
> pierre.bec...@sciensano.be ] ] >,
> "Colignon David" < [ mailto: [ mailto:david.colig...@uliege.be |
> david.colig...@uliege.be ] | [ mailto:david.colig...@uliege.be |
> david.colig...@uliege.be ] ] >
> Envoyé: Mercredi 30 Juin 2021 16:32:16
> Objet: [galaxy-dev] Re: Galaxy install problems
>
> Thanks, but these are not the pulsar application logs. Can you provide
> these please ?
>
> On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto: [ mailto: [
> mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] | [
> mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] ]
> | [ mailto: [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] | [
> mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] ] ] > wrote:
>
>
> Dear Marius,
>
> Many thank for your feedback.
>
> I join to this email: the playbook, the pulsarservers.yml file and the log
> of pulsar playbook.
>
> CLI plugin is for us the best solution since we have nothing to maintain.
> DRMAA is not actively developed for slurm, correct ?
>
>
> In the playbook, we use systemd which I think should restart pulsar but It
> might not be the case:
>
> TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start
> service]
> ****************************************************************************
>
> ok: [HPC]
>
> RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler]
> *********************************************************************************
>
>
> skipping: [HPC]
>
> Currently, we never used DRMAA. The job were executed immediately on the
> cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to
> activate CLI:
> managers:
> _default_:
> type: queued_cli
> job_plugin: slurm
> native_specification: "-p batch --tasks=1 --cpus-per-task=2
> --mem-per-cpu=1000 -t 10:00"
> min_polling_interval: 0.5
> amqp_publish_retry: True
> amqp_publish_retry_max_retries: 5
> amqp_publish_retry_interval_start: 10
> amqp_publish_retry_interval_step: 10
> amqp_publish_retry_interval_max: 60
>
>
> Thanks for your help,
> Luc
>
>
> ------------
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> ----- Mail original -----
> De: "Marius van den Beek" < [ mailto: [ mailto: [
> mailto:m.vandenb...@gmail.com | m.vandenb...@gmail.com ] | [
> mailto:m.vandenb...@gmail.com | m.vandenb...@gmail.com ] ] |
> [ mailto: [ mailto:m.vandenb...@gmail.com | m.vandenb...@gmail.com ] | [
> mailto:m.vandenb...@gmail.com | m.vandenb...@gmail.com ] ] ] >
> À: "Luc Cornet" < [ mailto: [ mailto: [ mailto:luc.cor...@uliege.be |
> luc.cor...@uliege.be ] | [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be
> ] ] | [ mailto: [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] | [
> mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ] ] ]
> >
> Cc: "HelpGalaxy" < [ mailto: [ mailto: [
> mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] | [
> mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] ] |
> [ mailto: [ mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] | [
> mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] ] ] >, "Baurain Denis" < [ mailto:
> [ mailto: [ mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] | [
> mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] ] | [ mailto: [
> mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] | [
> mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] ] ] >, "Pierre
> Becker" <
> [ mailto: [ mailto: [ mailto:pierre.bec...@sciensano.be |
> pierre.bec...@sciensano.be ] | [ mailto:pierre.bec...@sciensano.be |
> pierre.bec...@sciensano.be ] ] | [ mailto: [
> mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] | [
> mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] ] ] >,
> "Colignon David" < [ mailto: [ mailto: [ mailto:david.colig...@uliege.be |
> david.colig...@uliege.be ] | [ mailto:david.colig...@uliege.be |
> david.colig...@uliege.be ] ] |
> [ mailto: [ mailto:david.colig...@uliege.be | david.colig...@uliege.be ] | [
> mailto:david.colig...@uliege.be | david.colig...@uliege.be ] ] ] >
> Envoyé: Mercredi 30 Juin 2021 16:02:04
> Objet: [galaxy-dev] Re: Galaxy install problems
>
> Hi Luc,
>
> I'm sorry to hear that you're struggling to set up Galaxy to your liking.
> Let me start by pointing out that [ [ [ [ http://usegalaxy.org/ |
> http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ]
> |
> [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [ http://usegalaxy.org/
> | http://usegalaxy.org/ ] ] ] | [ [ [ http://usegalaxy.org/ |
> http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | http://usegalaxy.org/ ] ]
> | [ [ http://usegalaxy.org/ | http://usegalaxy.org/ ] | [
> http://usegalaxy.org/ | usegalaxy.org ] ] ] ]
> uses slurm with DRMAA, this is certainly going to be more performant and
> reliable than the CLI plugin.
> There is little maintenance necessary, so maybe that is why activity on
> slurm-drmaa is low (See also [ [ [ [ https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] | [
> https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] ] |
> [ [ https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] | [
> https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] ] ] | [
> [ [ https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] | [
> https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] ] |
> [ [ https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] | [
> https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] ] ] ] ).
> I would be curious to know how you came to the conclusion that there is
> some incompatibility between DRMAA and slurm
> Note that one of the setups we teach during the training submits via DRMAA
> to slurm.
>
> Then I'd like to point out that there are a huge variety of different ways
> in which you can configure Galaxy and the job submission.
> We teach the most common ones during the training week, with the aim that
> you understand how these things work together,
> as well as giving you a handle on how you can manage these different
> settings and services using a configuration management system.
> We cannot tailor a solution to your infrastructure during this week.
>
> About your problem specifically, I had asked this on gitter before:
>
> > Did you restart pulsar after rolling out the new config ?
>
> to which you've answered that you re-ran the playbook, but that's not a
> sufficient answer.
>
> Every playbook is different, and we cannot know if this includes a
> restarter service for pulsar.
> Also please don't assume that everyone that could potentially help you
> knows ansible and the playbooks that are being taught intimately,
> and in what ways you have customized your playbook.
> It is much more helpful to write up the relevant settings you've changed
> and the logs that go with it.
>
> You've also been asked to provide logs of the restart, which as far as I
> can tell you haven't provided.
> You had mentioned on gitter that pulsar continues to use DRMAA to submit
> jobs, so you'll
> want to double check whether you've really restarted pulsar after the
> config changes,
> and look at the startup logs for pulsar, and find out how it is possible
> for pulsar to submit jobs
> via drmaa if it is not set up to do so.
>
> Best,
> Marius
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
> [ [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] |
> [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ] | [
> [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [
> http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ] ]
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
> [ [ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] | [
> http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ] ]
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
%(web_page_url)s
To search Galaxy mailing lists use the unified search at:
[ http://galaxyproject.org/search/ | http://galaxyproject.org/search/ ]
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
%(web_page_url)s
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/
pulsar.log2
Description: Binary data
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
