External Email - Use Caution Hi Alex,
I am also running FreeSurfer 6.0.0 on SLURM and recently did some tests on how to parallelize it, how much ressources to ask for… This is what I learned: 1) FreeSurfer 6.0.0 is able to use OpenMP parallelization (I guess for the commands mri_em_register, mri_ca_register and mri_ca label used in the steps Skull Stripping, EM registration, CA reg and Subcortical registration). 2) From Version 6.0.0 FreeSurfer is also able to process the two hemispheres in parallel instead of sequentially (steps 16 – 30). Taking advantage of these changes reduces processing time dramatically. If I use 4 CPU per task on SLURM instead of 1 the processing time drops form around 8 hours to around 4 or 4 ½ hours. Increasing the CPU per task did not improve processing time much though, so to save a bit CPU time now use 4 CPU regularly. I don’t know if you have to submit your jobs exactly the same way but a typical job file for me looks like this: ------------------------------------------------------------------------------------ #!/bin/bash #SBATCH --time=10:00:00 #SBATCH --output=/your_working_directory/job-%j.out #SBATCH --error=/your_working_directory/job-%j.err #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --tasks-per-node=1 #SBATCH --cpus-per-task=4 #SBATCH --partition=haswell #SBATCH --mem-per-cpu=2500 #SBATCH -A your_project_name module load modenv/eb module load FreeSurfer source $FREESURFER_HOME/SetUpFreeSurfer.sh export SUBJECTS_DIR=/your_subjects_directory recon-all -s your_subject_ID -all -parallel -openmp 4 -------------------------------------------------------------------------------------- Of course you can just add the –hippocampal-subfields-T1 tag flag your command. I found that asking for 2500 MB works just fine for 4CPU per task (as the 2500 is multiplied times 4 in this case). The maximum RAM was about 7000 KB for my jobs. Just make sure you add the” -parallel -openmp 4” flag to your command line AND “#SBATCH --cpus-per-task=4” to your job file, otherwise the parallelization will not work. Hope this helps! Get back to me if you have any more questions! Best, Lea ------------------------------ Lea Backhausen Research Assistant Department of Child and Adolescent Psychiatry, Faculty of Medicine of the TU Dresden, Germany http://www.uniklinikum-dresden.de ------------------------------ Message: 5 Date: Thu, 19 Apr 2018 17:05:46 +0200 From: Alexandre Obert <obert.alexan...@gmail.com> Subject: [Freesurfer] SLURM - speed processing To: freesurfer@nmr.mgh.harvard.edu Message-ID: <cagteuh_xypwu4h+4ban62bfh8idf96hequul+ommawt_gek...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" External Email - Use Caution Hi all, I'm definitively a noob with both freesurfer and SLURM processes and I wonder if anyone could give me some advice. I tried processing a subject (including hippocampal segmentation) using the following command (freesurfer 6.0.0): recon-all -all -s s03 -hippocampal-subfields-T1 It took nearly 12h to process. I then submit this job to a supercomputer (using slurm) and I asked for nearly 30Go RAM It took nearly 7h to process. That seems good but I wonder if it's possible to speed it up a little more? Regards, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.nmr.mgh.harvard.edu/pipermail/freesurfer/attachments/20180419/cbcc829a/attachment-0001.html ------------------------------ Message: 6 Date: Thu, 19 Apr 2018 11:57:45 -0400 From: "Douglas N. Greve" <dgr...@mgh.harvard.edu> Subject: Re: [Freesurfer] LGI for the lobar annotation To: "Fernanda Hansen P. de Moraes" <fernandahmor...@gmail.com> Cc: "freesurfer@nmr.mgh.harvard.edu" <freesurfer@nmr.mgh.harvard.edu> Message-ID: <9822cb24-192a-2713-0418-b215da87d...@mgh.harvard.edu> Content-Type: text/plain; charset="utf-8"; format=flowed yes On 04/19/2018 07:17 AM, Fernanda Hansen P. de Moraes wrote: > Douglas, thank you very much again. > So I can assume it is the lGI for the area (in this case the lobe)? > > Thank you again! > > On Wed, Apr 18, 2018 at 11:48 PM Douglas Greve <dgr...@mgh.harvard.edu > <mailto:dgr...@mgh.harvard.edu>> wrote: > > The "intensity" is the mean value of the input (lh.pial_lgi in > this case) averaged over vertices. If? you wan the total number, > multiply it by the number of vertices or re-run with --accumulate. > The area is the surface area of the entire lobe. > > > On 4/18/18 3:23 PM, Fernanda Hansen P. de Moraes wrote: >> Sorry, my mistake. >> >> mri_segstats --annot DA011 lh lobesStrict --i >> DA011/surf/lh.pial_lgi --sum lh.lobesStrict.annot.pial_lgi.stats >> >> On Wed, Apr 18, 2018 at 4:13 PM Douglas N. Greve >> <dgr...@mgh.harvard.edu <mailto:dgr...@mgh.harvard.edu>> wrote: >> >> What is your command line? Also, please remember to post to >> the list >> >> >> On 04/18/2018 02:10 PM, Fernanda Hansen P. de Moraes wrote: >> > Dear Douglas, >> > >> > thank you. >> > >> > ?The code worked, but I didn't understand the result very >> well. What >> > means intensity? The area, is the ara from the piall_lgi >> surface >> > (outer surface)? >> > >> > Thank you again >> > >> > On Tue, Apr 17, 2018 at 1:07 PM Douglas Greve >> <dgr...@mgh.harvard.edu <mailto:dgr...@mgh.harvard.edu> >> > <mailto:dgr...@mgh.harvard.edu >> <mailto:dgr...@mgh.harvard.edu>>> wrote: >> > >> >? ? ?If you run it with --help you will see an example >> > >> > >> >? ? ?On 4/16/18 7:16 AM, Fernanda Hansen P. de Moraes wrote: >> >>? ? ?Thank you, Douglas, >> >> >> >>? ? ?do you have any suggestions?on how to make you work? >> >> >> >>? ? ?Fernanda >> >> >> >> >> >> >> >>? ? ?On Fri, Apr 13, 2018 at 5:56 PM Douglas Greve >> >>? ? ?<dgr...@mgh.harvard.edu >> <mailto:dgr...@mgh.harvard.edu> >> <mailto:dgr...@mgh.harvard.edu >> <mailto:dgr...@mgh.harvard.edu>>> wrote: >> >> >> >>? ? ? ? ?that should work, except the command line is not >> quite right >> >> >> >> >> >>? ? ? ? ?On 4/13/18 4:22 PM, Fernanda Hansen P. de Moraes >> wrote: >> >>>? ? ? ? ?Good night, >> >>> >> >>>? ? ? ? ?does anyone know if it is possible to calculate >> the lGI for >> >>>? ? ? ? ?the ?h.lobesStrict.annot? And if it yes, how can >> I do it? >> >>> >> >>>? ? ? ? ?I've already tried to use: >> >>> >> >>>? ? ? ? ?/mri_segstats --annot >> subjxx/label/lh.lobesStrict.annot --i >> >>> ?$SUBJECTS_DIR/subjxx/surf/lh.pial_lgi --sum >> >>> ?lh.lobesStrict.annot.pial_lgi.stats/ >> >>> >> >>>? ? ? ? ?Thank you! >> >>>? ? ? ? ?-- >> >>>? ? ? ? ?Fernanda Hansen P. de Moraes >> >>>? ? ? ? ?F?sica M?dica | Instituto D'Or de Pesquisa e >> Ensino?| Rede >> >>>? ? ? ? ?D'Or S?o Luiz >> >>> >> >>> >> >>> ?_______________________________________________ >> >>>? ? ? ? ?Freesurfer mailing list >> >>> Freesurfer@nmr.mgh.harvard.edu >> <mailto:Freesurfer@nmr.mgh.harvard.edu> >> >>>? ? ? ? ?<mailto:Freesurfer@nmr.mgh.harvard.edu >> <mailto:Freesurfer@nmr.mgh.harvard.edu>> >> >>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer >> >> >> >> ?_______________________________________________ >> >>? ? ? ? ?Freesurfer mailing list >> >> Freesurfer@nmr.mgh.harvard.edu >> <mailto:Freesurfer@nmr.mgh.harvard.edu> >> >>? ? ? ? ?<mailto:Freesurfer@nmr.mgh.harvard.edu >> <mailto:Freesurfer@nmr.mgh.harvard.edu>> >> >> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer >> >> >> >> >> >>? ? ? ? ?The information in this e-mail is intended only >> for the >> >>? ? ? ? ?person to whom it is >> >>? ? ? ? ?addressed. If you believe this e-mail was sent to >> you in >> >>? ? ? ? ?error and the e-mail >> >>? ? ? ? ?contains patient information, please contact the >> Partners >> >>? ? ? ? ?Compliance HelpLine at >> >> http://www.partners.org/complianceline . If the e-mail was >> >>? ? ? ? ?sent to you in error >> >>? ? ? ? ?but does not contain patient information, please >> contact the >> >>? ? ? ? ?sender and properly >> >>? ? ? ? ?dispose of the e-mail. >> >> >> >>? ? ?-- >> >>? ? ?Fernanda Hansen P. de Moraes >> >>? ? ?F?sica M?dica | Instituto D'Or de Pesquisa e Ensino?| >> Rede D'Or >> >>? ? ?S?o Luiz >> >> >> >> >> >> ?_______________________________________________ >> >>? ? ?Freesurfer mailing list >> >> Freesurfer@nmr.mgh.harvard.edu >> <mailto:Freesurfer@nmr.mgh.harvard.edu> >> >>? ? ?<mailto:Freesurfer@nmr.mgh.harvard.edu >> <mailto:Freesurfer@nmr.mgh.harvard.edu>> >> >> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer >> > >> > -- >> > Fernanda Hansen P. de Moraes >> > F?sica M?dica | Instituto D'Or de Pesquisa e Ensino?| Rede >> D'Or S?o Luiz >> >> -- >> Fernanda Hansen P. de Moraes >> F?sica M?dica | Instituto D'Or de Pesquisa e Ensino?| Rede D'Or >> S?o Luiz > > -- > Fernanda Hansen P. de Moraes > F?sica M?dica | Instituto D'Or de Pesquisa e Ensino?| Rede D'Or S?o > Luiz ------------------------------ _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer End of Freesurfer Digest, Vol 170, Issue 31 ******************************************* _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
