Re: [Freesurfer] Building Freesurfer dev with GPU support/CUDA enabled
Hi Z K, I went ahead and tried your suggestion of running *make install *and these are errors I have, mv ../../distribution/average/?h.EC_average \ /home/tyson/tyson/fs_install/subjects chmod -R 775 /home/tyson/tyson/fs_install/subjects/?h.EC_average rm -Rf /home/tyson/tyson/fs_install/average/mult-comp-cor rm -Rf ../../distribution/average/mult-comp-cor gunzip -c ../../distribution/average/mult-comp-cor.tar.gz | tar xvf - gzip: ../../distribution/average/mult-comp-cor.tar.gz: No such file or directory tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors make[2]: *** [install] Error 2 make[2]: Leaving directory `/media/data1/tyson/freesurfer/distribution/average' make[1]: *** [install-recursive] Error 1 make[1]: Leaving directory `/media/data1/tyson/freesurfer/distribution' make: *** [install-recursive] Error 1 Thanks, Tyson On Wed, Mar 1, 2017 at 1:28 AM, Francis Tyson Thomas < francisttho...@email.arizona.edu> wrote: > Hi Richard, > > I have attached the log file as you had requested. > > Thanks, > > > On Tue, Feb 28, 2017 at 3:36 AM, R Edgar wrote: > >> On 28 February 2017 at 01:10, Francis Tyson Thomas >> wrote: >> >> > >> >> > See utils/test/mriBuildVoronoiDiagramFloat/test-suite.log >> > Please report to freesurfer@nmr.mgh.harvard.edu >> > >> >> >> That's odd - to my knowledge, that's not a GPU accelerated program. >> >> What does the log say exactly? And which GPU accelerated versions have >> been run up to this point? >> >> Regards, >> >> Richard >> ___ >> Freesurfer mailing list >> Freesurfer@nmr.mgh.harvard.edu >> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer >> >> >> The information in this e-mail is intended only for the person to whom it >> is >> addressed. If you believe this e-mail was sent to you in error and the >> e-mail >> contains patient information, please contact the Partners Compliance >> HelpLine at >> http://www.partners.org/complianceline . If the e-mail was sent to you >> in error >> but does not contain patient information, please contact the sender and >> properly >> dispose of the e-mail. >> >> > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Re: [Freesurfer] Building Freesurfer dev with GPU support/CUDA enabled
Hi Richard, I have attached the log file as you had requested. Thanks, On Tue, Feb 28, 2017 at 3:36 AM, R Edgar wrote: > On 28 February 2017 at 01:10, Francis Tyson Thomas > wrote: > > > > > > See utils/test/mriBuildVoronoiDiagramFloat/test-suite.log > > Please report to freesurfer@nmr.mgh.harvard.edu > > > > > That's odd - to my knowledge, that's not a GPU accelerated program. > > What does the log say exactly? And which GPU accelerated versions have > been run up to this point? > > Regards, > > Richard > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > > test-suite.log Description: Binary data ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
[Freesurfer] Building Freesurfer dev with GPU support/CUDA enabled
Hi, I have been trying to build the freesurfer dev version with GPU support as per the information I had from my previous try a couple of months ago that Richard help me to successfully complete and I have not been successful in that. I followed the instructions from https://surfer.nmr.mgh.harvard.edu/fswiki/freesurfer_linux_developers_page with slight changes, After running, cd ~/freesurfer ./setup_configure I edited the configure file that it generated and then commented out, # with_cuda="" and I changed from CUDA_LIBS="-L$CUDA_DIR/lib $LIB_CUDA -lcudart" to CUDA_LIBS="-L$CUDA_DIR/lib64 $LIB_CUDA -lcudart" then I executed, ./configure --disable-Werror --with-pkgs-dir=/home/username/centos6-x86_64-packages --prefix=/home/username/fs_install --disable-xawplus-apps --disable-tcltk-apps --enable-fermi-gpu --with-cuda="/usr/local/cuda" Subsequently when I ran make -j4 it worked without any errors. But after that when I run make check I get errors and it fails. I had sent a previous email regarding the same in another thread but it looks like it has not shown up. So I'm re-sending this now. Infact I have recloned the freesurfer dev version to try it once again and all I get is new errors. Following are the errors now, Testsuite summary for Freesurfer dev # TOTAL: 1 # PASS: 0 # SKIP: 0 # XFAIL: 0 # FAIL: 1 # XPASS: 0 # ERROR: 0 See utils/test/mriBuildVoronoiDiagramFloat/test-suite.log Please report to freesurfer@nmr.mgh.harvard.edu make[5]: *** [test-suite.log] Error 1 make[5]: Leaving directory `/media/data1/tyson/freesurfer/utils/test/mriBuildVoronoiDiagramFloat' make[4]: *** [check-TESTS] Error 2 make[4]: Leaving directory `/media/data1/tyson/freesurfer/utils/test/mriBuildVoronoiDiagramFloat' make[3]: *** [check-am] Error 2 make[3]: Leaving directory `/media/data1/tyson/freesurfer/utils/test/mriBuildVoronoiDiagramFloat' make[2]: *** [check-recursive] Error 1 make[2]: Leaving directory `/media/data1/tyson/freesurfer/utils/test' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/media/data1/tyson/freesurfer/utils' make: *** [check-recursive] Error 1 If you want the log files, let me know and I'll send it as well. The idea behind trying to build this was see how freesurfer v6 dev version works with GPU. Also would it be possible to use the GPU binaries of mri_em_register and the other register binary and combine it with the current opnmp implemenation in freesurfer v6? My objective all along has been to reduce it from the current best performance of 4hrs and 38mins to something even lower like 2hrs or lesser on a per subject basis since the subjects are pushed to the server dynamically. Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Re: [Freesurfer] Freesurfer v6.0 / Developer version GPU implementation
Thanks Richard! I'll take a look at it and see how it turns up. On Mon, Feb 13, 2017 at 6:28 PM, R Edgar wrote: > On 13 February 2017 at 18:15, Francis Tyson Thomas > wrote: > > > I'm trying to reduce the freesurfer runtime as much as possible and I was > > wondering if the GPU flags still work for recon-all in Freesurfer v6? If > not > > is it possible to use the developer version to have it running? Also, if > I > > follow the second approach would the results be the same as that obtained > > through CPU implementation? > > I'm not sure about the status of the GPU flags for recon-all, but the > results will not be exactly the same - floating point arithmetic tends > not to work that way. > > Richard > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
[Freesurfer] Freesurfer v6.0 / Developer version GPU implementation
Hi, I'm trying to reduce the freesurfer runtime as much as possible and I was wondering if the GPU flags still work for recon-all in Freesurfer v6? If not is it possible to use the developer version to have it running? Also, if I follow the second approach would the results be the same as that obtained through CPU implementation? Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
[Freesurfer] Parallelization not working as implemented
Hi, I have run recon-all with both -parallel enabled (including different openmp threads) and disabled and I'm not able to get processing times same as the information provided in recon-all help. The CPU used is *Dual Intel **Xeon E5-2623 v3* paired with 32GB DDR4 RAM. Is the configuration presented in recon-all help a dual cpu configuration? Following are the runtimes I have obtained, no parallelization : 7 hrs and 58 mins -parallel (coarse parallelization) : 4 hrs and 39 mins -parallel (fine parallelization; -openmp 8): 4 hrs and 34 mins -parallel (fine parallelization; -openmp 14) : 4 hrs and 37 mins Somehow varying the number of threads is having no effect in the execution time of recon-all as can been seen from the processing times. If you can explain what could be possibly going wrong here, it will help me in speeding it up further. Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
[Freesurfer] Parallelization implementation in Freesurfer 6.0
Hello Freesurfer team! Having gone through the instructions for making use of the parallelization in Freesurfer v6, I'm confused as to how exactly the calls for fine grained and coarse grained paralleization differ. As per the release notes, following is stated - *Parallelization: a new flag was introduced which enables two forms of compute parallelization that significantly reduces the runtime. As a point of reference, using a new-ish workstation (2015+), the recon-all -all runtime is just under 3 hours. When the -parallel flag is specified at the end of the recon-all command-line, it will enable 'fine-grained' parallelized code, making use of OpenMP, embedded in many of the binaries, namely affecting mri_em_register and mri_ca_register. By default, it instructs the binaries to use 4 processors (cores), meaning, 4 threads will run in parallel in some operations (manifested in 'top' by mri_ca_register, for example, showing 400% CPU utilization). This can be overridden by including the flag -openmp after -parallel, where is the number of processors you'd like to use (ex. 8 if you have an 8 core machine). Note that this parallelization was introduced in v5.3, but many new routines were OpenMP-parallelized in v6. The other form of parallelization, a 'coarse' form, enabled when the -parallel flag is specified, is such that during the stages where left and right hemispheric data is processed, each hemi binary is run separately (and in parallel, manifesting itself in 'top' as two instances of mris_sphere, for example). Note that a couple of the hemi stages (eg. mris_sphere) make use of a tiny amount of OpenMP code, which means that for brief periods, as many as 8 cores are utilized (2 binaries running code that each make use of 4 threads). In general, though, a 4 core machine can easily handle those periods. Be aware that if you enable this -parallel flag on instances of recon-all running through a job scheduler (like a cluster), it may not make your System Administrator happy if you do not pre-allocate a sufficient number of cores for your job, as you will be taking cycles from other cores that may be running jobs belonging to other cluster users.* Does this mean a command executed as "recon-all -s vol0 -i junk/orig/001/IM-0001-0001.dcm -all -parallel" would be a coarse grained parallelization and the command executed as "recon-all -s vol0 -i junk/orig/001/IM-0001-0001.dcm -all -parallel -openmp 8" be a fine grained parallelization? Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Re: [Freesurfer] Freesurfer V6 beta
Hi Bruce, Is there any new timeline regarding the release of the v6 beta? Because I'm currently waiting on that version for my project's work because of the new hipppocampal module and knowing that will help me a lot. Thanks, Tyson On Tue, May 3, 2016 at 5:51 AM, Bruce Fischl wrote: > Hi Tyson > > we are moving through our testing and hoping to get it out soon. Hard to > say more definitively than that > > sorry > Bruce > > > On Mon, 2 May 2016, Francis Tyson Thomas wrote: > > > Hello Fresurfer admin! > > I was looking for the v6 beta and it looks like it still hasn't been > > released after it was pulled down the last time. I would like to know > what > > is the expected ETA for the beta. Last I remember it was mentioned it was > > on the verge of release about two weeks ago.I'm looking forward to get my > > hands on the beta! > > > > Thanks, > > Tyson > > > > > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
[Freesurfer] Freesurfer V6 beta
Hello Fresurfer admin! I was looking for the v6 beta and it looks like it still hasn't been released after it was pulled down the last time. I would like to know what is the expected ETA for the beta. Last I remember it was mentioned it was on the verge of release about two weeks ago.I'm looking forward to get my hands on the beta! Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Re: [Freesurfer] Freesurfer v6 beta
Hi Eugenio, I understood this part that you mentioned but I didn't understand how is is affected and what would be its consequences? Because I am referring to your preprint paper that makes use of v6, so will the results/validity of that paper also get affected by these differences. Because in my work the accuracy of the segmentation outputs are critical both for analysis and speeding it up. Thanks, Tyson On Wed, Apr 13, 2016 at 3:52 PM, Eugenio Iglesias wrote: > Hi Tyson, > the hippocampal module shouldn't change between your beta and the FS6 > release, but the main recon-all stream will. Since the latter is used to > initialize the former, the results will be a bit different. > Cheers, > /Eugenio > > Juan Eugenio Iglesias > Postdoctoral researcher BCBL > www.jeiglesias.com > www.bcbl.eu > > Legal disclaimer/Aviso legal/Lege-oharra: www.bcbl.eu/legal-disclaimer > > ------ > *From: *"Francis Tyson Thomas" > *To: *"Freesurfer support list" > *Sent: *Wednesday, April 13, 2016 8:19:18 PM > *Subject: *[Freesurfer] Freesurfer v6 beta > > Hi, > > I was wondering when would the new Freesurfer v6 beta be available? Also I > have been using the old beta for my hippocampal segmentation and I wanted > to know if this new beta will affect the results of that? > > Thanks, > Tyson > > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
[Freesurfer] Freesurfer v6 beta
Hi, I was wondering when would the new Freesurfer v6 beta be available? Also I have been using the old beta for my hippocampal segmentation and I wanted to know if this new beta will affect the results of that? Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Re: [Freesurfer] OpenMP and GPU implementation in freesurfer 6 beta
Hi Richard, That was really helpful. All the details were very clear and I was able to get it up and running except for some small hiccups which I fixed with a small hack. The issue I faced was after following all your instructions when I cam to the step of "make j4" it threw an error saying that it was not able to find -lcudart. And it looked like it was looking for it in "/usr/local/cuda/lib" when the directory itself didn't exist. I fixed this by editing another line in the configure.in file. I changed from CUDA_LIBS="-L$CUDA_DIR/lib $LIB_CUDA -lcudart" to CUDA_LIBS="-L$CUDA_DIR/lib64 $LIB_CUDA -lcudart". Was this change right? Also I noticed that the speedup wasn't extremely huge. The time for one recon-all run went down from 7 hrs 45 mins to 5hrs 20 mins. My understanding is that this is due to only certain modules being cuda-ised (if there is a word like that!) like the mri_ca_register and the mri_em_register. Are there any other modules that are parallelized currently other than these? With regard to the code base I guess I'm also using the read-only git repository of the main development trunk as per the instructions on the freesurfer webpage. Once again thank you very much for that detailed information. Best, Tyson On Mon, Mar 21, 2016 at 6:47 PM, R Edgar wrote: > On 21 March 2016 at 19:59, Francis Tyson Thomas > wrote: > > > The first part was smooth. I got it done pretty quickly (installing CUDA > and > > setting it up!). However for the freesurfer portion I have two questions, > > > > 1. Bruce mentioned in another email thread that v6 beta should be out in > a > > week or so after testing is completed. Should I wait for that version, > > assuming I get the source code for v6 when I download it as per the > > instructions on the freesurfer page. > > Which version of the source code do you have access to? I'm using the > read-only git repository of the main development trunk, sending > patches back to Zeke, which he puts into CVS. I'm not sure which of > those are making it over to v6. > > > 2. Also, would I have to compile and build all modules (mri_em_registe, > > mri_ca_register etc or can I limit to just these two when I build it > again. > > I imagine you'll have to run configure again. When you do, the key > options you want are: > --enable-fermi-gpu --with-cuda="/usr/local/cuda" > Note that on my machine: > [rge21@cudastation ~]$ which nvcc > /usr/local/cuda-7.5/bin/nvcc > [rge21@cudastation ~]$ ls -alF /usr/local/cuda > lrwxrwxrwx. 1 root root 8 Jan 9 16:30 /usr/local/cuda -> cuda-7.5/ > So if your CUDA installation went somewhere else, you'll have to > adjust that --with-cuda path. > > Then when you compile, make sure the nvcc being used really is the one > you've set (things will likely fail horribly if this isn't the case). > If it isn't, you'll have to look in the configure file for: > > # > # Nvidia CUDA enabling > > CUDA_DIR="" > with_cuda="" > > and then comment out > # with_cuda="" > or it will override the directory you set on the command line. > > Let me know if you need further help, > > Richard > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Re: [Freesurfer] OpenMP and GPU implementation in freesurfer 6 beta
Hi Richard, The first part was smooth. I got it done pretty quickly (installing CUDA and setting it up!). However for the freesurfer portion I have two questions, 1. Bruce mentioned in another email thread that v6 beta should be out in a week or so after testing is completed. Should I wait for that version, assuming I get the source code for v6 when I download it as per the instructions on the freesurfer page. 2. Also, would I have to compile and build all modules (mri_em_registe, mri_ca_register etc or can I limit to just these two when I build it again. I'm not very well versed with the second part and so if you could explain a little more it will be good. Thanks, Tyson On Mon, Mar 7, 2016 at 6:38 PM, R Edgar wrote: > On 7 March 2016 at 11:25, Francis Tyson Thomas > wrote: > > > That information was a lot helpful. At this point I'm currently trying to > > reduce the recon-all processing time as much as possible and for this > reason > > I was looking to get the -use-gpu flag working. I'm currently running a > > freesurfer v6 beta version on Ubuntu 14.04.4. With regard to the graphic > > card it is a dual K2200 configuration (I guess they are running in sli > > configuration - although I'm not completely sure). > > > > When you mentioned you compiled everything, I believe you were referring > to > > compiling CUDA 7.5 for Ubuntu 14.04. Because after seeing the link - > > https://developer.nvidia.com/cuda-gpus > > - we settled for CUDA 5 since it was the compatible version mentioned > for > > K2200. Does that mean CUDA 7.5 is backwards compatible with a slight > > tinkering and can be used with freesurfer 6.0 ? > > I think that you might have misunderstood the NVIDIA page. > > It lists the K2200 as a Compute Capability 5 GPU (just like my K1200). > The Compute Capability refers to the hardware: > > http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capabilities > Compute capability 5 is also called "Maxwell," while the work I did on > Freesurfer was around the time of Compute Capability 2, known as > Fermi. In CPU terms, GPU compute capabilities are a bit like Ivy > Bridge vs Sandy Bridge (although I think that GPU features vary more > than the CPU ones). > > In order to program the GPU, you need the NVIDIA CUDA Toolkit, which > contains the required compiler (nvcc). The current version of this is > 7.5. I installed it by following the instructions on NVIDIA's website: > http://docs.nvidia.com/cuda/index.html > This was quite straightforward (certainly moreso than it was five or > six years ago, when you could never be sure that your X11.conf would > survive). > > Either with the toolkit, or as a separate install, you can get a lot > of examples from NVIDIA. I'd suggest grabbing those, and making sure > that you can compile them. The "DeviceQuery" one will probe your PCIe > bus, and report what GPUs it finds. > > > I however tried to setup CUDA 5 following the instructions in the link - > > http://www.unixmen.com/how-to-install-cuda-5-0-toolkit-in-ubuntu/ - > however, > > I'm not able to get it running. I keep getting the following error > "Unable > > to acquire CUDA device". Does this sound familiar ? > > I suspect (although I wasn't following things at the time) that the > Toolkit v5 was before Maxwell cards were released. If so, then it > wouldn't know what to do with the GPUs. > > For the record, since I was writing the CUDA bits so long ago (for the > volume side of things - I didn't do the surface accelerations), they > only use Fermi features. For this reason, you'll want to make sure > that you have --enable-fermi-gpu when you run configure (and make sure > that it's picking out your CUDA installation - I had to tweak the > configure script for this). > > > If you can share some more information in setting this up it will be > great > > since the amount of time recon-all takes is quite too long for running > > multiple datasets. Most importantly we are concerned about the > hippocampal > > segmentation in freesurfer 6 rather than recon-all and so speeding this > up > > would be extremely helpful. > > I don't know if those portions benefit from CUDA acceleration at this > time. I focused on mri_em_register and mri_ca_register. Even if other > programs (which ones are they?) can be linked against some of the > accelerated routines, there is no guarantee of speed up - the time to > shuffle data to and from the GPU is typically greater than the speedup > of any one routine. > > Hope this helps, > > Richard > ___ > Freesurf
[Freesurfer] Freesurfer v6 beta
Hi folks, I was just wondering if there was any major bug in the freesurfer v6 beta that necessitated the removal of the beta from your servers because I cannot find them anymore. Does that mean I will have to go with the dev version at this point which are like the nightlies of the freesurfer v6 updated every few days. Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Re: [Freesurfer] OpenMP and GPU implementation in freesurfer 6 beta
Hi Richard, That information was a lot helpful. At this point I'm currently trying to reduce the recon-all processing time as much as possible and for this reason I was looking to get the -use-gpu flag working. I'm currently running a freesurfer v6 beta version on Ubuntu 14.04.4. With regard to the graphic card it is a dual K2200 configuration (I guess they are running in sli configuration - although I'm not completely sure). When you mentioned you compiled everything, I believe you were referring to compiling CUDA 7.5 for Ubuntu 14.04. Because after seeing the link - https://developer.nvidia.com/cuda-gpus - we settled for CUDA 5 since it was the compatible version mentioned for K2200. Does that mean CUDA 7.5 is backwards compatible with a slight tinkering and can be used with freesurfer 6.0 ? I however tried to setup CUDA 5 following the instructions in the link - http://www.unixmen.com/how-to-install-cuda-5-0-toolkit-in-ubuntu/ - however, I'm not able to get it running. I keep getting the following error "Unable to acquire CUDA device". Does this sound familiar ? If you can share some more information in setting this up it will be great since the amount of time recon-all takes is quite too long for running multiple datasets. Most importantly we are concerned about the hippocampal segmentation in freesurfer 6 rather than recon-all and so speeding this up would be extremely helpful. Thank you, Tyson On Thu, Mar 3, 2016 at 2:23 PM, R Edgar wrote: > On 3 March 2016 at 13:22, Francis Tyson Thomas > wrote: > > > Also, it looks like development for GPU usage has been halted for now > and so > > I was trying to use the CUDA 5 for getting it working under ubuntu > 14.04. I > > have been not successful that as the cuda device isn't getting selected. > Do > > you have any recommendations on that? > > I have recently started looking at the CUDA port again, although I'm > making no promises as to the amount of time I'll have to spend on it. > This said I've got mri_ca_register running with CUDA 7.5 on my > machine. On my E3 Xeon with a K1200, I can run mri_ca_register with > the test dataset in about 9 minutes. > > Are you compiling from source? I had to tamper a bit with the > configure script before compiling CUDA was enabled again. I don't have > access to the right machine at the moment, but as I recall, there was > a line > with_cuda="" > which I had to comment out, since it was overriding the path to the > machine's CUDA installation which I was passing on the command line. > There are a few other minor bug fixes and performance improvements for > GPU code which I've submitted for Zeke; no new kernels yet, I'm > afraid. > > If you could give me some more details about what you're doing, I may > be able to help. > > Just the standard warning: the GPU results will be different from the > CPU results on the same inputs. We've been kicking around some ideas > recently to quantify how different (and to devise input datasets where > the correct answers are unambiguous). > > Regards, > > Richard > ___ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > > ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
[Freesurfer] OpenMP and GPU implementation in freesurfer 6 beta
Hi, I have been working on speeding up the the hippocampal segmentation in freesurfer 6 beta and I have been playing around with both -openmp and -use-gpu flags. I noticed that freesurfer does seem to make use of all the threads in freesurfer 6 irrespective of passing the -openmp flag or not. Is that the right behaviour. I found this by analysing the cpu usage in both the cases. Also, it looks like development for GPU usage has been halted for now and so I was trying to use the CUDA 5 for getting it working under ubuntu 14.04. I have been not successful that as the cuda device isn't getting selected. Do you have any recommendations on that? And lastly, there is the itkthreads option available to -hippocampal-subfields-T1, so can I combine all three or two of these options to minimize the execution time - like GPU and itkthreads or GPU, OPENMP and itkthreads? Also the v6 beta is no longer available on your website. Is there any particular reason as to why it was removed and can I continue using the copy that I have for hippocampal segmentation ? Thanks, Tyson ___ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
