Dear *, we also run Freesurfer on a cluster and are starting to use singularity, so naturally I'm curious about this issue.
I was able to reproduce the error with a clean singularity run (-c -e).
Then, I rebuilt a container to start nu_correct with "strace -f -e
open,write" and I believe I found the problem:
[pid 35994] open("/subs/testsub/mri/tmp.mri_nu_correct.mni.35410/nu1.imp",
O_RDONLY) = 3
[pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
[pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_TRUNC, 0666) = 5
[pid 35994] write(5, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 35994] write(5,
"CDF\1\0\0\0\0\0\0\0\n\0\0\0\3\0\0\0\6xspace\0\0\0\0\1\0"..., 8192) = 8192
[pid 35994] write(5,
"|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"...,
8192) = 8192
[...]
[pid 35994] write(5,
"|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"...,
8192) = -1 ENOSPC (No space left on device)
[pid 35994] write(2, "ncendef: ", 9ncendef: ) = 9
[pid 35994] write(2, "ncid 5", 6ncid 5) = 6
[pid 35994] write(2, ": No space left on device", 25: No space left on
device) = 25
So there are two issues here I think:
1) nu_correct is saving things to /tmp when it could just use ./tmp
2) the singularity default for "sessiondir max size" is 16MB, which includes
/tmp, is too small for the minc-tmp files
I changed the value from 16 to 1024 and it passed the nu_correct step.
cheers,
Michael
On 10/5/18 5:57 PM, Greve, Douglas N.,Ph.D. wrote:
> Not that I know of, but this is not a native FS script, it comes from the
> MNI, so it could be doing something we don't understand. Can you do an
> experiment for us? Can you run it outside of the container using reprozip
> (or something equivalent) to see all the files it touches?
>
> On 10/5/18 5:31 AM, Jasper van den Bosch wrote:
>>
>> External Email - Use Caution
>>
>> We are trying to run recon_all inside an fmriprep singularity container,
>> but are running into an issue with nu_correct. It says it cannot write to
>> disk, however there is plenty of space on the tmpdir and output dir. Does
>> this script try to write to any other locations? Because this would fail
>> inside the container, where each writable location has to be 'mounted'.
>>
>> Error message:
>>
>> ncendef: ncid 5: No space left on device
>> Error outputting volume: possibly disk full?
>> nu_evaluate: crashed while running evaluate_field (termination status=139)
>> nu_correct: crashed while running nu_evaluate (termination status=65280)
>> ERROR: nu_correct
>>
>> Command ran:
>>
>> nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc
>> ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir
>> ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50
>> [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/]
>> [2018-10-04 14:59:29] running:
>> /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15
>> 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline
>> 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir
>> ./tmp.mri_nu_correct.mni.1121/0/ ./tmp.mri_nu_correct.mni.1121/nu0.mnc
>> ./tmp.mri_nu_correct.mni.1121/nu1.imp
>>
>>
>> this issue at the fmriprep
>> repo: https://github.com/poldracklab/fmriprep/issues/1308
>> possibly related: https://github.com/freesurfer/freesurfer/issues/462
>>
>> Many thanks for any advice,
>> Jasper van den Bosch
>>
>>
>>
>> _______________________________________________
>> Freesurfer mailing list
>> Freesurfer@nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
>
> _______________________________________________
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
>
signature.asc
Description: OpenPGP digital signature
External Email - Use Caution
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
