On one cluster that has default memory limits, I get this error, even when I
override those limits with qsub up to 16GB. On another cluster, without any
pre-set limits, I do not get the error. But when I qsub with a memory limit
(even 8GB), then the errors happen on this cluster too. So there is plenty
of memory, but somehow just having a limit at all seems to cause problems for
vtkmultithreader.
For now, I guess the solution will be to disable the ribbon making. Thanks,
Nick, for pointing out the downstream dependencies.
From: christopher.nu...@loni.ucla.edu
To: freesurfer@nmr.mgh.harvard.edu
Date: Tue, 9 Jun 2009 10:16:39 -0700
Subject: FW: RE: [Freesurfer] vtkMultiThreader error
I am getting similar error messages when running on our grid and pipeline
systems (see below). Our tech people had previously thought this could be
related to memory capacity issues, but after increasing memory to 8gb when
submitting my jobs with a qsub command, the problem is still not resolved. I'm
beginning to have doubts about this being related to memory capacity.
Could somebody please provide instructions on disabling the ribbon processing?
Thank you.
#...@# Cortical ribbon mask Sun May 31 18:21:54 PDT 2009
/ifs/edevel/CN_SAN_DIEGO/FREESURFER/OUTPUT/164/mri
\n mris_volmask --label_left_white 2 --label_left_ribbon 3 --label_right_white
41 --label_right_ribbon 42 --save_ribbon --save_distance 164 \n
ERROR: In
/usr/pubsw/packages/vtk/5.1.0-CVS/src/VTK/Common/vtkMultiThreader.cxx, line 372
vtkMultiThreader (0x3279bb60): Unable to create a thread. pthread_create()
returned 12
ERROR: In
/usr/pubsw/packages/vtk/5.1.0-CVS/src/VTK/Common/vtkMultiThreader.cxx, line 372
vtkMultiThreader (0x3279bb60): Unable to create a thread. pthread_create()
returned 12
ERROR: In
/usr/pubsw/packages/vtk/5.1.0-CVS/src/VTK/Common/vtkMultiThreader.cxx, line 372
vtkMultiThreader (0x3279bb60): Unable to create a thread. pthread_create()
returned 12
ERROR: In
/usr/pubsw/packages/vtk/5.1.0-CVS/src/VTK/Common/vtkMultiThreader.cxx, line 372
vtkMultiThreader (0x3279bb60): Unable to create a thread. pthread_create()
returned 12
ERROR: In
/usr/pubsw/packages/vtk/5.1.0-CVS/src/VTK/Common/vtkMultiThreader.cxx, line 372
vtkMultiThreader (0x3279bb60): Unable to create a thread. pthread_create()
returned 12
ERROR: In
/usr/pubsw/packages/vtk/5.1.0-CVS/src/VTK/Common/vtkMultiThreader.cxx, line 372
vtkMultiThreader (0x3279bb60): Unable to create a thread. pthread_create()
returned 12
ERROR: In
/usr/pubsw/packages/vtk/5.1.0-CVS/src/VTK/Common/vtkMultiThreader.cxx, line 372
vtkMultiThreader (0x3279bb60): Unable to create a thread. pthread_create()
returned 12
-------------------------------------------------------------------
S. Christopher Nuñez, PhD
Postdoctoral Fellow
Developmental Cognitive Neuroimaging Group
Laboratory of Neuro Imaging
UCLA Dept of Neurology
Neuroscience Research Building 1, Suite 225
635 Charles Young Drive South
Los Angeles, CA 90095-7334
(310) 267-5555
-------------------------------------------------------------------
________________________________________
From: hranilov...@ucla.edu [hranilov...@ucla.edu]
Sent: Monday, June 08, 2009 12:13 PM
To: Christopher Nunez
Subject: Fwd: RE: [Freesurfer] vtkMultiThreader error
Pretty much what I was saying the other day. It looks like maybe you
can disable this step if you don't want this output, though?
----- Forwarded message from dhagle...@hotmail.com -----
Date: Mon, 8 Jun 2009 11:19:43 -0700
From: Don Hagler <dhagle...@hotmail.com>
Reply-To: Don Hagler <dhagle...@hotmail.com>
Subject: RE: [Freesurfer] vtkMultiThreader error
To: ni...@nmr.mgh.harvard.edu, martin.ka...@gmail.com
Cc: freesurfer maillist <freesurfer@nmr.mgh.harvard.edu>
Hi Nick,
Thanks for the reply. I will look into the memory limit issue some
more (we recently instituted per job memory limits), but I have done a
test run with a 16GB memory limit (running one job per node).
Meanwhile, thanks for the novtk suggestion. Our fallback option is to
just disable the ribbon generation. I don't think we ever use them
for anything.
Don
> Subject: Re: [Freesurfer] vtkMultiThreader error
> From: ni...@nmr.mgh.harvard.edu
> To: martin.ka...@gmail.com; dhagle...@hotmail.com
> Date: Mon, 8 Jun 2009 11:15:51 -0400
> CC: freesurfer@nmr.mgh.harvard.edu
>
> Don and Martin,
>
> I dont have a good solution for you to this problem. The error code
> indicates an 'out-of-memory' error returned by pthread_create, and I do
> know the mris_vol_mask uses a lot of memory (at least 1GB).
>
> I have not been able to find any clues to the source of this problem on
> the web either. The only curious thing to it is that in the source
> code, that section is conditionally compiled to ignore the return code
> if 'VTK_HP_PTHREADS' is defined, which implies some sort of odd behavior
> on some special HP platform build.
>
> mris_vol_mask has been a real thorn in the side of the recon-all stream,
> but its not easy to replace because it does a subjectively better job of
> finding the voxels which intersect the pial and white surfaces. For
> comparison, or even to solve your problem, you can use
> mris_vol_mask_novtk to create the ribbon.mgz files. The output is not
> the same as that produced by mris_vol_mask though. Perhaps you can run
> it, and compare the results side-by-side and give us your opinion on
> which is the better ribbon (as there is no objective measure on what a
> 'correct' ribbon.mgz should like like, ie, should it include voxels that
> intersect more than half way through a surface, or not at all?).
>
> Nick
>
>
> On Mon, 2009-06-08 at 09:16 +0200, Martin Kavec wrote:
> > Hi Don,
> > > I am in a similar situation. I am on a IBM BladeCenter cluster
> (though with > one blade only) with 2 Quad core Intels, and I
> experience intermittent > crashes of mri_volmask as well. Similarly
> as you, when I run the mri_volmask > on the crashed subject on a
> desktop, this passes perfectly fine and sometimes > also on the
> cluster as well.
> > > I am wonder whether how do you obtain the debug information
> about the cause of > the crash (being in the vtkMultiThreader)?
> > > I also wonder, whether the crashes could be due to different
> compilation > environments of the VTK libraries used in teh FS
> distribution (centos) and > your redhat. I went on the bug-report
> site of VTK and there really isn't > anythink major reported on this
> class. > > In my experience the mri_volmask runs fine for quite a
> while and then crashes. > >From the debug messages you show it would
> thus could be that the process may > hit some limit of the threads
> which could be allocated to a process.
> > > What do you think?
> > > Martin > > On Saturday 06 June 2009 04:19:51 Don Hagler wrote:
> > > When we run v430 on our cluster, running rocks redhat 4.3, we
> get the error
> > > message below when mris_volmask is run. From a google search, I see that
> > > someone had the same error message back in Oct 2007. Was a
> solution found
> > > for this? It runs fine on our desktop machines as well as when we log in
> > > directly to a node and run the job interactively.
> > >
> > >
> > >
> > > mris_volmask --label_left_white 2 --label_left_ribbon 3
> > > --label_right_white 41 --label_right_ribbon 42 --save_ribbon
> > > --save_distance FREESURFERRECON_NTNU_PRE_095v2_20071126.173245.968000_1
> > >
> > > SUBJECTS_DIR is /home/dhagler/data_test
> > > ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx,
> > > line 372 vtkMultiThreader (0x1306ec80): Unable to create a
> thread. > > pthread_create() returned 12
> > >
> > > ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx,
> > > line 372 vtkMultiThreader (0x1306ec80): Unable to create a
> thread. > > pthread_create() returned 12
> > >
> > > ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx,
> > > line 372 vtkMultiThreader (0x1306ec80): Unable to create a
> thread. > > pthread_create() returned 12
> > >
> > > ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx,
> > > line 372 vtkMultiThreader (0x1306ec80): Unable to create a
> thread. > > pthread_create() returned 12
> > >
> > > ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx,
> > > line 372 vtkMultiThreader (0x1306ec80): Unable to create a
> thread. > > pthread_create() returned 12
> > >
> > > ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx,
> > > line 372 vtkMultiThreader (0x1306ec80): Unable to create a
> thread. > > pthread_create() returned 12
> > >
> > > ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx,
> > > line 372 vtkMultiThreader (0x1306ec80): Unable to create a
> thread. > > pthread_create() returned 12
> > >
> > > Segmentation fault
> > > Linux compute-0-12.local 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26
> 14:14:47 EDT
> > > 2007 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > recon-all exited with ERRORS at Fri Jun 5 18:58:09 PDT 2009
> > >
> > >
> > >
> > >
> > >
> > > _________________________________________________________________
> > > Hotmail® has ever-growing storage! Don't worry about storage limits.
> > >
> http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutori
> > >al_Storage_062009
> > > > _______________________________________________
> > Freesurfer mailing list
> > Freesurfer@nmr.mgh.harvard.edu
> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
> > > _______________________________________________
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_________________________________________________________________
Insert movie times and more without leaving Hotmail®.
http://windowslive.com/Tutorial/Hotmail/QuickAdd?ocid=TXT_TAGLM_WL_HM_Tutorial_QuickAdd_062009
_________________________________________________________________
Hotmail® has ever-growing storage! Don’t worry about storage limits.
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage_062009_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
