You can get a stack trace using the debugger. We also recommend tracking this down using valgrind.
Matt On Thu, Apr 15, 2010 at 9:14 PM, Harsh Sharma <hsharma.tgjobs at gmail.com>wrote: > Hi, > > I'm a first-time user of the PETSc toolkit. > > I'm getting a "Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range" error. > > My program (called "pPCA_makeC0" below) is doing simple stuff : create a > bunch of vectors, set its components to random numbers using VecSetRandom(), > then compute the NORM_2 type vector-norm for each of them, and finally write > them to a binary file. > > My vectors are quite large (2821728 dimensions) and even if I create just > one such vector, the above-mentioned error occurs. From the output of the > program, it appears that MPI is having some issue with the > binary-file-writing part of the program. > > This problem occurs regardless of the number of processes/processors I use > when invoking petscmpiexec. I've pasted the erroneous output at the end of > this mail, for two scenarios: 1 processor and 1 or 5 processes. > > Any help in resolving this would be much appreciated. Thanks! > > Best, > Harsh > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > output of my program (D = vector-dimension = 2821728 here, k = number of > vectors = 5 here), 1 process, 1 processor > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec > -np 1 ./pPCA_makeC0 -c 1 -m > /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5 > 2-norm of column 1 of C0 = 969.272 > 2-norm of column 2 of C0 = 969.218 > 2-norm of column 3 of C0 = 969.087 > 2-norm of column 4 of C0 = 969.599 > 2-norm of column 5 of C0 = 969.547 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: > or try > http://valgrind.org on linux or man libgmalloc on Apple to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c > [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c > [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c > [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 > src/sys/fileio/sysio.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 > CST 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by > hsharma Thu Apr 15 21:03:44 2010 > [0]PETSC ERROR: Libraries linked from > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib > [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 > [0]PETSC ERROR: Configure options > --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal > --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib > --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install > --with-large-file-io=1 --with-shared=0 --with-scalar-type=real > --with-precision=single --with-c++-support --with-c-support > --with-64-bit-indices=0 --with-log=1 --with-info=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > rank 0 in job 3 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all > ranks > exit status of rank 0: return code 59 > > > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > output of my program (D = vector-dimension = 2821728 here, k = number of > vectors = 5 here), 5 processes, 1 processor > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec > -np 5 ./pPCA_makeC0 -c 1 -m > /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5 > 2-norm of column 1 of C0 = 970.149 > 2-norm of column 2 of C0 = 969.699 > 2-norm of column 3 of C0 = 969.517 > 2-norm of column 4 of C0 = 970.253 > 2-norm of column 5 of C0 = 969.81 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: > or try > http://valgrind.org on linux or man libgmalloc on Apple to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Write to file failed! > [1]PETSC ERROR: Error writing to file.! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 > CST 2010 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by > hsharma Thu Apr 15 21:09:23 2010 > [1]PETSC ERROR: Libraries linked from > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib > [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 > [1]PETSC ERROR: Configure options > --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal > --with-blas-lapack-dir=/cwo[2]PETSC ERROR: --------------------- Error > Message ------------------------------------ > [2]PETSC ERROR: Write to file failed! > [2]PETSC ERROR: Error writing to file.! > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 > CST 2010 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by > hsharma Thu Apr 15 21:09:23 2010 > [2]PETSC ERROR: Libraries linked from > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib > [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 > [2]PETSC ERROR: Configure options > --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal > --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib > --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install > --with-large-file-io=1 --with-shared=0 --with-scalar-type=real > --with-precision=single --with-c++-support --with-c-support > --with-64-bit-indices=0 --with-log=1 --with-info=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c > [1]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in > src/sys/fileio/sysio.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Error: Unable to write C0 row dimension D! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 > CST 2010 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html > forkspace/ifp-32-2/hasegawa/hsharma/apps/myLib > --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install > --with-large-file-io=1 --with-shared=0 --with-scalar-type=real > --with-precision=single --with-c++-support --with-c-support > --with-64-bit-indices=0 --with-log=1 --with-info=1 > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c > [2]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in > src/sys/fileio/sysio.c > [2]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [2]PETSC ERROR: Error: Unable to write C0 row dimension D! > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 > CST 2010 > [2]PETSC ERROR: See docs/changes/index.html for recent updates. > [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [2]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by > hsharma Thu Apr 15 21:09:23 2010 > [1]PETSC ERROR: Libraries linked from > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib > [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 > [1]PETSC ERROR: Configure options > --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal > --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib > --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install > --with-large-file-io=1 --with-shared=0 --with-scalar-type=real > --with-precision=single --with-c++-support --with-c-support > --with-64-bit-indices=0 --with-log=1 --with-info=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: User provided function() line 123 in > /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c > r manual pages. > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by > hsharma Thu Apr 15 21:09:23 2010 > [2]PETSC ERROR: Libraries linked from > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib > [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 > [2]PETSC ERROR: Configure options > --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal > --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib > --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install > --with-large-file-io=1 --with-shared=0 --with-scalar-type=real > --with-precision=single --with-c++-support --with-c-support > --with-64-bit-indices=0 --with-log=1 --with-info=1 > [2]PETSC ERROR: > ------------------------------------------------------------------------ > [2]PETSC ERROR: User provided function() line 123 in > /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c > [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c > [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c > [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 > src/sys/fileio/sysio.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 > CST 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by > hsharma Thu Apr 15 21:09:23 2010 > [0]PETSC ERROR: Libraries linked from > /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib > [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 > [0]PETSC ERROR: Configure options > --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal > --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib > --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install > --with-large-file-io=1 --with-shared=0 --with-scalar-type=real > --with-precision=single --with-c++-support --with-c-support > --with-64-bit-indices=0 --with-log=1 --with-info=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > [3]PETSC ERROR: rank 2 in job 4 ifp-32.ifp.uiuc.edu_57355 caused > collective abort of all ranks > exit status of rank 2: return code 1 > rank 1 in job 4 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all > ranks > exit status of rank 1: return code 1 > > > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20100415/913c499d/attachment-0001.htm>
