Hi, I'm a first-time user of the PETSc toolkit.
I'm getting a "Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" error. My program (called "pPCA_makeC0" below) is doing simple stuff : create a bunch of vectors, set its components to random numbers using VecSetRandom(), then compute the NORM_2 type vector-norm for each of them, and finally write them to a binary file. My vectors are quite large (2821728 dimensions) and even if I create just one such vector, the above-mentioned error occurs. From the output of the program, it appears that MPI is having some issue with the binary-file-writing part of the program. This problem occurs regardless of the number of processes/processors I use when invoking petscmpiexec. I've pasted the erroneous output at the end of this mail, for two scenarios: 1 processor and 1 or 5 processes. Any help in resolving this would be much appreciated. Thanks! Best, Harsh ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- output of my program (D = vector-dimension = 2821728 here, k = number of vectors = 5 here), 1 process, 1 processor ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec -np 1 ./pPCA_makeC0 -c 1 -m /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5 2-norm of column 1 of C0 = 969.272 2-norm of column 2 of C0 = 969.218 2-norm of column 3 of C0 = 969.087 2-norm of column 4 of C0 = 969.599 2-norm of column 5 of C0 = 969.547 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 src/sys/fileio/sysio.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:03:44 2010 [0]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 [0]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 rank 0 in job 3 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all ranks exit status of rank 0: return code 59 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- output of my program (D = vector-dimension = 2821728 here, k = number of vectors = 5 here), 5 processes, 1 processor ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec -np 5 ./pPCA_makeC0 -c 1 -m /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5 2-norm of column 1 of C0 = 970.149 2-norm of column 2 of C0 = 969.699 2-norm of column 3 of C0 = 969.517 2-norm of column 4 of C0 = 970.253 2-norm of column 5 of C0 = 969.81 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Write to file failed! [1]PETSC ERROR: Error writing to file.! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010 [1]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 [1]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cwo[2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Write to file failed! [2]PETSC ERROR: Error writing to file.! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: See docs/index.html for manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010 [2]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 [2]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c [1]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in src/sys/fileio/sysio.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Error: Unable to write C0 row dimension D! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html forkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c [2]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in src/sys/fileio/sysio.c [2]PETSC ERROR: --------------------- Error Message ------------------------------------ [2]PETSC ERROR: Error: Unable to write C0 row dimension D! [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010 [2]PETSC ERROR: See docs/changes/index.html for recent updates. [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [2]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010 [1]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 [1]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: User provided function() line 123 in /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c r manual pages. [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010 [2]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 [2]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1 [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: User provided function() line 123 in /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 src/sys/fileio/sysio.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010 [0]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010 [0]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file [3]PETSC ERROR: rank 2 in job 4 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all ranks exit status of rank 2: return code 1 rank 1 in job 4 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all ranks exit status of rank 1: return code 1 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20100415/d7dcc518/attachment.htm>
