Hello,
As per your advice I am trying to pass the row and column indices to
MatCreateMPIAIJWithArrays(),
but there is a confusion, if we follow the matrix given in the manual.
---------------------------------------------------------------------------------------------------------------------------------------------
The i and j indices are 0 based, and i indices are indices corresponding to the
local j array.
3990: The format which is used for the sparse matrix input, is
equivalent to a
3991: row-major ordering.. i.e for the following matrix, the input data
expected is
3992: as shown:
3994: 1 0 0
3995: 2 0 3 P0
3996: -------
3997: 4 5 6 P1
3999: Process0 [P0]: rows_owned=[0,1]
4000: i = {0,1,3} [size = nrow+1 = 2+1]
4001: j = {0,0,2} [size = nz = 6]
4002: v = {1,2,3} [size = nz = 6]
4004: Process1 [P1]: rows_owned=[2]
4005: i = {0,3} [size = nrow+1 = 1+1]
4006: j = {0,1,2} [size = nz = 6]
4007: v = {4,5,6} [size = nz = 6]
---------------------------------------------------------------------------------------------------------------------------------------------
--> the i and j array should be 0 based.
if I want to add one row in P2 as below then what would be the i and j array
for that?
1 0 0
2 0 3 P0
-------
4 5 6 P1
-------
0 0 7 P2
Because the start position for the non-zero value is 2.
Thanks
Priyank
On 13-Aug-2014, at 6:11 pm, Barry Smith <[email protected]> wrote:
>
> Usually the matrix is generated in parallel so each process has part of the
> matrix. If the matrix is generated sequentially then that will limit the
> parallel performance you can get.
>
> The MPI model is not particularly friendly to “some parts of my code are
> purely sequential, but I want other parts to be parallel”
>
> Barry
>
> On Aug 13, 2014, at 3:52 PM, Priyank Patel <[email protected]> wrote:
>
>> Thank you Barry,
>>
>> But what if I have a matrix that is 1000000 X 1000000 and with 32
>> processors? I would be a very time consuming task to indicate column and row
>> indices for all of them I guess. Do you think there is another way to do it?
>> I just want to do parallel matrix matrix multiplication.
>>
>> Priyank Patel
>> [email protected]
>>
>>
>>
>>
>> On 12-Aug-2014, at 9:20 pm, Barry Smith <[email protected]> wrote:
>>
>>>
>>> On each process you should only pass the row and column indices for the
>>> rows you want on that process. In other words do not pass the entire matrix
>>> on each process. Take a look at the example matrix in the manual page
>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html
>>>
>>> Barry
>>>
>>> On Aug 12, 2014, at 8:02 PM, Priyank Patel <[email protected]> wrote:
>>>
>>>> Hello,
>>>> I am trying to do sparse - dense matrix multiplication and for that I am
>>>> using a CSR Matrix(that are three array) in PETSC using the function
>>>> MatCreateMPIAIJWithArrays() to create a petsc matrix.
>>>> My call to routine is as below
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> //Initialize PETSC
>>>> MPI_Init(&argc, &argv);
>>>> PetscInitialize(&argc, &argv, (char*) 0, help);
>>>>
>>>> MPI_Comm_rank(PETSC_COMM_WORLD, &taskid);
>>>> MPI_Comm_size(PETSC_COMM_WORLD, &numtasks);
>>>>
>>>> //Create A Sparse Matrix fro the files
>>>> ierr = MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, m, m,
>>>> PETSC_DETERMINE, PETSC_DETERMINE, h_iA, h_jA, h_valA, &A);CHKERRQ(ierr);
>>>>
>>>> //ierr = MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, m, m, h_iA, h_jA,
>>>> h_valA, &A);CHKERRQ(ierr);
>>>>
>>>> MatView(A,PETSC_VIEWER_STDOUT_WORLD);
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> My make file is as below
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> PETSC_DIR=/export/software/petsc-3.3
>>>>
>>>> INTEL_ROOT=/export/software/intel/composerxe-2011.4.191/
>>>> MKL_LIB_PATH_64=/export/software/intel/mkl/lib/intel64/
>>>> INTEL_LIB_PATH_64=/export/software/intel/composerxe-2011.4.191/compiler/lib/intel64/
>>>> MKL_INC_PATH=/export/software/intel/mkl/include/
>>>> SCALAPACK_LIB_PATH=/export/software/scalapack-2.0.2/lib/
>>>> SCALAPACK_INC_PATH=/export/software/scalapack-2.0.2/
>>>> GENERAL_LIB_PATH=/export/software/lib/
>>>>
>>>> CLINKER=mpicc
>>>>
>>>> PKG_INC = -I${MKL_INC_PATH}
>>>> PKG_LIB = -Wl,-rpath,${MKL_LIB_PATH_64} -L${MKL_LIB_PATH_64}
>>>> -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core
>>>> -Wl,-rpath,${INTEL_LIB_PATH_64} -L${INTEL_LIB_PATH_64} -liomp5
>>>> -Wl,-rpath,${SCALAPACK_LIB_PATH} -L${SCALAPACK_LIB_PATH} -lscalapack
>>>> -llapack -lblas
>>>>
>>>> CFLAGS =
>>>> FFLAGS =
>>>> CPPFLAGS = ${PKG_INC}
>>>> FPPFLAGS =
>>>> CLEANFILES =
>>>>
>>>> include ${PETSC_DIR}/conf/variables
>>>> include ${PETSC_DIR}/conf/rules
>>>>
>>>> blockCG_1: blockCG_1.o chkopts
>>>> -${CLINKER} -o blockCG_1 blockCG_1.o ${PKG_LIB} ${PETSC_LIB}
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> And I tried to run the program with following command
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> /opt/openmpi/bin/mpirun -np 3 ./blockCG_1 /home/ppatel/MPI/LF10_2
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> but the routine is not distributing the matrix and give me the result as
>>>> below.
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> loading matrix from : /home/ppatel/MPI/LF10_2
>>>> nRows: 18, nCoefs: 82.
>>>> Matrix Object: 1 MPI processes
>>>> type: mpiaij
>>>> row 0: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 1: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 2: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 3: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
>>>> 477.155)
>>>> row 4: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
>>>> 1.76724)
>>>> row 5: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
>>>> 477.155)
>>>> row 6: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
>>>> 1.76724)
>>>> row 7: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
>>>> 477.155)
>>>> row 8: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
>>>> 1.76724)
>>>> row 9: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
>>>> 477.155)
>>>> row 10: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
>>>> 1.76724)
>>>> row 11: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
>>>> 477.155)
>>>> row 12: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
>>>> 1.76724)
>>>> row 13: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
>>>> 477.155)
>>>> row 14: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
>>>> 1.76724)
>>>> row 15: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 16: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 17: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 18: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 19: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 20: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 21: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
>>>> 477.155)
>>>> row 22: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
>>>> 1.76724)
>>>> row 23: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
>>>> 477.155)
>>>> row 24: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
>>>> 1.76724)
>>>> row 25: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
>>>> 477.155)
>>>> row 26: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
>>>> 1.76724)
>>>> row 27: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
>>>> 477.155)
>>>> row 28: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
>>>> 1.76724)
>>>> row 29: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
>>>> 477.155)
>>>> row 30: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
>>>> 1.76724)
>>>> row 31: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
>>>> 477.155)
>>>> row 32: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
>>>> 1.76724)
>>>> row 33: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 34: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 35: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 36: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 37: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 38: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 39: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
>>>> 477.155)
>>>> row 40: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
>>>> 1.76724)
>>>> row 41: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
>>>> 477.155)
>>>> row 42: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
>>>> 1.76724)
>>>> row 43: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
>>>> 477.155)
>>>> row 44: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
>>>> 1.76724)
>>>> row 45: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
>>>> 477.155)
>>>> row 46: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
>>>> 1.76724)
>>>> row 47: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
>>>> 477.155)
>>>> row 48: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
>>>> 1.76724)
>>>> row 49: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
>>>> 477.155)
>>>> row 50: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
>>>> 1.76724)
>>>> row 51: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 52: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 53: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 54: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 55: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 56: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 57: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
>>>> 477.155)
>>>> row 58: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
>>>> 1.76724)
>>>> row 59: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
>>>> 477.155)
>>>> row 60: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
>>>> 1.76724)
>>>> row 61: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
>>>> 477.155)
>>>> row 62: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
>>>> 1.76724)
>>>> row 63: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
>>>> 477.155)
>>>> row 64: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
>>>> 1.76724)
>>>> row 65: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
>>>> 477.155)
>>>> row 66: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
>>>> 1.76724)
>>>> row 67: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
>>>> 477.155)
>>>> row 68: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
>>>> 1.76724)
>>>> row 69: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 70: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 71: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 72: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 73: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 74: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 75: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
>>>> 477.155)
>>>> row 76: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
>>>> 1.76724)
>>>> row 77: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
>>>> 477.155)
>>>> row 78: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
>>>> 1.76724)
>>>> row 79: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
>>>> 477.155)
>>>> row 80: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
>>>> 1.76724)
>>>> row 81: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
>>>> 477.155)
>>>> row 82: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
>>>> 1.76724)
>>>> row 83: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
>>>> 477.155)
>>>> row 84: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
>>>> 1.76724)
>>>> row 85: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9) (16,
>>>> 477.155)
>>>> row 86: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
>>>> 1.76724)
>>>> row 87: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 88: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 89: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>> row 90: (0, 3.53448) (1, -477.155) (2, 1.76724)
>>>> row 91: (0, -477.155) (1, 171776) (3, -85887.9) (4, 477.155)
>>>> row 92: (0, 1.76724) (2, 7.06896) (3, -477.155) (4, 1.76724)
>>>> row 93: (1, -85887.9) (2, -477.155) (3, 171776) (5, -85887.9) (6,
>>>> 477.155)
>>>> row 94: (1, 477.155) (2, 1.76724) (4, 7.06896) (5, -477.155) (6,
>>>> 1.76724)
>>>> row 95: (3, -85887.9) (4, -477.155) (5, 171776) (7, -85887.9) (8,
>>>> 477.155)
>>>> row 96: (3, 477.155) (4, 1.76724) (6, 7.06896) (7, -477.155) (8,
>>>> 1.76724)
>>>> row 97: (5, -85887.9) (6, -477.155) (7, 171776) (9, -85887.9) (10,
>>>> 477.155)
>>>> row 98: (5, 477.155) (6, 1.76724) (8, 7.06896) (9, -477.155) (10,
>>>> 1.76724)
>>>> row 99: (7, -85887.9) (8, -477.155) (9, 171776) (11, -85887.9) (12,
>>>> 477.155)
>>>> row 100: (7, 477.155) (8, 1.76724) (10, 7.06896) (11, -477.155) (12,
>>>> 1.76724)
>>>> row 101: (9, -85887.9) (10, -477.155) (11, 171776) (13, -85887.9) (14,
>>>> 477.155)
>>>> row 102: (9, 477.155) (10, 1.76724) (12, 7.06896) (13, -477.155) (14,
>>>> 1.76724)
>>>> row 103: (11, -85887.9) (12, -477.155) (13, 171776) (15, -85887.9)
>>>> (16, 477.155)
>>>> row 104: (11, 477.155) (12, 1.76724) (14, 7.06896) (15, -477.155) (16,
>>>> 1.76724)
>>>> row 105: (13, -85887.9) (14, -477.155) (15, 171776) (17, 477.155)
>>>> row 106: (13, 477.155) (14, 1.76724) (16, 7.06896) (17, 1.76724)
>>>> row 107: (15, 477.155) (16, 1.76724) (17, 3.53448)
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> that is creating a matrix with rows repeated 6 times instead of
>>>> distributed over 6 process.
>>>>
>>>> Can you please tell me where I am doing wrong??
>>>>
>>>> Thanks for your time
>>>>
>>>> Priyank Patel
>>>> [email protected]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>
>
>
>
> --
> BEGIN-ANTISPAM-VOTING-LINKS
> ------------------------------------------------------
>
> Teach CanIt if this mail (ID 03MCmbAxj) is spam:
> Spam:
> https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=s
> Not spam:
> https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=n
> Forget vote:
> https://www.spamtrap.odu.edu/canit/b.php?i=03MCmbAxj&m=52032b34dad0&t=20140813&c=f
> ------------------------------------------------------
> END-ANTISPAM-VOTING-LINKS
