a couple of small mistakes in the previous c++ file:
The memory_read(..) call should be:
// read data back into our product buffers
viennacl::backend::memory_read(handle1, product_size_row * 4, 0,
product_row_ptr, false);
viennacl::backend::memory_read(handle2, product_NNz * 4, 0, product_col_ptr,
false);
viennacl::backend::memory_read(handle, product_NNz * 8, 0,
product_values_ptr, false);
(read product_NNz * x bytes instead of product_size_row * x)
I've attached the corrected file.
Thanks
Andy
________________________________
From: Andrew Palumbo <ap....@outlook.com>
Sent: Thursday, July 21, 2016 11:03:59 PM
To: Karl Rupp; viennacl-devel
Subject: Re: [ViennaCL-devel] Copying Values out of a compressed_matrix
Hello,
I've mocked up a sample of the compressed_matrix multiplication that I've been
working with javacpp on in C++. I am seeing the same type of memory errors
when I try to read the data out of product, and into the output buffers as I
was with javacpp. By printing the matrix to stdout as in the compressed_matrix
example we can see that there are values there, and they seem reasonable, but
when i use backend::memory_read(...) to retrive the buffers, I'm getting
values consistent with a memory error, and similar to what i was seeing in the
javacpp code. Maybe I am not using the handles correctly? Admittedly my C++
is more than rusty, but I believe I am referencing the buffers correctly in the
output.
Below is the output of the attached file: sparse.cpp
Thanks very much,
Andy
ViennaCL: compressed_matrix of size (10, 10) with 24 nonzeros:
(1, 2) 0.329908
(1, 3) 0.0110522
(1, 4) 0.336839
(2, 5) 0.0150778
(2, 7) 0.0143518
(3, 3) 0.217256
(3, 6) 0.346854
(3, 9) 0.45353
(4, 3) 0.407954
(4, 6) 0.651308
(5, 2) 0.676061
(5, 3) 0.0226486
(5, 4) 0.690264
(6, 5) 0.0998838
(6, 7) 0.0950744
(7, 2) 0.346173
(7, 3) 0.0115971
(7, 4) 0.353446
(7, 9) 0.684458
(8, 5) 0.0448123
(8, 7) 0.0426546
(8, 9) 0.82782
(9, 5) 0.295356
(9, 7) 0.281134
row jumpers: [
-36207072,32642,-39708721,32642,6390336,0,2012467744,32767,2012467968,32767,4203729,]
col ptrs: [
0,0,-39655605,32642,-36207072,32642,6390336,0,10,0,-39672717,32642,2012466352,32767,-32892691,32642,1,0,6390336,0,2012466344,32767,60002304,2059362829,]
elements: [
0.289516,0.304161,0.795779,0.334456,0.935264,0.585813,0.871237,0.811508,0.828558,0.0271863,6.92683e-310,6.92683e-310,1.061e-313,1.061e-313,6.36599e-314,4.24399e-314,6.36599e-314,6.92683e-310,4.24399e-314,1.2732e-313,2.122e-313,6.95324e-310,0.406537,0.0495716,0.370862,]
and similarly for multiplication of 2 1x1 matrices:
Result:
ViennaCL: compressed_matrix of size (1, 1) with 1 nonzeros:
(0, 0) 0.117699
row jumpers: [
-717571424,32767,]
col ptrs: [
6386240,]
elements: [
0.289516,6.9479e-310,]
________________________________
From: Andrew Palumbo <ap....@outlook.com>
Sent: Wednesday, July 20, 2016 5:40:31 PM
To: Karl Rupp; viennacl-devel
Subject: Re: [ViennaCL-devel] Copying Values out of a compressed_matrix
Oops, sorry about not cc'ing all.
I do not get correct data back for a (Random.nextDouble() populated) 1 x 1
Matrix.
A:
Row Pointer: [0, 1 ]
Col Pointer: [0 ]
element Pointer: [0.6465821602909256 ]
B:
Row Pointer: [0, 1 ]
Col Pointer: [0 ]
element Pointer: [0.9513577109193919 ]
C = A %*% B
Row Pointer: [469762248, 32632]
Col Pointer: [469762248 ]
element Pointer: [6.9245198744523E-310 ]
ouch.
It looks like I'm not copying the Buffers correctly at all. I'm may be using
the javacpp buffers incorrectly here, or I have possibly wrapped the
viennacl::backend::memory_handle class incorrectly, so I'm using a pointer to
the wrong memory from eg. viennacl::compressed_matrix::handle.
I mentioned before that the multiplication completed in on small <~300 x 300
matrices because if I try to multiply two larger sparse matrices, an err the
JVM crashes with a SIGSEGV.
Since this code is all wrapped with javacpp, I don't really have a small sample
that I can show you (not going to dump a whole bunch of code on you).
I'll keep trying to figure it out. Pretty sure the problem is on my end here
[?] I really mainly wanted to ask you if I was using the correct methods at
this point, or if there was anything very obviously that I was doing wrong.
Thanks a lot for your help!
Andy
________________________________
From: Karl Rupp <r...@iue.tuwien.ac.at>
Sent: Wednesday, July 20, 2016 5:00:36 PM
To: Andrew Palumbo; viennacl-devel
Subject: Re: [ViennaCL-devel] Copying Values out of a compressed_matrix
Hi,
please keep viennacl-devel in CC:
Just to clarify: Do you get incorrect values for a 1-by-1 matrix as
indicated in your sample data? In your previous email you mentioned that
results are fine for small matrices...
I'm afraid I can only guess at the source of the error with the
informations provided. Any chance that you can provide a standalone code
to reproduce the problem with reasonable effort?
Best regards,
Karli
On 07/20/2016 10:16 PM, Andrew Palumbo wrote:
> Thanks so much for your quick answer!
>
>
> I actually am sorry to say that I made a mistake when writing the last
> email, I copied the wrong signature from the VCL documentation, and then
> the mistake propagated through the rest of the e-mail.
>
>
> I am actually using viennacl::backend::memory_read().
>
>
> Eg, for the row_jumpers and column_idx I read use:
>
> @Name("backend::memory_read")
> public static native void memoryReadInt(@Const @ByRef MemHandle src_buffer,
> int bytes_to_read,
> int offset,
> IntPointer ptr,
> boolean async);
>
> and for the Values:
>
>
> @Name("backend::memory_read")
> public static native void memoryReadDouble(@Const @ByRef MemHandle src_buffer,
> int bytes_to_read,
> int offset,
> DoublePointer ptr,
> boolean async);
>
> And then call:
>
>
> memoryReadInt(row_ptr_handle, (m +1) *4,0, row_ptr,false)
> memoryReadInt(col_idx_handle, NNz *4,0,col_idx,false)
> memoryReadDouble(element_handle, NNz *8,0, values,false)
>
>
> and after convetring them to java.nio.Buffers, am getting results like:
>
>
> rowBuff.get(1): 0 colBuff(1): 402653448 valBuff(1): 6.91730177312166E-310
>
>
> Have also tried reading into BytePointers similarly with the same type
> of results. I know that the use of Javacpp obfuscates what the problem
> may be. But I believe the Memorry is properly allocated.
>
>
>
> Sorry for the mistake.
>
>
> Thanks,
>
>
> Andy
>
>
> ------------------------------------------------------------------------
> *From:* Karl Rupp <r...@iue.tuwien.ac.at>
> *Sent:* Wednesday, July 20, 2016 3:50:07 PM
> *To:* Andrew Palumbo; ViennaCL-devel@lists.sourceforge.net
> *Subject:* Re: [ViennaCL-devel] Copying Values out of a compressed_matrix
> Hi Andy,
>
> instead of viennacl::backend::memory_copy(), you want to use
> viennacl::backend::memory_read(), which directly transfers the data into
> your buffer(s).
>
> If you *know* that your handles are in host memory, you can even grab
> the values directly via
> viennacl::linalg::host_based::detail::extract_raw_pointer<T>();
> defined in viennacl/linalg/host_based/common.hpp, around line 40.
>
> Please let me know if you still get errors after using that.
>
> Best regards,
> Karli
>
>
>
>
> On 07/20/2016 09:05 PM, Andrew Palumbo wrote:
>> Hello,
>>
>>
>> I'm Having some difficulties with compressed_matrix multiplication.
>>
>>
>> Essentially I am copying three buffers, the CSR conversion of an Apache
>> Mahout SparseMatrix, into two compressed_matrices performing matrix
>> multiplication. I am doing this in scala and Java using javacpp.
>>
>>
>> For example, I have a 5 x 10 matrix of ~20% non-zero values which in CSR
>> format looks like this:
>>
>>
>> NNz: 12
>>
>> Row Pointer: [0, 1, 4, 6, 9, 12, ]
>>
>> Col Pointer: [9, 0, 8, 7, 2, 9, 0, 8, 9, 0, 3, 5, ]
>>
>> element Pointer: [0.4065367203992265, 0.04957158909682802,
>> 0.5205586068847993, 0.3708618354358446, 0.6963900565931678,
>> 0.8330915529787706, 0.32839112750638844, 0.7856168903297948,
>> 0.4265801782090245, 0.14733066454561583, 0.9501663495824946,
>> 0.9710498974366047, ]
>>
>> Multiplied by a similarly Sparse 10 x 5 compressed_matrix
>>
>> I use a CompressedMatrix wrapper which essentially wraps the
>>
>> viennacl:: compressed_matrix (vcl_size_t rows, vcl_size_t cols,
>> vcl_size_t nonzeros=0, viennacl::context ctx=viennacl::context())
>>
>> constructor as well as the
>>
>> compressed_matrix (matrix_expression< const compressed_matrix,
>> const compressed_matrix, op_prod > const &proxy).
>>
>> I have a helper function, /toVclCompressedMatrix/(..) which essentially
>> does the CSR conversion from a Mahout src matrix, calls the constructor
>> and uses viennacl::compressed_matrix::set(...) to set the buffers:
>>
>> val ompA =toVclCompressedMatrix(src = mxA, ompCtx)
>> val ompB =toVclCompressedMatrix(src = mxB, ompCtx)
>>
>>
>> and then create a new viennacl::compressed_matrix from the
>> viennacl::linalg::prod of the 2 matrices i.e.:
>>
>> val ompC =new CompressedMatrix(prod(ompA, ompB))
>>
>> The context in the above case is either the Host or OpenMP (I know that
>> there is some special casting of the row_jumpers and col_idxs that needs
>> to be done in the OpenCL version)
>>
>> The Matrix multiplication completes without error on small Matrices eg.
>> < 300 x 300
>> but seems to overwrite the resulting buffers on larger Matrices.
>>
>> My real problem, though is getting the memory back out of the
>> resulting`ompC` compresed_matrix so that i can write it back to a mahout
>> SparseMatrix.
>>
>> currently I am using:
>>
>> void viennacl::backend::memory_copy (mem_handle const & src_buffer,
>> mem_handle & dst_buffer,
>> vcl_size_t src_offset,
>> vcl_size_t dst_offset,
>> vcl_size_t bytes_to_copy
>> )
>>
>> on ompC.handel1,ompC.handel2 and ompC.handel source handels
>>
>> to copy into pre-allocated row_jumper, col_index and element buffers
>> (of size ompC.size1() + 1, ompC.nnz and ompC.nnz, respectivly).
>>
>> I am getting nonsensical values back that one would expect from memory
>> errors. eg:
>>
>> the Matrix geometry of the result: ompC.size1(), and omp.size2() are
>> correct and ompC.nnz is a reasonable value.
>>
>> It is possible that I have mis-allocated some of the memory on my side,
>> but I am pretty sure that most of the Buffers are allocated correctly
>> (usually JavaCPP does a pretty good job of this).
>>
>>
>> I guess, long story short, my question is am i using the correct method
>> of copying the memory out of a compressed_matrix? is there something
>> glaringly incorrect that i am doing here? Should I be using
>> viennacl::backend::memory_copy or is there a different method that i
>> should be using?
>>
>>
>> Thanks very much,
>>
>> Andy
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
>> patterns at an interface-level. Reveals which users, apps, and protocols are
>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>> planning
>> reports.http://sdm.link/zohodev2dev
>>
>>
>>
>> _______________________________________________
>> ViennaCL-devel mailing list
>> ViennaCL-devel@lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/viennacl-devel
>>
>
#include <iostream>
// #define VIENNACL_WITH_OPENMP 1
// ViennaCL includes
#include "viennacl/compressed_matrix.hpp"
#include "viennacl/linalg/prod.hpp"
#include "viennacl/backend/memory.hpp"
// compile line w/o OpenMP: g++ sparse.cpp -I/usr/include/viennacl/ -o sp.out
int main()
{
// trying to reproduce javacpp wrapper functionalliy as closely as possible
// so not using typedef, unsigned ints, etc, and defining templates as doubles
// creating buffers as int/double arrays and then setting pointers to them.
// (not 100% sure that this is how javacpp passes pointers but should be close.
//typedef double ScalarType;
// casting to ints from jni/javacpp
int m = 10;
int n = 10;
int s = 5;
int NNz_left = 12;
int NNz_right = 10;
// allocate buffers and set pointers (similarly to javacpp)
// using ints (not unsigned ints) here from jni/javacpp
int lhs_row_jumpers[m + 1] = {0, 0, 1, 2, 4, 5, 6, 7, 9, 11, 12};
int *lhs_row_ptr = lhs_row_jumpers;
// using ints (not unsigned ints) here from jni/javacpp
int lhs_col_idxs[NNz_left] = {4, 0, 2, 3, 2, 4, 0, 4, 3, 0, 3, 0};
int *lhs_col_ptr = lhs_col_idxs;
double lhs_values[NNz_left] = {0.4065367203992265, 0.04957158909682802, 0.3708618354358446,
0.5205586068847993, 0.6963900565931678, 0.8330915529787706, 0.32839112750638844,
0.4265801782090245, 0.7856168903297948, 0.14733066454561583, 0.9501663495824946,
0.9710498974366047};
double* lhs_values_ptr = lhs_values;
// using ints (not unsigned ints) here from jni/javacpp
int rhs_row_jumpers[s + 1] = {0, 2, 4, 6, 7, 10};
int *rhs_row_ptr = rhs_row_jumpers;
// using ints (not unsigned ints) here from jni/javacpp
int rhs_col_idxs[NNz_right] = {7, 5, 9, 5, 6, 3, 9, 2, 4, 3};
int *rhs_col_ptr = rhs_col_idxs;
double rhs_values[NNz_right] = {0.28951581746443, 0.30416122244949784, 0.7957787845969375,
0.3344555460382037, 0.9352637072084851, 0.5858126554486242, 0.8712367888745259,
0.8115081053090448, 0.8285577209925045, 0.027186259028062043};
double* rhs_values_ptr = rhs_values;
// using double values in Mahout setting template directly as double in javacpp
viennacl::compressed_matrix<double> lhs_compressed_matrix(m, s);
viennacl::compressed_matrix<double> rhs_compressed_matrix(s, n);
// set the ptrs
lhs_compressed_matrix.set(lhs_row_ptr, lhs_col_ptr, lhs_values_ptr, m, s, NNz_left);
rhs_compressed_matrix.set(rhs_row_ptr, rhs_col_ptr, rhs_values_ptr, s, n, NNz_right);
// perform multiplication and inside of a compressed_matrix constructor
viennacl::compressed_matrix<double> matrix_product(viennacl::linalg::prod(lhs_compressed_matrix,
rhs_compressed_matrix));
// print out matrix
std::cout << "ViennaCL: " << matrix_product << std::endl;
int product_size_row = matrix_product.size1();
int product_size_column = matrix_product.size2();
int product_NNz = matrix_product.nnz();
//const long unsigned int product_NNz = matrix_product.nnz();
// Allocate Buffers and specifiy pointers to pass to read()
// (similarly to as is done in javacpp)
int product_row_jumpers[ product_size_row + 1 ];
int *product_row_ptr = product_row_jumpers;
int product_col_idxs[product_NNz];
int *product_col_ptr = product_col_idxs;
double product_values[product_NNz];
double* product_values_ptr = product_values;
// rows, column and value handles, respectivly
viennacl::backend::mem_handle handle1 = matrix_product.handle1();
viennacl::backend::mem_handle handle2 = matrix_product.handle2();
viennacl::backend::mem_handle handle = matrix_product.handle();
// read data back into our product buffers
viennacl::backend::memory_read(handle1, product_size_row * 4, 0, product_row_ptr, false);
viennacl::backend::memory_read(handle2, product_NNz * 4, 0, product_col_ptr, false);
viennacl::backend::memory_read(handle, product_NNz * 8, 0, product_values_ptr, false);
int i = 0;
std::cout << "row jumpers: [" << std::endl;
for (i = 0 ; i < product_size_row + 1; i++) {
std::cout << product_row_jumpers[i] << ",";
}
std::cout << "]" << std::endl;
std::cout << "col ptrs: [" << std::endl;
for (i = 0 ; i < product_NNz; i++) {
std::cout << product_col_idxs[i] << ",";
}
std::cout << "]" << std::endl;
std::cout << "elements: [" << std::endl;
for (i = 0 ; i < product_NNz+ 1; i++) {
std::cout << product_values[i] << ",";
}
std::cout << "]" << std::endl;
return EXIT_SUCCESS;
}
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
_______________________________________________
ViennaCL-devel mailing list
ViennaCL-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/viennacl-devel
