Re: large, dense matrix multiplication

Hi Burak, That’s interesting. I’ll try and give it a go. Eilidh On 14 Nov 2015, at 04:19, Burak Yavuz wrote: > Hi, > > The BlockMatrix multiplication should be much more efficient on the current > master (and will be available with Spark 1.6). Could you please give that a

Re: large, dense matrix multiplication

We have done this by blocking but without using BlockMatrix. We used our own blocking mechanism because BlockMatrix didn't exist in Spark 1.2. What is the size of your block? How much memory are you giving to the executors? I assume you are running on YARN, if so you would want to make sure your

Re: large, dense matrix multiplication

Hi Sab, Thanks for your response. We’re thinking of trying a bigger cluster, because we just started with 2 nodes. What we really want to know is whether the code will scale up with larger matrices and more nodes. I’d be interested to hear how large a matrix multiplication you managed to do?

Re: large, dense matrix multiplication

Hi, The BlockMatrix multiplication should be much more efficient on the current master (and will be available with Spark 1.6). Could you please give that a try if you have the chance? Thanks, Burak On Fri, Nov 13, 2015 at 10:11 AM, Sabarish Sasidharan < sabarish.sasidha...@manthan.com> wrote:

large, dense matrix multiplication

Hi, I’m trying to multiply a large squarish matrix with its transpose. Eventually I’d like to work with matrices of size 200,000 by 500,000, but I’ve started off first with 100 by 100 which was fine, and then with 10,000 by 10,000 which failed with an out of memory exception. I used MLlib and