Hi Burak,
That’s interesting. I’ll try and give it a go.
Eilidh
On 14 Nov 2015, at 04:19, Burak Yavuz <brk...@gmail.com> wrote:
> Hi,
>
> The BlockMatrix multiplication should be much more efficient on the current
> master (and will be available with Spark 1.6). Could you please give that a
> try if you have the chance?
>
> Thanks,
> Burak
>
> On Fri, Nov 13, 2015 at 10:11 AM, Sabarish Sasidharan
> <sabarish.sasidha...@manthan.com> wrote:
> Hi Eilidh
>
> Because you are multiplying with the transpose you don't have to necessarily
> build the right side of the matrix. I hope you see that. You can broadcast
> blocks of the indexed row matrix to itself and achieve the multiplication.
>
> But for similarity computation you might want to use some approach like
> locality sensitive hashing first to identify a bunch of similar customers and
> then apply cosine similarity on that narrowed down list. That would scale
> much better than matrix multiplication. You could try the following options
> for the same.
>
> https://github.com/soundcloud/cosine-lsh-join-spark
> http://spark-packages.org/package/tdebatty/spark-knn-graphs
> https://github.com/marufaytekin/lsh-spark
>
> Regards
> Sab
>
> Hi Sab,
>
> Thanks for your response. We’re thinking of trying a bigger cluster, because
> we just started with 2 nodes. What we really want to know is whether the code
> will scale up with larger matrices and more nodes. I’d be interested to hear
> how large a matrix multiplication you managed to do?
>
> Is there an alternative you’d recommend for calculating similarity over a
> large dataset?
>
> Thanks,
> Eilidh
>
> On 13 Nov 2015, at 09:55, Sabarish Sasidharan
> <sabarish.sasidha...@manthan.com> wrote:
>
>> We have done this by blocking but without using BlockMatrix. We used our own
>> blocking mechanism because BlockMatrix didn't exist in Spark 1.2. What is
>> the size of your block? How much memory are you giving to the executors? I
>> assume you are running on YARN, if so you would want to make sure your yarn
>> executor memory overhead is set to a higher value than default.
>>
>> Just curious, could you also explain why you need matrix multiplication with
>> transpose? Smells like similarity computation.
>>
>> Regards
>> Sab
>>
>> On Thu, Nov 12, 2015 at 7:27 PM, Eilidh Troup <e.tr...@epcc.ed.ac.uk> wrote:
>> Hi,
>>
>> I’m trying to multiply a large squarish matrix with its transpose.
>> Eventually I’d like to work with matrices of size 200,000 by 500,000, but
>> I’ve started off first with 100 by 100 which was fine, and then with 10,000
>> by 10,000 which failed with an out of memory exception.
>>
>> I used MLlib and BlockMatrix and tried various block sizes, and also tried
>> switching disk serialisation on.
>>
>> We are running on a small cluster, using a CSV file in HDFS as the input
>> data.
>>
>> Would anyone with experience of multiplying large, dense matrices in spark
>> be able to comment on what to try to make this work?
>>
>> Thanks,
>> Eilidh
>>
>>
>> --
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>>
>>
>> --
>>
>> Architect - Big Data
>> Ph: +91 99805 99458
>>
>> Manthan Systems | Company of the year - Analytics (2014 Frost and Sullivan
>> India ICT)
>> +++
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
