[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

[srowen]

Github user srowen commented on the pull request:

    https://github.com/apache/incubator-spark/pull/575#issuecomment-34711528
  
    I see the other discussion -- https://github.com/mesos/spark/pull/736 ? I 
didn't see the benchmark but maybe missed it.
    
    I think there was an impression there that `mahout-math` is being actively 
improved, and I don't think there have or appear to be any substantial changes 
or work on older open issues/bugs: 
https://github.com/apache/mahout/commits/trunk/math 
    
    No obvious right answers indeed. Breeze sounds like the best fit but I have 
no familiarity with how much work it would be to speed up whatever needs to be 
sped up.
    
    Commons Math 3.x will be the mainline release for I presume ~6 more months, 
with 3.3 coming soon and still including this code. If the idea is that this 
might be a temporary step, I would suggest it as the least-bad temporary 
solution for sparse vectors. It is at least somewhat actively supported. And 
pretty well known in Java land.
    
    Either way, I think it's probably bad to use DenseVector *and* JBlas both.
    
    Agree with trying to shield the user and even implementation details from 
the choice as much as possible. It's hard to keep abstractions up and get good 
performance in some cases. 
    
    Take that strong sentiment for what it's worth, having worked with Mahout, 
JBlas, Commons Math extensively.