GitHub user larryxiao opened a pull request:
https://github.com/apache/spark/pull/2446
[SPARK-1987] EdgePartitionBuilder: More memory-efficient graph construction
https://issues.apache.org/jira/browse/SPARK-1987
To save overhead of Edge objects, separate an array of Edge objects into
three arrays of srdId, dstId and data, for later EdgePartition(srcIdsTrim,
dstIdsTrim, dataTrim, index, vertices).
To sort arrays directly, I use
[ParallelSorter](http://cglib.sourceforge.net/apidocs/net/sf/cglib/ParallelSorter.html)
from mockito.cglib
Can't compile at the moment and I don't really know to solve.
```
[info] Compiling 8 Scala sources to
/home/xd/Developer/spark/graphx/target/scala-2.10/classes...
[error]
/home/xd/Developer/spark/graphx/src/main/scala/org/apache/spark/graphx/impl/EdgePartitionBuilder.scala:49:
type mismatch;
[error] found : Array[Array[_ >: ED with Long]]
[error] required: Array[Object]
[error] Note: Array[_ >: ED with Long] <: Object, but class Array is
invariant in type T.
[error] You may wish to investigate a wildcard type such as `_ <: Object`.
(SLS 3.2.10)
[error] val sorter = ParallelSorter.create(edgeArray)
[error] ^
[error] one error found
[error] (graphx/compile:compile) Compilation failed
[error] Total time: 5 s, completed Sep 18, 2014 9:01:14 PM
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/larryxiao/spark 1987
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2446.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2446
----
commit eabc9c26948f12cc2874a553bf668a38b08b3301
Author: Larry Xiao <xia...@sjtu.edu.cn>
Date: 2014-08-25T07:29:56Z
[SPARK-1987] EdgePartitionBuilder: More memory-efficient graph construction
use ParallelSorter from mockito.cglib
commit c1d7e1171f30da7b26cdc80f0345ff2a69ec5649
Author: Larry Xiao <xia...@sjtu.edu.cn>
Date: 2014-09-18T13:10:53Z
add dependency in build.sbt
can't compile yet
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
