[ 
https://issues.apache.org/jira/browse/GIRAPH-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14643247#comment-14643247
 ] 

Hudson commented on GIRAPH-1022:
--------------------------------

FAILURE: Integrated in Giraph-trunk-Commit #1560 (See 
[https://builds.apache.org/job/Giraph-trunk-Commit/1560/])
[GIRAPH-1022] Adaptive out-of-core mechanism for input superstep and graph 
(aching: 
http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=03ade425dd5a65d3a713d5e7d85aa7605956fbd2)
* giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java
* giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java
* giraph-core/src/main/java/org/apache/giraph/ooc/MemoryEstimator.java
* giraph-examples/src/test/java/org/apache/giraph/TestOutOfCore.java
* 
giraph-core/src/main/java/org/apache/giraph/partition/DiskBackedPartitionStore.java
* findbugs-exclude.xml
* giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java
* 
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerVerticesRequest.java
* giraph-core/src/main/java/org/apache/giraph/ooc/CheckMemoryCallable.java
* giraph-core/src/test/java/org/apache/giraph/partition/TestPartitionStores.java
* 
giraph-core/src/test/java/org/apache/giraph/comm/messages/TestIntFloatPrimitiveMessageStores.java
* giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java
* giraph-core/src/main/java/org/apache/giraph/edge/primitives/LongEdgeStore.java
* giraph-core/src/main/java/org/apache/giraph/ooc/package-info.java
* giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java
* 
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerEdgesRequest.java
* giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
* giraph-core/src/main/java/org/apache/giraph/ooc/JVMMemoryEstimator.java
* giraph-core/src/main/java/org/apache/giraph/ooc/AdaptiveOutOfCoreEngine.java
* giraph-core/src/main/java/org/apache/giraph/edge/AbstractEdgeStore.java
* giraph-core/src/main/java/org/apache/giraph/edge/primitives/IntEdgeStore.java
* giraph-core/src/main/java/org/apache/giraph/partition/PartitionStore.java
* giraph-core/src/main/java/org/apache/giraph/ooc/OutOfCoreEngine.java
* giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java
* 
giraph-core/src/test/java/org/apache/giraph/comm/messages/TestLongDoublePrimitiveMessageStores.java
* giraph-core/src/main/java/org/apache/giraph/edge/SimpleEdgeStore.java
* CHANGELOG
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java
* 
giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionStore.java
* 
giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/long_id/LongAbstractListMessageStore.java
* giraph-core/src/main/java/org/apache/giraph/ooc/DiskBackedPartitionStore.java
* 
giraph-core/src/main/java/org/apache/giraph/ooc/OutOfCoreProcessorCallable.java


> Out-of-core mechanism for input superstep and graph data
> --------------------------------------------------------
>
>                 Key: GIRAPH-1022
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-1022
>             Project: Giraph
>          Issue Type: New Feature
>          Components: graph
>            Reporter: Hassan Eslami
>            Assignee: Hassan Eslami
>
> Currently the out-of-core mechanism has several issues:
> - It does not work properly in input superstep
> - It has several bugs (correctness, and performance) for graph data structure
> - It is not considering specialized message classes for out-of-core messages.
> This diff addresses some of these problems, and provides a cohesive mechanism 
> for input superstep and graph data. With this way of doing out-of-core, 
> efficiently handling out-of-core messages is straightforward (will be 
> provided in a separate diff).
> This diff provides an adaptive mechanism to remove users out of the loop for 
> using out-of-core. If the graph can be fit in memory, all the computation 
> remains in memory. If memory is very limited and GC is killing the 
> performance, or the graph cannot fit in memory, out-of-core mechanism kicks 
> in and helps improve performance in many cases, and avoids application 
> failure due to OutOfMemory exception or it's related exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to