[
https://issues.apache.org/jira/browse/GIRAPH-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hassan Eslami updated GIRAPH-1022:
----------------------------------
Description:
Currently the out-of-core mechanism has several issues:
- It does not work properly in input superstep
- It has several bugs (correctness, and performance) for graph data structure
- It is not considering specialized message classes for out-of-core messages.
This diff addresses some of these problems, and provides a cohesive mechanism
for input superstep and graph data. With this way of doing out-of-core,
efficiently handling out-of-core messages is straightforward (will be provided
in a separate diff).
This diff provides an adaptive mechanism to remove users out of the loop for
using out-of-core. If the graph can be fit in memory, all the computation
remains in memory. If memory is very limited and GC is killing the performance,
or the graph cannot fit in memory, out-of-core mechanism kicks in and helps
improve performance in many cases, and avoids application failure due to
OutOfMemory exception or it's related exceptions.
was:
Currently the out-of-core mechanism has several issues:
- It does not work properly in input superstep
- It has several bugs (correctness, and performance) for graph data structure
- It is not considering specialized message classes for out-of-core messages.
This diff addresses some of these problems, and provides a cohesive mechanism
for input superstep and graph data. With this way of doing out-of-core,
efficiently handling out-of-core messages is straightforward (will be provided
in a separate diff).
This diff provides an adaptive mechanism to remove users out of the loop for
using out-of-core. If the graph can be fit in memory, all the computation
remains in memory. If memory is very limited and GC is killing performance, or
the graph cannot fit in memory, out-of-core mechanism kicks in and helps
improve performance in many cases, and avoids application failure due to
OutOfMemory exception or it's related exceptions.
> Out-of-core mechanism for input superstep and graph data
> --------------------------------------------------------
>
> Key: GIRAPH-1022
> URL: https://issues.apache.org/jira/browse/GIRAPH-1022
> Project: Giraph
> Issue Type: New Feature
> Components: graph
> Reporter: Hassan Eslami
>
> Currently the out-of-core mechanism has several issues:
> - It does not work properly in input superstep
> - It has several bugs (correctness, and performance) for graph data structure
> - It is not considering specialized message classes for out-of-core messages.
> This diff addresses some of these problems, and provides a cohesive mechanism
> for input superstep and graph data. With this way of doing out-of-core,
> efficiently handling out-of-core messages is straightforward (will be
> provided in a separate diff).
> This diff provides an adaptive mechanism to remove users out of the loop for
> using out-of-core. If the graph can be fit in memory, all the computation
> remains in memory. If memory is very limited and GC is killing the
> performance, or the graph cannot fit in memory, out-of-core mechanism kicks
> in and helps improve performance in many cases, and avoids application
> failure due to OutOfMemory exception or it's related exceptions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
