[ https://issues.apache.org/jira/browse/GIRAPH-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290341#comment-15290341 ]
Hassan Eslami commented on GIRAPH-1066: --------------------------------------- https://reviews.facebook.net/D55479 > Functional adaptive out-of-core mechanism > ----------------------------------------- > > Key: GIRAPH-1066 > URL: https://issues.apache.org/jira/browse/GIRAPH-1066 > Project: Giraph > Issue Type: New Feature > Components: bsp, graph > Reporter: Hassan Eslami > Assignee: Hassan Eslami > > In this JIRA we propose the following contributions to the out-of-core > mechanism: > • A simpler API is provided to try various out-of-core policies using the > basic infrastructure proposed in GIRAPH-1048. This new API helps developers > of out-of-core policies to only focus on the out-of-core logic, rather than > the complications in multi-threading, disk interactions, etc. The policy > logic is abstracted out as much as possible to make it as simple as possible > to develop and try other out-of-core policies. > • Two adaptive out-of-core policies are implemented using the proposed API. > One is based on few recent GC behaviors, and the other is based on some > user-defined thresholds to control the memory pressure. With the adaptive > out-of-core policies, the job automatically uses secondary storage devices in > case the data cannot fit into memory. Also, if at some point in the > computation the memory pressure goes down, the spilled data to secondary > storage will be automatically loaded to memory again. > • The out-of-core infrastructure is integrated with message flow control > proposed in GIRAPH-1027. Using credit-based flow control, an out-of-core > policy can predict the amount of memory usage by messages in a near future, > hence the policy can have a fine control over messages and their memory > footprint. > • A new feature, called data generation tethering, is also added. This > feature let the out-of-core policy to decide how many threads (input/compute) > should be active at each moment, indirectly controlling the rate of data > generation, and in turn, controlling the memory footprint of graph data. > With this JIRA landed, we will have a full-functional out-of-core > infrastructure preventing any reasonable job to fail due to OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)