Re: Problems with running page rank using OutOfCore setting

> > > > So, does this mean that there is no other solution but to increase the > physical memory? > > > Cheers, > > Darshan > > > On 16 Jul 2017, at 23:04, Hassan Eslami <hsn.esl...@gmail.com sn.esl...@gmail.com>> wrote: > > Hi, > > giraph.useOutOfC

Re: Problems with running page rank using OutOfCore setting

Hi, giraph.useOutOfCoreMessages is no longer in use. The main problem here is that you are using default flow control mechanism (NoOpFlowControl), that causes a lot of outstanding/received messages. As a consequence, you fill up the memory so fast, and the job would fail for various reasons.

[jira] [Created] (GIRAPH-1150) Fixing MemoryEstimatorOracle

Hassan Eslami created GIRAPH-1150: - Summary: Fixing MemoryEstimatorOracle Key: GIRAPH-1150 URL: https://issues.apache.org/jira/browse/GIRAPH-1150 Project: Giraph Issue Type: Bug

[jira] [Resolved] (GIRAPH-1125) Add memory estimation mechanism to out-of-core

[ https://issues.apache.org/jira/browse/GIRAPH-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami resolved GIRAPH-1125. --- Resolution: Fixed > Add memory estimation mechanism to out-of-c

[jira] [Commented] (GIRAPH-1144) Out-of-core control flow sends messages after Netty shuts down

[ https://issues.apache.org/jira/browse/GIRAPH-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979192#comment-15979192 ] Hassan Eslami commented on GIRAPH-1144: --- This is the "resume signal" sent after the N

[jira] [Created] (GIRAPH-1137) Remove channel probing from Netty worker thread for credit-based flow-control

Hassan Eslami created GIRAPH-1137: - Summary: Remove channel probing from Netty worker thread for credit-based flow-control Key: GIRAPH-1137 URL: https://issues.apache.org/jira/browse/GIRAPH-1137

[jira] [Created] (GIRAPH-1125) Add memory estimation mechanism to out-of-core

Hassan Eslami created GIRAPH-1125: - Summary: Add memory estimation mechanism to out-of-core Key: GIRAPH-1125 URL: https://issues.apache.org/jira/browse/GIRAPH-1125 Project: Giraph Issue Type

Re: [VOTE] Apache Giraph 1.2.0 RC1

iple new features. Among the most important are: > - New out of core implementation (Hassan Eslami & Dionysios Logothetis) > - FacebookConfiguration, this is a set of parameters used internally > in Facebook > to run Giraph jobs. (Maja Kabiljo) > - Multiple improvements

[jira] [Updated] (GIRAPH-1088) We should be able to enable out-of-core mechanism with minimal flags/options

[ https://issues.apache.org/jira/browse/GIRAPH-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami updated GIRAPH-1088: -- Fix Version/s: (was: 1.2.0) > We should be able to enable out-of-core mechan

[jira] [Resolved] (GIRAPH-1089) In fixed out-of-core policy the definition of in-memory partition is not clear

[ https://issues.apache.org/jira/browse/GIRAPH-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami resolved GIRAPH-1089. --- Resolution: Fixed Thanks Maja for reviewing and committing the diff! > In fixed out-of-c

[jira] [Updated] (GIRAPH-1074) Update website description for out-of-core

[ https://issues.apache.org/jira/browse/GIRAPH-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami updated GIRAPH-1074: -- Fix Version/s: (was: 1.2.0) > Update website description for out-of-c

[jira] [Created] (GIRAPH-1089) In fixed out-of-core policy the definition of in-memory partition is not clear

Hassan Eslami created GIRAPH-1089: - Summary: In fixed out-of-core policy the definition of in-memory partition is not clear Key: GIRAPH-1089 URL: https://issues.apache.org/jira/browse/GIRAPH-1089

[jira] [Commented] (GIRAPH-1088) We should be able to enable out-of-core mechanism with minimal flags/options

[ https://issues.apache.org/jira/browse/GIRAPH-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373318#comment-15373318 ] Hassan Eslami commented on GIRAPH-1088: --- https://reviews.facebook.net/D60609 > We should be a

[jira] [Created] (GIRAPH-1088) We should be able to enable out-of-core mechanism with minimal flags/options

Hassan Eslami created GIRAPH-1088: - Summary: We should be able to enable out-of-core mechanism with minimal flags/options Key: GIRAPH-1088 URL: https://issues.apache.org/jira/browse/GIRAPH-1088

[jira] [Assigned] (GIRAPH-1088) We should be able to enable out-of-core mechanism with minimal flags/options

[ https://issues.apache.org/jira/browse/GIRAPH-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami reassigned GIRAPH-1088: - Assignee: Hassan Eslami > We should be able to enable out-of-core mechan

[jira] [Commented] (GIRAPH-1074) Update website description for out-of-core

[ https://issues.apache.org/jira/browse/GIRAPH-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373315#comment-15373315 ] Hassan Eslami commented on GIRAPH-1074: --- https://reviews.facebook.net/D60639 > Update webs

[jira] [Created] (GIRAPH-1074) Update website description for out-of-core

Hassan Eslami created GIRAPH-1074: - Summary: Update website description for out-of-core Key: GIRAPH-1074 URL: https://issues.apache.org/jira/browse/GIRAPH-1074 Project: Giraph Issue Type

[jira] [Commented] (GIRAPH-1073) Decouple out-of-core persistence infrastructure from out-of-core computation

[ https://issues.apache.org/jira/browse/GIRAPH-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332260#comment-15332260 ] Hassan Eslami commented on GIRAPH-1073: --- https://reviews.facebook.net/D59691 > Decouple

[jira] [Created] (GIRAPH-1073) Decouple out-of-core persistence infrastructure from out-of-core computation

Hassan Eslami created GIRAPH-1073: - Summary: Decouple out-of-core persistence infrastructure from out-of-core computation Key: GIRAPH-1073 URL: https://issues.apache.org/jira/browse/GIRAPH-1073

[jira] [Commented] (GIRAPH-1071) Make out-of-core capable of handling massive mutations

[ https://issues.apache.org/jira/browse/GIRAPH-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332226#comment-15332226 ] Hassan Eslami commented on GIRAPH-1071: --- https://reviews.facebook.net/D58509 > Make out-of-c

[jira] [Resolved] (GIRAPH-1067) Out-of-core message mechanism needs clean up

[ https://issues.apache.org/jira/browse/GIRAPH-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami resolved GIRAPH-1067. --- Resolution: Fixed Thanks to Maja Kabiljo for reviewing and committing the code. >

[jira] [Commented] (GIRAPH-1067) Out-of-core message mechanism needs clean up

[ https://issues.apache.org/jira/browse/GIRAPH-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308816#comment-15308816 ] Hassan Eslami commented on GIRAPH-1067: --- https://reviews.facebook.net/D58701 > Out-of-core mess

[jira] [Resolved] (GIRAPH-1026) New Out-of-core mechanism does not work

[ https://issues.apache.org/jira/browse/GIRAPH-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami resolved GIRAPH-1026. --- Resolution: Fixed This issue has been resolved in the new out-of-core design in GIRAPH-1066

[jira] [Commented] (GIRAPH-462) Multithreading breaks out-of-core graph

[ https://issues.apache.org/jira/browse/GIRAPH-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294416#comment-15294416 ] Hassan Eslami commented on GIRAPH-462: -- The new out-of-core feature has been pushed to the trunk

[jira] [Resolved] (GIRAPH-1066) Functional adaptive out-of-core mechanism

[ https://issues.apache.org/jira/browse/GIRAPH-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami resolved GIRAPH-1066. --- Resolution: Fixed Thanks Dionysios Logothetis for reviewing the code and Sergey Edunov

[jira] [Commented] (GIRAPH-1066) Functional adaptive out-of-core mechanism

[ https://issues.apache.org/jira/browse/GIRAPH-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290341#comment-15290341 ] Hassan Eslami commented on GIRAPH-1066: --- https://reviews.facebook.net/D55479 > Functional adapt

[jira] [Resolved] (GIRAPH-1057) Decoupling NettyClient from control flow policy

[ https://issues.apache.org/jira/browse/GIRAPH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami resolved GIRAPH-1057. --- Resolution: Fixed Thanks Dionysis Logothetis for reviewing the code and Sergey Edunov

[jira] [Created] (GIRAPH-1057) Decoupling NettyClient from control flow policy

Hassan Eslami created GIRAPH-1057: - Summary: Decoupling NettyClient from control flow policy Key: GIRAPH-1057 URL: https://issues.apache.org/jira/browse/GIRAPH-1057 Project: Giraph Issue

[jira] [Commented] (GIRAPH-1057) Decoupling NettyClient from control flow policy

[ https://issues.apache.org/jira/browse/GIRAPH-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258591#comment-15258591 ] Hassan Eslami commented on GIRAPH-1057: --- https://reviews.facebook.net/D56367 > Decoupl

[jira] [Resolved] (GIRAPH-1025) Flow-control mechanism improving out-of-core feature

[ https://issues.apache.org/jira/browse/GIRAPH-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami resolved GIRAPH-1025. --- Resolution: Fixed Addressed in GIRAPH-1027 > Flow-control mechanism improving out-of-c

[jira] [Commented] (GIRAPH-1056) Setting auto-read in Netty to false

[ https://issues.apache.org/jira/browse/GIRAPH-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258565#comment-15258565 ] Hassan Eslami commented on GIRAPH-1056: --- Thanks Maja for reviewing and committing the diff

[jira] [Commented] (GIRAPH-1056) Setting auto-read in Netty to false

[ https://issues.apache.org/jira/browse/GIRAPH-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258564#comment-15258564 ] Hassan Eslami commented on GIRAPH-1056: --- https://reviews.facebook.net/D57213 > Setting auto-r

[jira] [Created] (GIRAPH-1056) Setting auto-read in Netty to false

Hassan Eslami created GIRAPH-1056: - Summary: Setting auto-read in Netty to false Key: GIRAPH-1056 URL: https://issues.apache.org/jira/browse/GIRAPH-1056 Project: Giraph Issue Type

[jira] [Assigned] (GIRAPH-1056) Setting auto-read in Netty to false

[ https://issues.apache.org/jira/browse/GIRAPH-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami reassigned GIRAPH-1056: - Assignee: Hassan Eslami > Setting auto-read in Netty to fa

[jira] [Commented] (GIRAPH-1027) Improve flow-control on sender side

[ https://issues.apache.org/jira/browse/GIRAPH-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225292#comment-15225292 ] Hassan Eslami commented on GIRAPH-1027: --- Thanks Dionysios and Maja for reviewing and committing

[jira] [Commented] (GIRAPH-1048) Redesign of out-of-core mechanism (first patch -- out-of-core mechanism keeping fixed number of partitions in memory)

[ https://issues.apache.org/jira/browse/GIRAPH-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194615#comment-15194615 ] Hassan Eslami commented on GIRAPH-1048: --- https://reviews.facebook.net/D54549 > Redesign of

[jira] [Created] (GIRAPH-1048) Redesign of out-of-core mechanism (first patch -- out-of-core mechanism keeping fixed number of partitions in memory)

Hassan Eslami created GIRAPH-1048: - Summary: Redesign of out-of-core mechanism (first patch -- out-of-core mechanism keeping fixed number of partitions in memory) Key: GIRAPH-1048 URL: https://issues.apache.org

[jira] [Commented] (GIRAPH-1027) Improve flow-control on sender side

[ https://issues.apache.org/jira/browse/GIRAPH-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662203#comment-14662203 ] Hassan Eslami commented on GIRAPH-1027: --- Code review discussion

[jira] [Commented] (GIRAPH-1026) New Out-of-core mechanism does not work

[ https://issues.apache.org/jira/browse/GIRAPH-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662220#comment-14662220 ] Hassan Eslami commented on GIRAPH-1026: --- Thanks Vitaly for describing the issues. I

[jira] [Created] (GIRAPH-1025) Flow-control mechanism improving out-of-core feature

Hassan Eslami created GIRAPH-1025: - Summary: Flow-control mechanism improving out-of-core feature Key: GIRAPH-1025 URL: https://issues.apache.org/jira/browse/GIRAPH-1025 Project: Giraph

[jira] [Commented] (GIRAPH-1025) Flow-control mechanism improving out-of-core feature

[ https://issues.apache.org/jira/browse/GIRAPH-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650608#comment-14650608 ] Hassan Eslami commented on GIRAPH-1025: --- Code review discussion: https

[jira] [Updated] (GIRAPH-1022) Out-of-core mechanism for input superstep and graph data

[ https://issues.apache.org/jira/browse/GIRAPH-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami updated GIRAPH-1022: -- Description: Currently the out-of-core mechanism has several issues: - It does not work

[jira] [Assigned] (GIRAPH-1022) Out-of-core mechanism for input superstep and graph data

[ https://issues.apache.org/jira/browse/GIRAPH-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hassan Eslami reassigned GIRAPH-1022: - Assignee: Hassan Eslami Out-of-core mechanism for input superstep and graph data

[jira] [Created] (GIRAPH-1020) TaskInfo equality condition

Hassan Eslami created GIRAPH-1020: - Summary: TaskInfo equality condition Key: GIRAPH-1020 URL: https://issues.apache.org/jira/browse/GIRAPH-1020 Project: Giraph Issue Type: Bug

[jira] [Created] (GIRAPH-1019) Optimizing and debugging vertex mutation

Hassan Eslami created GIRAPH-1019: - Summary: Optimizing and debugging vertex mutation Key: GIRAPH-1019 URL: https://issues.apache.org/jira/browse/GIRAPH-1019 Project: Giraph Issue Type

[jira] [Created] (GIRAPH-1009) Spammy 'lost reservation' messages from ZooKeeper in workers' log at the end of the computation.

Hassan Eslami created GIRAPH-1009: - Summary: Spammy 'lost reservation' messages from ZooKeeper in workers' log at the end of the computation. Key: GIRAPH-1009 URL: https://issues.apache.org/jira/browse/GIRAPH