Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Maja Kabiljo
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5987/#review10614 --- Alessandro, can you please try the same test with a lot of vertices (

[jira] [Updated] (GIRAPH-293) Should aggregators be checkpointed?

2012-08-22 Thread Maja Kabiljo (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-293: Attachment: GIRAPH-293.patch Forgot to mark as 'Patch Available' last time. I'm just rebasing it n

[jira] [Commented] (GIRAPH-293) Should aggregators be checkpointed?

2012-08-22 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439414#comment-13439414 ] Alessandro Presta commented on GIRAPH-293: -- If there's no merge conflict, you don

[jira] [Updated] (GIRAPH-249) Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Presta updated GIRAPH-249: - Attachment: GIRAPH-249.patch Updated to match review https://reviews.apache.org/r/5987/di

[jira] [Updated] (GIRAPH-249) Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Presta updated GIRAPH-249: - Attachment: GIRAPH-249.patch https://reviews.apache.org/r/5987/diff/11/

[jira] [Commented] (GIRAPH-293) Should aggregators be checkpointed?

2012-08-22 Thread Maja Kabiljo (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439475#comment-13439475 ] Maja Kabiljo commented on GIRAPH-293: - https://reviews.apache.org/r/6731/diff/#index_h

[jira] [Commented] (GIRAPH-211) Add secure authentication to Netty IPC

2012-08-22 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439481#comment-13439481 ] Alessandro Presta commented on GIRAPH-211: -- Any update on this? I don't know the

[jira] [Created] (GIRAPH-310) Simplify vertex initialization

2012-08-22 Thread Alessandro Presta (JIRA)
Alessandro Presta created GIRAPH-310: Summary: Simplify vertex initialization Key: GIRAPH-310 URL: https://issues.apache.org/jira/browse/GIRAPH-310 Project: Giraph Issue Type: Improvement

[jira] [Created] (GIRAPH-311) Master halting in superstep 0 is ignored by workers

2012-08-22 Thread Maja Kabiljo (JIRA)
Maja Kabiljo created GIRAPH-311: --- Summary: Master halting in superstep 0 is ignored by workers Key: GIRAPH-311 URL: https://issues.apache.org/jira/browse/GIRAPH-311 Project: Giraph Issue Type:

Re: Giraph Job "Task attempt_* failed to report status" Problem

2012-08-22 Thread Vishal Patel
After several supersteps, sometimes a worker thread dies (say it ran out of memory). Zookeeper waits for ~5 mins (600 seconds) and then decides that the worker is not responsive and fails the entire job. At this point if you have a checkpoint saved it will resume from there otherwise you have to st

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta
> On Aug. 22, 2012, 8:55 a.m., Maja Kabiljo wrote: > > Alessandro, can you please try the same test with a lot of vertices (you > > only had 10 vertices here)? If vertices are visited in random order in the > > prepareSuperstep, I think you should have much more disk operations. > > > > > Also

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5987/ --- (Updated Aug. 22, 2012, 10:34 a.m.) Review request for giraph and Avery Ching.

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Maja Kabiljo
> On Aug. 22, 2012, 8:55 a.m., Maja Kabiljo wrote: > > Alessandro, can you please try the same test with a lot of vertices (you > > only had 10 vertices here)? If vertices are visited in random order in the > > prepareSuperstep, I think you should have much more disk operations. > > > > > Also

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta
> On Aug. 22, 2012, 8:55 a.m., Maja Kabiljo wrote: > > Alessandro, can you please try the same test with a lot of vertices (you > > only had 10 vertices here)? If vertices are visited in random order in the > > prepareSuperstep, I think you should have much more disk operations. > > > > > Also

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Maja Kabiljo
> On Aug. 22, 2012, 8:55 a.m., Maja Kabiljo wrote: > > Alessandro, can you please try the same test with a lot of vertices (you > > only had 10 vertices here)? If vertices are visited in random order in the > > prepareSuperstep, I think you should have much more disk operations. > > > > > Also

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta
> On Aug. 22, 2012, 8:55 a.m., Maja Kabiljo wrote: > > Alessandro, can you please try the same test with a lot of vertices (you > > only had 10 vertices here)? If vertices are visited in random order in the > > prepareSuperstep, I think you should have much more disk operations. > > > > > Also

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5987/ --- (Updated Aug. 22, 2012, 12:36 p.m.) Review request for giraph and Avery Ching.

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Alessandro Presta
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5987/ --- (Updated Aug. 22, 2012, 1 p.m.) Review request for giraph and Avery Ching. Cha

Review Request: Fixing checkpointing for aggregators

2012-08-22 Thread Maja Kabiljo
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6731/ --- Review request for giraph. Description --- Making aggregators work correctl

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-22 Thread Maja Kabiljo
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5987/#review10621 --- Looks good to me now. Again, great work! - Maja Kabiljo On Aug. 22

[jira] [Updated] (GIRAPH-301) InputSplit Reservations are clumping, leaving many workers asleep while other process too many splits and get overloaded.

2012-08-22 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-301: --- Attachment: GIRAPH-301-7.patch just a quick rebase, no changes. > InputSplit Reserva

[jira] [Commented] (GIRAPH-211) Add secure authentication to Netty IPC

2012-08-22 Thread Eugene Koontz (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439703#comment-13439703 ] Eugene Koontz commented on GIRAPH-211: -- Hi Alessandro, I am working actively on it an

[jira] [Commented] (GIRAPH-211) Add secure authentication to Netty IPC

2012-08-22 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439739#comment-13439739 ] Alessandro Presta commented on GIRAPH-211: -- Awesome, looking forward to it!

[jira] [Created] (GIRAPH-312) Giraph needs an admin script

2012-08-22 Thread Eli Reisman (JIRA)
Eli Reisman created GIRAPH-312: -- Summary: Giraph needs an admin script Key: GIRAPH-312 URL: https://issues.apache.org/jira/browse/GIRAPH-312 Project: Giraph Issue Type: New Feature Com

[jira] [Updated] (GIRAPH-312) Giraph needs an admin script

2012-08-22 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-312: --- Attachment: GIRAPH-312-1.patch > Giraph needs an admin script > > >