[jira] [Commented] (GIRAPH-246) Periodic worker calls to context.progress() will prevent timeout on some Hadoop clusters during barrier waits

2012-08-20 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438487#comment-13438487 ] Hudson commented on GIRAPH-246: --- Integrated in Giraph-trunk-Commit #181 (See [https://build

[jira] [Commented] (GIRAPH-214) GiraphJob should have configuration split out of it to be cleaner (GiraphConf)

2012-08-20 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438482#comment-13438482 ] Eli Reisman commented on GIRAPH-214: Still trying to decide how to trick Giraph into a

[jira] [Commented] (GIRAPH-246) Periodic worker calls to context.progress() will prevent timeout on some Hadoop clusters during barrier waits

2012-08-20 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438480#comment-13438480 ] Eli Reisman commented on GIRAPH-246: yes this is unintentional, thanks. Out it goes!

[jira] [Commented] (GIRAPH-26) Improve PseudoRandomVertexInputFormat to create a more realistic synthetic graph (e.g. power-law distributed vertex-cardinality).

2012-08-20 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438378#comment-13438378 ] Eli Reisman commented on GIRAPH-26: --- This is looking really great, nice work. - I'd like

[jira] [Commented] (GIRAPH-26) Improve PseudoRandomVertexInputFormat to create a more realistic synthetic graph (e.g. power-law distributed vertex-cardinality).

2012-08-20 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438327#comment-13438327 ] Jakob Homan commented on GIRAPH-26: --- The colt library is explicitly ok to include (for so

[jira] [Updated] (GIRAPH-309) Message count is wrong

2012-08-20 Thread Avery Ching (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-309: --- Attachment: GIRAPH-309.patch The fix is quite simple, no need for rb. > Message coun

[jira] [Assigned] (GIRAPH-309) Message count is wrong

2012-08-20 Thread Avery Ching (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching reassigned GIRAPH-309: -- Assignee: Avery Ching > Message count is wrong > -- > >

[jira] [Created] (GIRAPH-309) Message count is wrong

2012-08-20 Thread Avery Ching (JIRA)
Avery Ching created GIRAPH-309: -- Summary: Message count is wrong Key: GIRAPH-309 URL: https://issues.apache.org/jira/browse/GIRAPH-309 Project: Giraph Issue Type: Bug Reporter: Avery

Re: [jira] [Updated] (GIRAPH-306) Netty requests should be reliable and implement exactly once semantics

2012-08-20 Thread Eli Reisman
I've run this many times this weekend in its original form "version 1". Ran 1000+ workers on it here with no problem. Barring any check style issues I skipped, ;) this thing is solid. In general Netty is less happy than before with large numbers of connections to maintain as we scale out, but I sus

[jira] [Commented] (GIRAPH-305) Adding an argument to GiraphRunner for Master Compute classes

2012-08-20 Thread Avery Ching (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438150#comment-13438150 ] Avery Ching commented on GIRAPH-305: No, I reran the build and it was fine =).

[jira] [Commented] (GIRAPH-305) Adding an argument to GiraphRunner for Master Compute classes

2012-08-20 Thread Sean Choi (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438142#comment-13438142 ] Sean Choi commented on GIRAPH-305: -- Are there actions that I need to take for this? Does

[jira] [Commented] (GIRAPH-304) Closed channels between workers

2012-08-20 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438120#comment-13438120 ] Eli Reisman commented on GIRAPH-304: After running jobs all weekend I am reluctantly t

[jira] [Commented] (GIRAPH-304) Closed channels between workers

2012-08-20 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438118#comment-13438118 ] Eli Reisman commented on GIRAPH-304: Turns out we get errors now too (see mailing list

[jira] [Updated] (GIRAPH-306) Netty requests should be reliable and implement exactly once semantics

2012-08-20 Thread Avery Ching (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-306: --- Attachment: GIRAPH-306.2.patch - Added small change to not connect to one's self - Also changed the b

Re: [jira] [Commented] (GIRAPH-306) Netty requests should be reliable and implement exactly once semantics

2012-08-20 Thread Eli Reisman
Damn I wish I'd have read this last night! Thanks for the tip. I will try that, I am finding as I (optimistically!) tested at the low memory levels the logs just don't make it to me at all. As I ramp it up a bit, I finally start to get them again. This is why I didn't know if the 246-NEW-FIX-2 patc

Re: [jira] [Commented] (GIRAPH-304) Closed channels between workers

2012-08-20 Thread Eli Reisman
Its been a few weeks since I could run jobs all the time like I have been the last few days. We're seeing all sorts of connection errors now as you guys have been. We are also seeing a point as we scale out to more workers where Netty just can't handle that many connections at once. These Netty im

Re: [jira] [Commented] (GIRAPH-249) Move part of the graph out-of-core when memory is low

2012-08-20 Thread Eli Reisman
Great discussion here, very interesting read. In some cases it seems like a simple command line options (spill to disk or not boolean etc.) are great when the vertex requires no other changes or recompilation to run under this strategy. For more dramatic changes like using more compact, reusable da

Re: [jira] [Commented] (GIRAPH-249) Move part of the graph out-of-core when memory is low

2012-08-20 Thread Gianmarco De Francisci Morales
Hi Benjamin, Thanks for sharing your ideas. First, let me clarify that my proposal does not aim at excluding any of the current use cases from Giraph. I just would like to improve some common use cases that come up very often. Now, on the technical side of processing RDF. The type of a vertex/edg

Re: Review Request: Move part of the graph out-of-core when memory is low

2012-08-20 Thread Alessandro Presta
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5987/ --- (Updated Aug. 20, 2012, 2:04 p.m.) Review request for giraph and Avery Ching.

Re: [jira] [Commented] (GIRAPH-249) Move part of the graph out-of-core when memory is low

2012-08-20 Thread Benjamin Heitmann
Hello, just a few in-line comments regarding the simplification of vertex classes. In my opinion the proposed change might exclude all typed graphs, and all Sematic Web style processing from Giraph. On 17 Aug 2012, at 14:30, Gianmarco De Francisci Morales wrote: > In any case, if one wanted

[jira] [Updated] (GIRAPH-249) Move part of the graph out-of-core when memory is low

2012-08-20 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Presta updated GIRAPH-249: - Attachment: GIRAPH-249.patch This restores Hadoop RPC functionality, so that we're not bl

[jira] [Updated] (GIRAPH-304) Closed channels between workers

2012-08-20 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Presta updated GIRAPH-304: - Attachment: GIRAPH-304.patch This sets a TCP backlog equal to the number of workers.

[jira] [Commented] (GIRAPH-306) Netty requests should be reliable and implement exactly once semantics

2012-08-20 Thread Maja Kabiljo (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437763#comment-13437763 ] Maja Kabiljo commented on GIRAPH-306: - As for connecting to ourselves, even though we