[jira] [Commented] (GIRAPH-453) Pure hive I/O

2013-02-21 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583965#comment-13583965 ] Hudson commented on GIRAPH-453: --- Integrated in Giraph-trunk-Commit #748 (See [https://build

Re: Review Request: GIRAPH-453: Pure Hive I/O (nitay)

2013-02-21 Thread Nitay Joffe
> On Feb. 21, 2013, 6:45 p.m., Maja Kabiljo wrote: > > This is a lot of great work, Nitay, thanks! I really like that user doesn't > > have to extend the whole Input/Output format anymore, that was a lot of > > code duplication every time. > > > > Is it possible to provide some examples/tests

[jira] [Created] (GIRAPH-534) Add tests / examples for new giraph-hive I/O

2013-02-21 Thread Nitay Joffe (JIRA)
Nitay Joffe created GIRAPH-534: -- Summary: Add tests / examples for new giraph-hive I/O Key: GIRAPH-534 URL: https://issues.apache.org/jira/browse/GIRAPH-534 Project: Giraph Issue Type: Bug

[jira] [Created] (GIRAPH-533) When a pseudo Hadoop cluster (and standalone ZK instance) are running on the local machine, a test fails

2013-02-21 Thread Eli Reisman (JIRA)
Eli Reisman created GIRAPH-533: -- Summary: When a pseudo Hadoop cluster (and standalone ZK instance) are running on the local machine, a test fails Key: GIRAPH-533 URL: https://issues.apache.org/jira/browse/GIRAPH-533

[jira] [Commented] (GIRAPH-13) Port Giraph to YARN

2013-02-21 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583853#comment-13583853 ] Eli Reisman commented on GIRAPH-13: --- I think I can steal a more robust version of the abo

[jira] [Commented] (GIRAPH-532) Give an explanation when trying to use unregistered aggregators

2013-02-21 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583852#comment-13583852 ] Eli Reisman commented on GIRAPH-532: +1 ditto! > Give an explanation

[jira] [Commented] (GIRAPH-532) Give an explanation when trying to use unregistered aggregators

2013-02-21 Thread Nitay Joffe (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583830#comment-13583830 ] Nitay Joffe commented on GIRAPH-532: +1 love it > Give an explanation

[jira] [Commented] (GIRAPH-524) Giraph can receive input from vertex or edge-centric data sets; its output is graph data, not "vertices"

2013-02-21 Thread Nitay Joffe (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583820#comment-13583820 ] Nitay Joffe commented on GIRAPH-524: I think if there is a demand for it we should add

[jira] [Updated] (GIRAPH-13) Port Giraph to YARN

2013-02-21 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-13: -- Attachment: GIRAPH-13-5.patch Here's the latest placeholder. Things are going very well so far, almost d

[jira] [Created] (GIRAPH-532) Give an explanation when trying to use unregistered aggregators

2013-02-21 Thread Maja Kabiljo (JIRA)
Maja Kabiljo created GIRAPH-532: --- Summary: Give an explanation when trying to use unregistered aggregators Key: GIRAPH-532 URL: https://issues.apache.org/jira/browse/GIRAPH-532 Project: Giraph

[jira] [Updated] (GIRAPH-532) Give an explanation when trying to use unregistered aggregators

2013-02-21 Thread Maja Kabiljo (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-532: Attachment: GIRAPH-532.diff > Give an explanation when trying to use unregistered aggregators >

Re: [jira] [Commented] (GIRAPH-283) Change hadoop_trunk profile to hadoop_snapshot

2013-02-21 Thread Eli Reisman
I am thinking in the Hadoop repos its still called "trunk" as far as the git branches go, and we named our profiles around those branches (hadoop version names), "snapshot" is more of a Maven thing. So I'm going say please convince me further. But in theory, not a bad idea. On Sun, Feb 17, 2013 a

[jira] [Created] (GIRAPH-531) EdgeIterables#getEdges() should reuse Edge objects

2013-02-21 Thread Alessandro Presta (JIRA)
Alessandro Presta created GIRAPH-531: Summary: EdgeIterables#getEdges() should reuse Edge objects Key: GIRAPH-531 URL: https://issues.apache.org/jira/browse/GIRAPH-531 Project: Giraph Iss

[jira] [Commented] (GIRAPH-524) Giraph can receive input from vertex or edge-centric data sets; its output is graph data, not "vertices"

2013-02-21 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583739#comment-13583739 ] Eli Reisman commented on GIRAPH-524: That makes sense, and if this doesn't get changed

[jira] [Created] (GIRAPH-530) GiraphInputFormat#getSplits() should be aware of multithreaded input

2013-02-21 Thread Alessandro Presta (JIRA)
Alessandro Presta created GIRAPH-530: Summary: GiraphInputFormat#getSplits() should be aware of multithreaded input Key: GIRAPH-530 URL: https://issues.apache.org/jira/browse/GIRAPH-530 Project: G

[jira] [Commented] (GIRAPH-524) Giraph can receive input from vertex or edge-centric data sets; its output is graph data, not "vertices"

2013-02-21 Thread Alessandro Presta (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583542#comment-13583542 ] Alessandro Presta commented on GIRAPH-524: -- All I'm saying is that the concept of

[jira] [Commented] (GIRAPH-524) Giraph can receive input from vertex or edge-centric data sets; its output is graph data, not "vertices"

2013-02-21 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583491#comment-13583491 ] Eli Reisman commented on GIRAPH-524: Good discussion, and good points. I guess that es

Re: more concurrency issues with requests management?

2013-02-21 Thread Eli Reisman
Sorry thats what I meant, the supernode got the messages, and boom. So even using the new byte buffering, one per vertex, you are hitting the wall here. This is FloatWritible messages, converted to 4 bytes each, in a byte buffer, and the hard limit for messages per vertex is going to be (2^30)/4 an

[jira] [Commented] (GIRAPH-285) Release Giraph-0.2

2013-02-21 Thread Eli Reisman (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583429#comment-13583429 ] Eli Reisman commented on GIRAPH-285: Thanks Alessandro! (My last reply ended up on the

Re: Review Request: GIRAPH-453: Pure Hive I/O (nitay)

2013-02-21 Thread Maja Kabiljo
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8611/#review16868 --- Ship it! Forgot to say, I'm +1 on this :-) - Maja Kabiljo On Feb.

Re: Review Request: GIRAPH-453: Pure Hive I/O (nitay)

2013-02-21 Thread Maja Kabiljo
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8611/#review16867 --- This is a lot of great work, Nitay, thanks! I really like that user d

Re: Review Request: GIRAPH-453: Pure Hive I/O (nitay)

2013-02-21 Thread Nitay Joffe
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8611/ --- (Updated Feb. 21, 2013, 6:17 p.m.) Review request for giraph. Changes ---

Re: more concurrency issues with requests management?

2013-02-21 Thread Maja Kabiljo
Yeah, that's exactly what I told you. We have one ExtendedDataOutput per vertex. And the limit is about (2^30)/4, since you will have double of the size allocated at some point. What do you mean you don't want to use a combiner for evaluation reasons? On 2/21/13 5:38 AM, "Claudio Martella" wrote:

Re: more concurrency issues with requests management?

2013-02-21 Thread Claudio Martella
Yep. Actually more than a supernode sending messages it looks more like a supernode receiving a lot of messages, and hence filling the inbox queue (in bytearrayformat). I assumed a max (2^31)/4 (for float messages) limit because I assumed we had such an object per vertex, but it looks like the Byte

[jira] [Commented] (GIRAPH-528) Decouple vertex implementation from edge storage

2013-02-21 Thread Claudio Martella (JIRA)
[ https://issues.apache.org/jira/browse/GIRAPH-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583187#comment-13583187 ] Claudio Martella commented on GIRAPH-528: - This would be really awesome. I agree w