Great discussion here, very interesting read. In some cases it seems like a
simple command line options (spill to disk or not boolean etc.) are great
when the vertex requires no other changes or recompilation to run under
this strategy. For more dramatic changes like using more compact, reusable
da
Hi Benjamin,
Thanks for sharing your ideas.
First, let me clarify that my proposal does not aim at excluding any of the
current use cases from Giraph. I just would like to improve some common use
cases that come up very often.
Now, on the technical side of processing RDF.
The type of a vertex/edg
Hello, just a few in-line comments regarding the simplification of vertex
classes.
In my opinion the proposed change might exclude all typed graphs, and all
Sematic Web style processing from Giraph.
On 17 Aug 2012, at 14:30, Gianmarco De Francisci Morales wrote:
> In any case, if one wanted
I think at that point it would be worth having a new logical place for
vertex/edge representation at worker- or partition-level.
Avery had some ideas about this.
Basically right now we're giving the user the freedom (and responsibility)
to choose a representation (both in-memory and for serializat
I was under the impression that 100k was the upper limit to make things
work without crashing.
In any case, if one wanted to use a compressed memory representation by
aggregating different edge lists together, could one use the worker context
as a central point of access to the compressed graphs?
The example where we actually go out of memory was with 500K vertices and
500M edges, but yes, as a general rule we should strive to reduce our
memory footprint in order to push the point where we need to go out of
core as far away as possible.
On 8/17/12 2:11 PM, "Gianmarco De Francisci Morales"
Very interesting.
On a side note, a graph with 100k vertices and 100M edges is largish but
not that big after all.
If it does not fit on 10+ GB of memory, it means that each edge occupies
around 100B (amortizing the cost of the vertex over the edges).
In my opinion this deserves some thought.
If m
Great metrics, this made a very interesting read, and great code too as
always. This must have been a lot of work. I like the idea of eliminating
the extra temporary storage data structures where possible, even when not
going out-of-core. I think that + avoiding extra object creation during the
wor
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435437#comment-13435437
]
Alessandro Presta commented on GIRAPH-249:
--
Thanks Claudio, good observation.
You
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435293#comment-13435293
]
Claudio Martella commented on GIRAPH-249:
-
Hi Alessandro,
this is very interestin
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435290#comment-13435290
]
Alessandro Presta commented on GIRAPH-249:
--
https://reviews.apache.org/r/5987/dif
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419531#comment-13419531
]
Claudio Martella commented on GIRAPH-249:
-
Yes, agreed. We should definitely re-us
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418244#comment-13418244
]
Maja Kabiljo commented on GIRAPH-249:
-
We do create a lot of objects and GC doesn't ge
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417267#comment-13417267
]
Claudio Martella commented on GIRAPH-249:
-
of course where I mentioned Message(s),
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417085#comment-13417085
]
Claudio Martella commented on GIRAPH-249:
-
Just to be clear, i'm cautious mostly w
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417075#comment-13417075
]
Alessandro Presta commented on GIRAPH-249:
--
I also think we should be cautious wi
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417059#comment-13417059
]
Claudio Martella commented on GIRAPH-249:
-
Hey guys, great work and great discussi
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417050#comment-13417050
]
Alessandro Presta commented on GIRAPH-249:
--
Got it, thanks.
I don't know, it soun
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416721#comment-13416721
]
Eli Reisman commented on GIRAPH-249:
sorry formatting did not come out. so after cycli
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416716#comment-13416716
]
Eli Reisman commented on GIRAPH-249:
The partitioning when we actually get them to the
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416081#comment-13416081
]
Alessandro Presta commented on GIRAPH-249:
--
I totally believe you on the input su
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415831#comment-13415831
]
Eli Reisman commented on GIRAPH-249:
I did test against trunk, I what I meant above is
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415530#comment-13415530
]
Alessandro Presta commented on GIRAPH-249:
--
Eli, thanks a lot for running these b
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415456#comment-13415456
]
Eli Reisman commented on GIRAPH-249:
Died again on slightly larger job but one that is
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415433#comment-13415433
]
Eli Reisman commented on GIRAPH-249:
Second run on same data with smaller splitmb made
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415377#comment-13415377
]
Eli Reisman commented on GIRAPH-249:
I just ran the latest 249 patch with 21% of the d
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415352#comment-13415352
]
Eli Reisman commented on GIRAPH-249:
That makes a lot of sense. I guess the key differ
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415139#comment-13415139
]
Alessandro Presta commented on GIRAPH-249:
--
As far as I know, out-of-core is part
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414729#comment-13414729
]
Eli Reisman commented on GIRAPH-249:
Yes, I had to drop GIRAPH-247 and start GIRAPH-25
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413847#comment-13413847
]
Alessandro Presta commented on GIRAPH-249:
--
(By improve, I mean scale to bigger d
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413845#comment-13413845
]
Alessandro Presta commented on GIRAPH-249:
--
We're not currently combining on the
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413828#comment-13413828
]
Eli Reisman commented on GIRAPH-249:
I think they handle the transient messages that w
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413250#comment-13413250
]
Eli Reisman commented on GIRAPH-249:
What you have said really has got me thinking abo
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413078#comment-13413078
]
Alessandro Presta commented on GIRAPH-249:
--
Thanks for the tips. I see that the r
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412971#comment-13412971
]
Eli Reisman commented on GIRAPH-249:
The input I have from jakob is don't mess with th
[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412968#comment-13412968
]
Eli Reisman commented on GIRAPH-249:
I've been looking at all this stuff too for my in
36 matches
Mail list logo
