Re: how do I maintain a cached List across supersteps?
I would use a workercontext, it is shared and persistent during computation by all vertices in a worker. If it's readonly, you won't have to manage concurrency. On Tue, Sep 16, 2014 at 9:42 PM, Matthew Cornell m...@matthewcornell.org wrote: Hi Folks. I have a custom argument that's passed into my Giraph job that needs parsing. The parsed value is accessed by my Vertex#compute. To avoid excessive GC I'd like to cache the parsing results. What's a good way to do so? I looked at using the ImmutableClassesGiraphConfiguration returned by getConf(), but it supports only String properties. I looked at using my custom MasterCompute to manage it, but I couldn't find how to access the master compute instance from the vertex. My last idea is to use (abuse?) an aggregator to do this. I'd appreciate your thoughts! -- matt -- Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson Street, Amherst MA 01002 | matthewcornell.org -- Claudio Martella
Re: how do I maintain a cached List across supersteps?
Thanks to Claudio and Matthew, I went with the WorkerContext solution. Note that I wrote a MasterCompute.validate() to verify the correct WorkerContext class was set. Otherwise I was worried my cast would fail. -- matt On Wed, Sep 17, 2014 at 11:49 AM, Claudio Martella claudio.marte...@gmail.com wrote: I would use a workercontext, it is shared and persistent during computation by all vertices in a worker. If it's readonly, you won't have to manage concurrency. On Tue, Sep 16, 2014 at 9:42 PM, Matthew Cornell m...@matthewcornell.org wrote: Hi Folks. I have a custom argument that's passed into my Giraph job that needs parsing. The parsed value is accessed by my Vertex#compute. To avoid excessive GC I'd like to cache the parsing results. What's a good way to do so? I looked at using the ImmutableClassesGiraphConfiguration returned by getConf(), but it supports only String properties. I looked at using my custom MasterCompute to manage it, but I couldn't find how to access the master compute instance from the vertex. My last idea is to use (abuse?) an aggregator to do this. I'd appreciate your thoughts! -- matt -- Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson Street, Amherst MA 01002 | matthewcornell.org -- Claudio Martella -- Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson Street, Amherst MA 01002 | matthewcornell.org
how do I maintain a cached List across supersteps?
Hi Folks. I have a custom argument that's passed into my Giraph job that needs parsing. The parsed value is accessed by my Vertex#compute. To avoid excessive GC I'd like to cache the parsing results. What's a good way to do so? I looked at using the ImmutableClassesGiraphConfiguration returned by getConf(), but it supports only String properties. I looked at using my custom MasterCompute to manage it, but I couldn't find how to access the master compute instance from the vertex. My last idea is to use (abuse?) an aggregator to do this. I'd appreciate your thoughts! -- matt -- Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson Street, Amherst MA 01002 | matthewcornell.org
