No worries. Just by the way, I realized after I sent that that using the
"public static int numPreprocessingSteps" to store the value in the
MasterCompute class doesn't work; you need to register a permanent
aggregator to hold on to it, if you need to.
Best,
Matthew
On Wed, Sep 10, 2014 at 3:15 PM, Matthew Cornell <m...@matthewcornell.org>
wrote:
> Sorry for the long delay, Matthew. That's really helpful. Right now I'm
> stuck on apparently running out of memory on our little cluster, but the
> log messages are confusing. I'm putting together a question, but in the
> meantime I'll try one of the simpler examples such as degree count to see
> if /anything/ will run against my graph, which is very small (100K and
> edges nodes). -- matt
>
> On Thu, Aug 28, 2014 at 2:26 PM, Matthew Saltz <sal...@gmail.com> wrote:
>
>> Matt,
>>
>> I'm not sure if you've resolved this problem already or not, but if you
>> haven't: The initialize() method isn't limited to registering aggregators,
>> and in fact, in my project I use it to do exactly what you're describing to
>> check and load custom configuration parameters. Inside the initialize()
>> method, I do this:
>>
>> * String numPreprocessingStepsConf =
>> getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);*
>> * numPreprocessingSteps = (numPreprocessingStepsConf != null) ?*
>> * Integer.parseInt(numPreprocessingStepsConf.trim()) :*
>> * DEFAULT_NUMBER_OF_PREPROCESSING_STEPS;*
>> * System.out.println("Number of preprocessing steps: " +
>> numPreprocessingSteps);*
>>
>> where at the class level I declare:
>>
>> public static final String NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT =
>> "wcc.numPreprocessingSteps";
>> public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS = 1;
>> public static int numPreprocessingSteps;
>>
>> To set the property, I use the option "-ca
>> wcc.numPreprocessingSteps=<number of steps I want>". If you need to check
>> that it's properly formatted and not store them, this is a fine place to do
>> it as well, given that it's run before the input superstep (see the giraph
>> code in BspServiceMaster, line 1617 in the stable 1.1.0 release). What
>> happens is that on the master, the MasterThread calls coordinateSuperstep()
>> on a BspServiceMaster object, which checks if it's the input superstep, and
>> if so, calls initialize() on the MasterCompute object (created in the
>> becomeMaster() method of BspServiceMaster).
>>
>> Hope this helps,
>> Matthew
>>
>>
>>
>> On Tue, Aug 26, 2014 at 4:36 PM, Matthew Cornell <m...@matthewcornell.org
>> > wrote:
>>
>>> Hi again. My application needs to pass in a String argument to the
>>> computation which each Vertex needs access to. (The argument is a list of
>>> the form "[item1, item2, ...]".) I found --customArguments (which I set in
>>> my tests via conf.set(<arg_name>, <arg_val>)) but I need to check that it's
>>> properly formatted. Where do I do that? The only thing I thought of is to
>>> specify a DefaultMasterCompute subclass whose initialize() does the check,
>>> but all the initialize() examples do is register aggregators; none of them
>>> check args or do anything else. Thanks in advance! -- matt
>>>
>>> --
>>> Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson
>>> Street, Amherst MA 01002 | matthewcornell.org
>>>
>>
>>
>
>
> --
> Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson
> Street, Amherst MA 01002 | matthewcornell.org
>
