Will do, I just need to figure out how to submit the patch exactly, as well as having some extra time.
And that worked, actually. I'd already figured it out without realizing it; the text file with the aggregator values was created, it was just placed somewhere on my HDFS that I wasn't expecting. On Wed, Jul 31, 2013 at 12:46 PM, Claudio Martella <claudio.marte...@gmail.com> wrote: > Hi Kyle, > > good catch. ALWAYS should be set to 1. Want to write a patch to fix this? > Try to set the property on the command line by putting -D > giraph.textAggregatorWriter.frequency=-1 right after the GiraphRunner class > in your command line. > > Hope this helps. > > Best, > Claudio > > > On Wed, Jul 24, 2013 at 10:31 PM, Kyle Orlando <kyle.r.orla...@gmail.com> > wrote: >> >> Hi Claudio, >> >> So I checked out TextAggregatorWriter and was, initially, still a bit >> confused on how to use it to write to a text file. That's when I >> noticed that, in org.apache.giraph.utils.ConfigurationUtils, there is >> an option "aw", which corresponds to an AggregatorWriterClass. I >> tried this out when running the SimplePageRankComputation program >> using my data as input by specifying this as an option: >> >> -aw org.apache.giraph.aggregators.TextAggregatorWriter. >> >> Here's the full command: >> >> hadoop jar /home/hduser/Documents/combined.jar >> org.apache.giraph.GiraphRunner >> org.apache.giraph.examples.SimplePageRankComputation -eif >> StackExchangeParsee.StackExchangeLongFloatTextEdgeInput -vif >> StackExchangeParsee.StackExchangeLongDoubleTextVertexValueInput -eip >> /in/gaming_edges.txt -vip /in/gaming_vertices.txt -of >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -aw >> org.apache.giraph.aggregators.TextAggregatorWriter -op /outPR -w 2 -mc >> >> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute >> >> According to TextAggregatorWriter, it, by default, writes to a file >> called "aggregatorValues". I checked my HDFS, and did not see that >> particular file. That's when I noticed that there is a configuration >> "giraph.textAggregatorWriter.frequency", and that by default the >> frequency is set to "NEVER", which means that nothing is ever >> created/written to a file for the aggregators. The other two >> frequencies are "AT_THE_END" and "ALWAYS", which strangely both >> correspond to the same integer: -1. Could someone explain why this is >> so? >> >> Ignoring the above uncertainty, I surmised that the property >> "giraph.textAggregatorWriter.frequency" was to be added to my >> giraph-site.xml. I wanted the "AT_THE_END" frequency, which >> corresponds to the value of -1. Here's the contents of my >> giraph-site.xml file: >> >> <configuration> >> <property> >> <name>giraph.textAggregatorWriter.frequency</name> >> <value>-1</value> >> </property> >> </configuration> >> >> I ran the SimplePageRankComputation program again (using the verbose >> "hadoop jar" command above), and still, I couldn't find >> "aggregatorValues" on my HDFS. >> >> Could someone help me out, or at the very least rectify any >> misconceptions and uncertainties that I have? >> >> >> >> On Wed, Jul 24, 2013 at 12:25 PM, Claudio Martella >> <claudio.marte...@gmail.com> wrote: >> > Hi Kyle, >> > >> > you can check out the AggregatorWriter interface which allows you to do >> > that. As a matter of fact there is already a class that implements what >> > you >> > need (org.apache.giraph.aggregators.TextAggregatorWriter). >> > >> > Hope it helps. >> > >> > >> > On Wed, Jul 24, 2013 at 5:19 PM, Kyle Orlando <kyle.r.orla...@gmail.com> >> > wrote: >> >> >> >> Hello, >> >> >> >> I am new to Giraph and was just wondering how one could retrieve and >> >> display the certain global values/statistics that the aggregators keep >> >> track of. What classes and methods would I use, and would this be >> >> done in a class that extends VertexOutputFormat, or would it be done >> >> elsewhere? >> >> >> >> As an example, in the provided SimplePageRankComputation in >> >> org.apache.giraph.examples, there are three aggregators: sum, min, and >> >> max. I would like to display all of their final values (after the >> >> final superstep) in some way, such as writing them to a text file. >> >> >> >> -- >> >> Kyle Orlando >> >> Computer Engineering Major >> >> University of Maryland >> > >> > >> > >> > >> > -- >> > Claudio Martella >> > claudio.marte...@gmail.com >> >> >> >> -- >> Kyle Orlando >> Computer Engineering Major >> University of Maryland > > > > > -- > Claudio Martella > claudio.marte...@gmail.com -- Kyle Orlando Computer Engineering Major University of Maryland
