Ok, ignoring everything about I/O I can run the below code.
But then it hangs at 0% reduce forever
-bash-3.2$ hadoop jar
target/geocoderV2-1.0-SNAPSHOT-jar-with-dependencies.jar
com.hrl.issl.osi.networks.HelloGiraph0p2
13/02/21 17:33:28 INFO graph.GiraphJob: run: Since checkpointing is
disabled (default), do not allow any task retries (setting
mapred.map.max.attempts = 0, old value = 4)
13/02/21 17:33:29 WARN bsp.BspOutputFormat: checkOutputSpecs:
ImmutableOutputCommiter will not check anything
13/02/21 17:33:29 INFO mapred.JobClient: Running job: job_201302201124_0136
13/02/21 17:33:30 INFO mapred.JobClient: map 0% reduce 0%
13/02/21 17:34:03 INFO mapred.JobClient: map 74% reduce 0%
13/02/21 17:34:04 INFO mapred.JobClient: map 80% reduce 0%
13/02/21 17:34:06 INFO mapred.JobClient: map 100% reduce 0%
What did I miss?
import java.io.IOException;
import org.apache.giraph.combiner.DoubleSumCombiner;
import org.apache.giraph.conf.GiraphConfiguration;
import org.apache.giraph.graph.GiraphJob;
import org.apache.giraph.io.formats.PseudoRandomVertexInputFormat;
import org.apache.giraph.vertex.EdgeListVertex;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.Logger;
/**
* Default Pregel-style PageRank computation.
*/
public class HelloGiraph0p2 implements Tool {
/**
* Configuration from Configurable
*/
private Configuration conf;
@Override
public Configuration getConf() {
return conf;
}
@Override
public void setConf(Configuration conf) {
this.conf = conf;
}
@Override
public final int run(final String[] args) throws Exception {
String name = getClass().getName();
GiraphJob job = new GiraphJob(getConf(), name);
GiraphConfiguration configuration = job.getConfiguration();
configuration.setVertexClass(EdgeListVertexTwoPlusTwo.class);
configuration.useUnsafeSerialization(true);
configuration.setVertexCombinerClass(DoubleSumCombiner.class);
configuration.setVertexInputFormatClass(PseudoRandomVertexInputFormat.class);
configuration.setLong(PseudoRandomVertexInputFormat.AGGREGATE_VERTICES,
Long.parseLong("1000") );
configuration.setLong(PseudoRandomVertexInputFormat.EDGES_PER_VERTEX,Long.parseLong("10"));
int workers = Integer.parseInt("30");
configuration.setWorkerConfiguration(workers, workers, 100.0f);
//configuration.setInt(PageRankComputation.SUPERSTEP_COUNT,
Integer.parseInt("10"));
boolean isVerbose = true;
if (job.run(isVerbose)) {
return 0;
} else {
return -1;
}
}
int NUM_SUPERSTEPS = 5;
public class EdgeListVertexTwoPlusTwo extends
EdgeListVertex<LongWritable, DoubleWritable, DoubleWritable,
DoubleWritable> {
@Override
public void compute(Iterable<DoubleWritable> messages) throws IOException {
if (this.getSuperstep() >= 1) {
double four = 2+2;
this.setValue( new DoubleWritable(four));
}
if (this.getSuperstep() < NUM_SUPERSTEPS) {
this.sendMessageToAllEdges( new DoubleWritable(this.getValue().get()));
} else {
this.voteToHalt();
}
}
}
/**
* Execute the benchmark.
*
* @param args Typically the command line arguments.
* @throws Exception Any exception from the computation.
*/
public static void main(final String[] args) throws Exception {
System.exit(ToolRunner.run(new HelloGiraph0p2(), args));
}
}
On Thu, Feb 21, 2013 at 4:10 PM, Ryan Compton <compton.r...@gmail.com> wrote:
> Ok, I've been looking at the PageRankBenchmark. There's a lot going on
> in there...
>
> It looks like the minimum amount of stuff I need to run a do-nothing
> job is what I've got below.
>
> But now it's telling me that (and PageRankBenchmark doesn't have the
> word "output" anywhere).
>
> 13/02/21 16:08:37 ERROR security.UserGroupInformation:
> PriviledgedActionException as:rfcompton (auth:SIMPLE)
> cause:org.apache.hadoop.mapred.InvalidJobConfException: Output
> directory not set.
> Exception in thread "main"
> org.apache.hadoop.mapred.InvalidJobConfException: Output directory not
> set.
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:123)
> at
> org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:48)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:872)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
> at org.apache.giraph.graph.GiraphJob.run(GiraphJob.java:282)
> at com.hrl.issl.osi.networks.HelloGiraph0p2.run(HelloGiraph0p2.java:49)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at com.hrl.issl.osi.networks.HelloGiraph0p2.main(HelloGiraph0p2.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>
>
>
> import org.apache.giraph.conf.GiraphConfiguration;
> import org.apache.giraph.graph.GiraphJob;
> import org.apache.giraph.io.formats.JsonBase64VertexInputFormat;
> import org.apache.giraph.io.formats.JsonBase64VertexOutputFormat;
> import org.apache.giraph.io.formats.TextVertexInputFormat;
> import org.apache.giraph.io.formats.TextVertexOutputFormat;
> import org.apache.giraph.vertex.SimpleVertex;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.util.Tool;
> import org.apache.hadoop.util.ToolRunner;
>
> /**
> *
> * Hello world giraph 0.2...
> *
> */
> public class HelloGiraph0p2 implements Tool {
> /** Configuration */
> private Configuration conf;
>
> @Override
> public void setConf(Configuration conf) {
> this.conf = conf;
> }
>
> @Override
> public Configuration getConf() {
> return conf;
> }
>
> @Override
> public int run(String[] arg0) throws Exception {
>
> GiraphJob job = new GiraphJob(getConf(), getClass().getName());
>
> GiraphConfiguration configuration = job.getConfiguration();
>
> configuration.setVertexClass(SimpleVertex.class);
>
> configuration.setVertexInputFormatClass(JsonBase64VertexInputFormat.class);
> configuration.setVertexOutputFormatClass(JsonBase64VertexOutputFormat.class);
>
> configuration.setWorkerConfiguration(30, 30, 100.0f);
>
> return job.run(true) ? 0 : -1;
>
> }
>
> public static void main(String[] args) throws Exception {
>
> System.exit(ToolRunner.run(new HelloGiraph0p2(), args));
> }
>
> }
>
> On Thu, Feb 21, 2013 at 2:58 PM, Maja Kabiljo <majakabi...@fb.com> wrote:
>> Hi Ryan,
>>
>> Before running the job, you need to set Vertex and input/output format
>> classes on it. Please take a look at one of the benchmarks to see how to
>> do that. Alternatively, you can try using GiraphRunner, where you pass
>> these classes as command line arguments.
>>
>> Maja
>>
>> On 2/21/13 2:43 PM, "Ryan Compton" <compton.r...@gmail.com> wrote:
>>
>>>I'm still struggling with this. I am trying to use 0.2, I dont have
>>>permissions to edit core-site.xml
>>>
>>>I think this the most basic boiler plate code for a 0.2 Giraph
>>>project, but I still can't run it.
>>>
>>>Exception in thread "main" java.lang.NullPointerException
>>>at
>>>org.apache.giraph.utils.ReflectionUtils.getTypeArguments(ReflectionUtils.j
>>>ava:85)
>>>at
>>>org.apache.giraph.conf.GiraphClasses.readFromConf(GiraphClasses.java:117)
>>>at org.apache.giraph.conf.GiraphClasses.<init>(GiraphClasses.java:105)
>>>at
>>>org.apache.giraph.conf.ImmutableClassesGiraphConfiguration.<init>(Immutabl
>>>eClassesGiraphConfiguration.java:84)
>>>at
>>>com.hrl.issl.osi.networks.HelloGiraph0p2.setConf(HelloGiraph0p2.java:34)
>>>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:61)
>>>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>at com.hrl.issl.osi.networks.HelloGiraph0p2.main(HelloGiraph0p2.java:70)
>>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>at
>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>>39)
>>>at
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>>pl.java:25)
>>>at java.lang.reflect.Method.invoke(Method.java:597)
>>>at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>>>
>>>
>>>
>>>package networks;
>>>
>>>import java.io.IOException;
>>>
>>>import org.apache.giraph.conf.ImmutableClassesGiraphConfiguration;
>>>import org.apache.giraph.graph.GiraphJob;
>>>import org.apache.giraph.vertex.EdgeListVertex;
>>>import org.apache.hadoop.conf.Configuration;
>>>import org.apache.hadoop.io.LongWritable;
>>>import org.apache.hadoop.io.Text;
>>>import org.apache.hadoop.util.Tool;
>>>import org.apache.hadoop.util.ToolRunner;
>>>import org.apache.log4j.Logger;
>>>
>>>/**
>>> *
>>> * Hello world giraph 0.2...
>>> *
>>> */
>>>public class HelloGiraph0p2 extends EdgeListVertex<LongWritable, Text,
>>>Text, Text> implements Tool {
>>>/** Configuration */
>>>private ImmutableClassesGiraphConfiguration<LongWritable, Text, Text,
>>>Text> conf;
>>>/** Class logger */
>>>private static final Logger LOG = Logger.getLogger(HelloGiraph0p2.class);
>>>
>>>@Override
>>>public void compute(Iterable<Text> arg0) throws IOException {
>>>int four = 2+2;
>>>}
>>>@Override
>>>public void setConf(Configuration configurationIn) {
>>>this.conf = new ImmutableClassesGiraphConfiguration<LongWritable,
>>>Text, Text, Text>(configurationIn);
>>>return;
>>>}
>>>@Override
>>>public ImmutableClassesGiraphConfiguration<LongWritable, Text, Text,
>>>Text> getConf() {
>>>return conf;
>>>}
>>>
>>>/**
>>>*
>>>* ToolRunner run
>>>*
>>>* @param arg0
>>>* @return
>>>* @throws Exception
>>>*/
>>>@Override
>>>public int run(String[] arg0) throws Exception {
>>>GiraphJob job = new GiraphJob(getConf(), getClass().getName());
>>>
>>>return job.run(true) ? 0 : -1;
>>>
>>>}
>>>/**
>>>* main...
>>>*
>>>* @param args
>>>* @throws Exception
>>>*/
>>>public static void main(String[] args) throws Exception {
>>>System.exit(ToolRunner.run(new HelloGiraph0p2(), args));
>>>}
>>>
>>>}
>>>
>>>
>>>
>>>On Tue, Feb 5, 2013 at 4:24 AM, Gustavo Enrique Salazar Torres
>>><gsala...@ime.usp.br> wrote:
>>>> Hi Ryan:
>>>>
>>>> I got that same error and discovered that I have to start a zookeeper
>>>> instance. What I did was to download Zookeeper, write a new zoo.cfg file
>>>> under conf directory with the following:
>>>>
>>>> dataDir=/home/user/zookeeper-3.4.5/tmp
>>>> clientPort=2181
>>>>
>>>> Also I added some lines in Hadoop's core-site.xml:
>>>> <property>
>>>> <name>giraph.zkList</name>
>>>> <value>localhost:2181</value>
>>>> </property>
>>>>
>>>> Then I start Zookeper with bin/zkServer.sh start (also you will have to
>>>> restart Hadoop) and then you can launch your Giraph Job.
>>>> This setup worked for me (maybe there is an easiest way :D), hope it is
>>>> useful.
>>>>
>>>> Best regards
>>>> Gustavo
>>>>
>>>>
>>>> On Mon, Feb 4, 2013 at 10:06 PM, Ryan Compton <compton.r...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Ok great, thanks. I've been working with 0.1, I can get things to
>>>>> compile (see below code) but they still are not running, the maps hang
>>>>> (also below). I have no idea how to fix it, I may consider updating
>>>>> that code I have that compiles to 0.2 and see if it works then. The
>>>>> only difference I can see is that 0.2 requires everything have a
>>>>> "message"
>>>>>
>>>>> -bash-3.2$ hadoop jar target/giraph-0.1-jar-with-dependencies.jar
>>>>> com.SimpleGiraphSumEdgeWeights /user/rfcompton/giraphTSPInput
>>>>> /user/rfcompton/giraphTSPOutput 3 3
>>>>> 13/02/04 15:48:23 INFO mapred.JobClient: Running job:
>>>>> job_201301230932_1199
>>>>> 13/02/04 15:48:24 INFO mapred.JobClient: map 0% reduce 0%
>>>>> 13/02/04 15:48:35 INFO mapred.JobClient: map 25% reduce 0%
>>>>> 13/02/04 15:58:40 INFO mapred.JobClient: Task Id :
>>>>> attempt_201301230932_1199_m_000003_0, Status : FAILED
>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable
>>>>> exception setup: Offlining servers due to exception...
>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at
>>>>>
>>>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio
>>>>>n.java:1157)
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>>>> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
>>>>> exception...
>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>>>>> ... 7 more
>>>>> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>>>>> at
>>>>>
>>>>>org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582
>>>>>)
>>>>> at org.apache.
>>>>> Task attempt_201301230932_1199_m_000003_0 failed to report status for
>>>>> 600 seconds. Killing!
>>>>> 13/02/04 15:58:43 INFO mapred.JobClient: Task Id :
>>>>> attempt_201301230932_1199_m_000002_0, Status : FAILED
>>>>> Task attempt_201301230932_1199_m_000002_0 failed to report status for
>>>>> 600 seconds. Killing!
>>>>> 13/02/04 15:58:43 INFO mapred.JobClient: Task Id :
>>>>> attempt_201301230932_1199_m_000000_0, Status : FAILED
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>========================================================================
>>>>>=========================
>>>>> This is the code I was using:
>>>>>
>>>>> import com.google.common.base.Preconditions;
>>>>> import com.google.common.collect.Maps;
>>>>>
>>>>> import org.apache.giraph.comm.ArrayListWritable;
>>>>> import org.apache.giraph.graph.BasicVertex;
>>>>> import org.apache.giraph.graph.BspUtils;
>>>>> import org.apache.giraph.graph.GiraphJob;
>>>>> import org.apache.giraph.graph.EdgeListVertex;
>>>>> import org.apache.giraph.graph.VertexReader;
>>>>> import org.apache.giraph.graph.VertexWriter;
>>>>> import org.apache.giraph.lib.TextVertexInputFormat;
>>>>> import org.apache.giraph.lib.TextVertexInputFormat.TextVertexReader;
>>>>> import org.apache.giraph.lib.TextVertexOutputFormat;
>>>>> import org.apache.giraph.lib.TextVertexOutputFormat.TextVertexWriter;
>>>>> import org.apache.hadoop.conf.Configuration;
>>>>> import org.apache.hadoop.fs.Path;
>>>>> import org.apache.hadoop.io.DoubleWritable;
>>>>> import org.apache.hadoop.io.FloatWritable;
>>>>> import org.apache.hadoop.io.IntWritable;
>>>>> import org.apache.hadoop.io.LongWritable;
>>>>> import org.apache.hadoop.io.Text;
>>>>> import org.apache.hadoop.mapreduce.InputSplit;
>>>>> import org.apache.hadoop.mapreduce.RecordReader;
>>>>> import org.apache.hadoop.mapreduce.RecordWriter;
>>>>> import org.apache.hadoop.mapreduce.TaskAttemptContext;
>>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>>> import org.apache.hadoop.util.Tool;
>>>>> import org.apache.hadoop.util.ToolRunner;
>>>>> import org.apache.log4j.Logger;
>>>>> import org.json.JSONArray;
>>>>> import org.json.JSONException;
>>>>>
>>>>> import java.io.IOException;
>>>>> import java.util.ArrayList;
>>>>> import java.util.Iterator;
>>>>> import java.util.Map;
>>>>> import java.util.StringTokenizer;
>>>>>
>>>>> /**
>>>>> * Shows an example of a brute-force implementation of the Travelling
>>>>> Salesman Problem
>>>>> */
>>>>> public class SimpleGiraphSumEdgeWeights extends
>>>>> EdgeListVertex<LongWritable, ArrayListWritable<DoubleWritable>,
>>>>> FloatWritable, ArrayListWritable<Text>> implements Tool {
>>>>> /** Configuration */
>>>>> private Configuration conf;
>>>>> /** Class logger */
>>>>> private static final Logger LOG =
>>>>> Logger.getLogger(SimpleGiraphSumEdgeWeights.class);
>>>>> /** The shortest paths id */
>>>>> public static String SOURCE_ID =
>>>>>"SimpleShortestPathsVertex.sourceId";
>>>>> /** Default shortest paths id */
>>>>> public static long SOURCE_ID_DEFAULT = 1;
>>>>>
>>>>>
>>>>> /**
>>>>> * Is this vertex the source id?
>>>>> *
>>>>> * @return True if the source id
>>>>> */
>>>>> private boolean isSource() {
>>>>> return (getVertexId().get() ==
>>>>> getContext().getConfiguration().getLong(SOURCE_ID,
>>>>>
>>>>>SOURCE_ID_DEFAULT));
>>>>> }
>>>>> public class Message extends ArrayListWritable<Text> {
>>>>> public Message() {
>>>>> super();
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void setClass() {
>>>>> // TODO Auto-generated method stub
>>>>> }
>>>>> }
>>>>> public class Valeur extends ArrayListWritable<DoubleWritable> {
>>>>> public Valeur() {
>>>>> super();
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void setClass() {
>>>>> // TODO Auto-generated method stub
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void compute(Iterator<ArrayListWritable<Text>> msgIterator)
>>>>>{
>>>>> System.out.println("**** LAUNCHING COMPUTATION FOR VERTEX
>>>>> "+this.getVertexId().get()+", SUPERSTEP "+this.getSuperstep()+"
>>>>> ****");
>>>>> //We get the source ID, we will need it
>>>>> String sourceID = new
>>>>> LongWritable(this.getContext().getConfiguration().getLong(SOURCE_ID,
>>>>> SOURCE_ID_DEFAULT)).toString();
>>>>> //We get the total number of verticles, and the current superstep
>>>>> number, we will need it too
>>>>> int J=1;
>>>>>
>>>>> voteToHalt();
>>>>> }
>>>>>
>>>>> /**
>>>>> * VertexInputFormat that supports {@link
>>>>>SimpleGiraphSumEdgeWeights}
>>>>> */
>>>>> public static class SimpleShortestPathsVertexInputFormat extends
>>>>> TextVertexInputFormat<LongWritable, ArrayListWritable<DoubleWritable>,
>>>>> FloatWritable,
>>>>> DoubleWritable> {
>>>>> @Override
>>>>> public VertexReader<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>, FloatWritable, DoubleWritable>
>>>>> createVertexReader(InputSplit split,
>>>>> TaskAttemptContext context)
>>>>> throws IOException {
>>>>> return new SimpleShortestPathsVertexReader(
>>>>> textInputFormat.createRecordReader(split, context));
>>>>> }
>>>>> }
>>>>>
>>>>> /**
>>>>> * VertexReader that supports {@link SimpleGiraphSumEdgeWeights}.
>>>>>In
>>>>> this
>>>>> * case, the edge values are not used. The files should be in the
>>>>> * following JSON format:
>>>>> * JSONArray(<vertex id>, <vertex value>,
>>>>> * JSONArray(JSONArray(<dest vertex id>, <edge value>),
>>>>> ...))
>>>>> * Here is an example with vertex id 1, vertex value 4.3, and two
>>>>> edges.
>>>>> * First edge has a destination vertex 2, edge value 2.1.
>>>>> * Second edge has a destination vertex 3, edge value 0.7.
>>>>> * [1,4.3,[[2,2.1],[3,0.7]]]
>>>>> */
>>>>> public static class SimpleShortestPathsVertexReader extends
>>>>> TextVertexReader<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>, FloatWritable,
>>>>> DoubleWritable> {
>>>>>
>>>>> public SimpleShortestPathsVertexReader(
>>>>> RecordReader<LongWritable, Text> lineRecordReader) {
>>>>> super(lineRecordReader);
>>>>> }
>>>>>
>>>>> public class Valeur extends ArrayListWritable<DoubleWritable> {
>>>>> public Valeur() {
>>>>> super();
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void setClass() {
>>>>> // TODO Auto-generated method stub
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> @Override
>>>>> public BasicVertex<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>, FloatWritable,
>>>>> DoubleWritable> getCurrentVertex()
>>>>> throws IOException, InterruptedException {
>>>>> BasicVertex<LongWritable, ArrayListWritable<DoubleWritable>,
>>>>> FloatWritable,
>>>>> DoubleWritable> vertex = BspUtils.<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>, FloatWritable,
>>>>>
>>>>> DoubleWritable>createVertex(getContext().getConfiguration());
>>>>>
>>>>> Text line = getRecordReader().getCurrentValue();
>>>>> try {
>>>>> JSONArray jsonVertex = new JSONArray(line.toString());
>>>>> LongWritable vertexId = new
>>>>> LongWritable(jsonVertex.getLong(0));
>>>>> Valeur vertexValue = new Valeur();
>>>>> vertexValue.add(new
>>>>> DoubleWritable(jsonVertex.getDouble(1)));
>>>>> Map<LongWritable, FloatWritable> edges =
>>>>> Maps.newHashMap();
>>>>> JSONArray jsonEdgeArray = jsonVertex.getJSONArray(2);
>>>>> for (int i = 0; i < jsonEdgeArray.length(); ++i) {
>>>>> JSONArray jsonEdge = jsonEdgeArray.getJSONArray(i);
>>>>> edges.put(new LongWritable(jsonEdge.getLong(0)),
>>>>> new FloatWritable((float)
>>>>> jsonEdge.getDouble(1)));
>>>>> }
>>>>> vertex.initialize(vertexId, vertexValue, edges, null);
>>>>> } catch (JSONException e) {
>>>>> throw new IllegalArgumentException(
>>>>> "next: Couldn't get vertex from line " +
>>>>> line.toString(), e);
>>>>> }
>>>>> return vertex;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean nextVertex() throws IOException,
>>>>> InterruptedException {
>>>>> return getRecordReader().nextKeyValue();
>>>>> }
>>>>> }
>>>>>
>>>>> /**
>>>>> * VertexOutputFormat that supports {@link
>>>>>SimpleGiraphSumEdgeWeights}
>>>>> */
>>>>> public static class SimpleShortestPathsVertexOutputFormat extends
>>>>> TextVertexOutputFormat<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>,
>>>>> FloatWritable> {
>>>>>
>>>>> @Override
>>>>> public VertexWriter<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>, FloatWritable>
>>>>> createVertexWriter(TaskAttemptContext context)
>>>>> throws IOException, InterruptedException {
>>>>> RecordWriter<Text, Text> recordWriter =
>>>>> textOutputFormat.getRecordWriter(context);
>>>>> return new SimpleShortestPathsVertexWriter(recordWriter);
>>>>> }
>>>>> }
>>>>>
>>>>> /**
>>>>> * VertexWriter that supports {@link SimpleGiraphSumEdgeWeights}
>>>>> */
>>>>> public static class SimpleShortestPathsVertexWriter extends
>>>>> TextVertexWriter<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>, FloatWritable> {
>>>>> public SimpleShortestPathsVertexWriter(
>>>>> RecordWriter<Text, Text> lineRecordWriter) {
>>>>> super(lineRecordWriter);
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void writeVertex(BasicVertex<LongWritable,
>>>>> ArrayListWritable<DoubleWritable>,
>>>>> FloatWritable, ?> vertex)
>>>>> throws IOException, InterruptedException {
>>>>> String sourceID = new
>>>>> LongWritable(vertex.getContext().getConfiguration().getLong(SOURCE_ID,
>>>>> SOURCE_ID_DEFAULT)).toString();
>>>>> JSONArray jsonVertex = new JSONArray();
>>>>> try {
>>>>> jsonVertex.put(vertex.getVertexId().get());
>>>>> jsonVertex.put(vertex.getVertexValue().toString());
>>>>> JSONArray jsonEdgeArray = new JSONArray();
>>>>> for (LongWritable targetVertexId : vertex) {
>>>>> JSONArray jsonEdge = new JSONArray();
>>>>> jsonEdge.put(targetVertexId.get());
>>>>>
>>>>> jsonEdge.put(vertex.getEdgeValue(targetVertexId).get());
>>>>> jsonEdgeArray.put(jsonEdge);
>>>>> }
>>>>> jsonVertex.put(jsonEdgeArray);
>>>>> } catch (JSONException e) {
>>>>> throw new IllegalArgumentException(
>>>>> "writeVertex: Couldn't write vertex " + vertex);
>>>>> }
>>>>> getRecordWriter().write(new Text(jsonVertex.toString()),
>>>>> null);
>>>>> }
>>>>> }
>>>>>
>>>>> @Override
>>>>> public Configuration getConf() {
>>>>> return conf;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void setConf(Configuration conf) {
>>>>> this.conf = conf;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int run(String[] argArray) throws Exception {
>>>>> Preconditions.checkArgument(argArray.length == 4,
>>>>> "run: Must have 4 arguments <input path> <output path> " +
>>>>> "<source vertex id> <# of workers>");
>>>>>
>>>>> GiraphJob job = new GiraphJob(getConf(), getClass().getName());
>>>>> job.setVertexClass(getClass());
>>>>> job.setVertexInputFormatClass(
>>>>> SimpleShortestPathsVertexInputFormat.class);
>>>>> job.setVertexOutputFormatClass(
>>>>> SimpleShortestPathsVertexOutputFormat.class);
>>>>> FileInputFormat.addInputPath(job, new Path(argArray[0]));
>>>>> FileOutputFormat.setOutputPath(job, new Path(argArray[1]));
>>>>>
>>>>> job.getConfiguration().setLong(SimpleGiraphSumEdgeWeights.SOURCE_ID,
>>>>> Long.parseLong(argArray[2]));
>>>>> job.setWorkerConfiguration(Integer.parseInt(argArray[3]),
>>>>> Integer.parseInt(argArray[3]),
>>>>> 100.0f);
>>>>>
>>>>> return job.run(true) ? 0 : -1;
>>>>> }
>>>>>
>>>>> public static void main(String[] args) throws Exception {
>>>>> System.exit(ToolRunner.run(new SimpleGiraphSumEdgeWeights(),
>>>>> args));
>>>>> }
>>>>> }
>>>>>
>>>>> On Fri, Feb 1, 2013 at 5:37 PM, Eli Reisman <apache.mail...@gmail.com>
>>>>> wrote:
>>>>> > Your best bet is to look over the two code components that users most
>>>>> > often
>>>>> > have to tweak or implement to write application code. That is, the
>>>>> > Vertex
>>>>> > implementations in examples/ and benchmark/ and the IO formats and
>>>>> > related
>>>>> > goodies like RecordReaders etc. that are mostly in the io/ dir. You
>>>>> > might
>>>>> > also take a look at the test suite for some quick ideas of how some
>>>>>of
>>>>> > the
>>>>> > moving parts fit together.
>>>>> >
>>>>> > If you have real work to do with Giraph, you're going to need to get
>>>>> > used to
>>>>> > 0.2 and its API. The old API is both limited in what kind of data it
>>>>> > will
>>>>> > process, and not compatible into the future. The API we have now,
>>>>>while
>>>>> > evolving, is much much closer to being "final" than anything in 0.1
>>>>>And
>>>>> > regardless, we now have (in hindsight) the sure knowledge that none
>>>>>of
>>>>> > the
>>>>> > code you write for 0.1 will be portable into the future.
>>>>> >
>>>>> > I am first in line to be sorry about the state of the docs. There are
>>>>> > efforts underway now to fix this. We all owe the users a collective
>>>>> > apology
>>>>> > for this. In lieu of proper apologies, feel free to ask any and all
>>>>> > questions, no matter how dumb, they can't be as dumb as mine! The
>>>>> > codebase
>>>>> > is under heavy development and has a lot of confusingly-named moving
>>>>> > parts
>>>>> > so first get used to the plumbing an app writer has to know to
>>>>>function,
>>>>> > get
>>>>> > some apps up and running, then dig into the framework code and it
>>>>>will
>>>>> > make
>>>>> > more sense.
>>>>> >
>>>>> > One string to pull on to begin to look inside the framework is
>>>>> > bin/giraph ->
>>>>> > org.apache.giraph.GiraphRunner (hands job to Hadoop) -> ... ->
>>>>> > o.a.g.graph.GraphMapper (is a mapper instance on a Hadoop cluster,
>>>>> > started
>>>>> > according to the Job sumbitted to Hadoop, but running our BSP code
>>>>> > instead)
>>>>> > -> o.a.g.graph.GraphTaskManager -> lots of places from there...
>>>>> >
>>>>> > The overarching BSP activity management for a single job run is
>>>>> > basically
>>>>> > all stemming out of GraphTaskManager now. You can look at setup() and
>>>>> > execute() and get a decent idea of the major events in a job run, and
>>>>> > where
>>>>> > to look to get a better peek under the hood at any given task or
>>>>>event.
>>>>> > Good
>>>>> > luck!
>>>>> >
>>>>> >
>>>>> > On Fri, Feb 1, 2013 at 4:59 PM, Gustavo Enrique Salazar Torres
>>>>> > <gsala...@ime.usp.br> wrote:
>>>>> >>
>>>>> >> Hi Ryan:
>>>>> >>
>>>>> >> It's the simplest thing:
>>>>> >> 1. Define your type of parameters for a type of Vertex (for example
>>>>> >> EdgeListVertex)
>>>>> >> 2. Implement compute method.
>>>>> >>
>>>>> >> From what I saw out there in the M/R world, Giraph provides the
>>>>> >> simplest
>>>>> >> way to work with graphs.
>>>>> >>
>>>>> >> Take a look at
>>>>> >>
>>>>> >>
>>>>>https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Exampl
>>>>>e
>>>>> >> and use release 0.1
>>>>> >> (http://www.apache.org/dyn/closer.cgi/incubator/giraph/)
>>>>> >> because 0.2-SNAPSHOT is under heavy work.
>>>>> >>
>>>>> >> Hope this helps you.
>>>>> >>
>>>>> >> Gustavo
>>>>> >>
>>>>> >> On Fri, Feb 1, 2013 at 9:17 PM, Ryan Compton
>>>>><compton.r...@gmail.com>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> I am having trouble understand what all the classes do and the
>>>>> >>> documentation looks like it might be out of date. I searched around
>>>>> >>> and found this: https://github.com/edaboussi/Giraph but it won't
>>>>> >>> compile with 0.2, any suggestions?
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >
>>
