Thanks Charith, but my main question still remains, even with the examples that
comes with Giraph, such as simple shortest path computation example, the
System.out.println or the Log.Debug (or I also tried Log.Info) they all do not
print out customer messages that I write in the compute method.
For example, I modified the class SimpleShortestPathsComputation to use println
instead of Log.Debug, here is the compute method (I’ve highlighted the print
statments):
@Override
public void compute(
Vertex<LongWritable, DoubleWritable, FloatWritable> vertex,
Iterable<DoubleWritable> messages) throws IOException {
if (getSuperstep() == 0) {
vertex.setValue(new DoubleWritable(Double.MAX_VALUE));
}
double minDist = isSource(vertex) ? 0d : Double.MAX_VALUE;
for (DoubleWritable message : messages) {
minDist = Math.min(minDist, message.get());
}
System.out.println("Vertex " + vertex.getId() + " got minDist = " + minDist
+ " vertex value = " + vertex.getValue());
if (minDist < vertex.getValue().get()) {
vertex.setValue(new DoubleWritable(minDist));
for (Edge<LongWritable, FloatWritable> edge : vertex.getEdges())
{
double distance = minDist + edge.getValue().get();
System.out.println("Vertex " + vertex.getId() + " sent to " +
edge.getTargetVertexId() + " = " + distance);
sendMessage(edge.getTargetVertexId(), new DoubleWritable(distance));
}
}
vertex.voteToHalt();
}
Still the statements for println do not write these messages out, are they
supposed to be somewhere else or are they are not written?
I also tried with log.info, but again these statements were not written, I
prefer to use println.
Thanks,
Tamer
From: Charith Wickramarachchi [mailto:charith.dhanus...@gmail.com]
Sent: Thursday, October 02, 2014 4:53 PM
To: user
Subject: Re: How to Write to HDFS?
Hi Tamer,
The reason you see this behavior is IntIntNullTextInputFormat sets the value of
the vertex as same as the vertex id when creating a vertex. Since you do not
change the value vertex id will be written to the output as the vertex value.
See the class
org.apache.giraph.io.formats.IntIntNullTextInputFormat.IntIntNullVertexReader
and you will understand.
Hope this helps.
Thanks,
Charith
On Thu, Oct 2, 2014 at 1:35 PM, Tamer Yousef
<tyou...@boardreader.com<mailto:tyou...@boardreader.com>> wrote:
Hello All!
I’m learning Giraph and trying few things, but I fail to write out output to
hdfs. I created my own .java file, and I placed it in the folder
${GIRAPH_HOME}//giraph-examples/src/main/java/org/apache/giraph/examples/ then
I ran the mvn compile to get a jar file that includes my class. The function is
doing nothing other than trying to:
1- Write using the stdout
2- Write using log4j
The program runs and it creates an output directory in hdfs as I specify in the
command below, but the output file does not reflect what the program should
write out.
Here is the output I get in the output file in HDFS (the vertices I have are
very similar):
6 6
5 5
13 13
12 12
8 8
7 7
2 2
15 15
9 9
16 16
10 10
1 1
3 3
14 14
11 11
4 4
Even if I completely comment out the code in the compute class, I still get the
output above (with keeping the voteToHalt method).
I execute the code using the command:
hadoop jar
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner org.apache.giraph.examples.HelloWorld -vif
org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /in/graph2.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /out5 -w 1
I’m working with Hadoop 1.2.1 and the latest Giraph from the trunk.
and here is my full class:
package org.apache.giraph.examples;
import org.apache.giraph.GiraphRunner;
import org.apache.hadoop.util.ToolRunner;
import org.apache.giraph.graph.BasicComputation;
import org.apache.giraph.conf.LongConfOption;
import org.apache.giraph.edge.Edge;
import org.apache.giraph.graph.Vertex;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.FloatWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.log4j.Logger;
import java.io.IOException;
@Algorithm(
name = "Hellow",
description = "test class"
)
public class HelloWorld extends
BasicComputation<IntWritable, IntWritable,
NullWritable, NullWritable> {
@Override
public void compute(Vertex<IntWritable,
IntWritable, NullWritable>
vertex,
Iterable<NullWritable>
messages) {
System.out.println("Hello world from print
ln");
LOG.info("Hello world from log info");
vertex.voteToHalt();
}
public static void main(String[] args) throws Exception
{
//log4j.logger.org.apache.hadoop = DEBUG;
System.exit(ToolRunner.run(new
GiraphRunner(), args));
}
/** Class logger */
private static final Logger LOG =
Logger.getLogger(SimpleShortestPathsComputation.class);
}
Any ideas?
Thanks!
--
Charith Dhanushka Wickramaarachchi
Tel +1 213 447 4253
Web http://apache.org/~charith<http://www-scf.usc.edu/~cwickram/>
Blog http://charith.wickramaarachchi.org/<http://charithwiki.blogspot.com/>
Twitter @charithwiki<https://twitter.com/charithwiki>
This communication may contain privileged or other confidential information and
is intended exclusively for the addressee/s. If you are not the intended
recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating
that fact and delete the copy you received and in addition, you should not
print, copy, retransmit, disseminate, or otherwise use the information
contained in this communication. Internet communications cannot be guaranteed
to be timely, secure, error or virus-free. The sender does not accept liability
for any errors or omissions
