Hmm, it seems there have been some recent pushes that have broken trunk
as I discovered yesterday. :)
But I guess that brings me back to my original question, to build
against 2.0.5 using the MR interface rather than YARN as Milinda
suggested, what profile would I use? Would I let it think it's 0.20xxx
or tell it 2.0.5, or something else?
Thanks.
On 13-10-07 11:50 AM, Claudio Martella wrote:
trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
the site.
On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lai...@sfu.ca
<mailto:lai...@sfu.ca>> wrote:
Thanks, which is my next curiosity... if I head over to
http://giraph.apache.org/ I see at the top is says
"Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
However it's not clickable and the only download links I see are for
1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
snapshot since more pushes have occurred since August, and I don't
see a tag in git for 1.1.0
(https://git-wip-us.apache.__org/repos/asf?p=giraph.git
<https://git-wip-us.apache.org/repos/asf?p=giraph.git>). So where
does one get 1.1.0-SNAPSHOT from?
Thanks.
On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
Hi Matthew
I use giraph 1.1.0 and cdh4 hadoop 0.23
the patch of Marco:
http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>>
and build
mvn clean install -P hadoop_cdh4.1.2
Il giorno 07/ott/2013 20:20, "Matthew Laird" <lai...@sfu.ca
<mailto:lai...@sfu.ca>
<mailto:lai...@sfu.ca <mailto:lai...@sfu.ca>>> ha scritto:
Hello, so I'm watching this thread with interest as well, since
that's the exact example I'll be trying to run as well, and
I see
Silvio appears to be asking about running Giraph 1.0
against Hadoop
2.0.0.
I'm curious, is that a known working combination, 1.0 vs
2.0.x? And
if so, are there patches to apply first and what maven
command line
should I build with? (ie. will a -Dhadoop.version=2.0.5
give the
needed MR based API or will it use YARN?) I figure it's
better to
ask than aimlessly try all the combinations. :)
Thanks!
On 13-10-07 11:01 AM, Claudio Martella wrote:
try to increase the mappers' heap space through
mapred.child.java.opts.
On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
<silvio.digrego...@gmail.com <mailto:silvio.digrego...@gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>
<mailto:silvio.digregorio@
<mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>>> wrote:
I think now my issue is another, thanks to all.
java.lang.____IllegalStateException: run: Caught an
unrecoverable exception waitFor: ExecutionException
occurred
while waiting for
org.apache.giraph.utils.____ProgressableUtils$____FutureWaitable@73a01e8a
at
org.apache.giraph.graph.____GraphMapper.run(GraphMapper.____java:102)
......
Caused by: java.lang.____IllegalStateException:
waitFor:
ExecutionException occurred while waiting for
org.apache.giraph.utils.____ProgressableUtils$____FutureWaitable@73a01e8a
at
org.apache.giraph.utils.____ProgressableUtils.waitFor(____ProgressableUtils.java:151)
... 7 more
Caused by:
java.util.concurrent.____ExecutionException:
java.lang.OutOfMemoryError:*____Java heap space*
at
org.apache.giraph.utils.____ProgressableUtils.waitFor(____ProgressableUtils.java:143)
... 13 more
2013/10/7 Claudio Martella
<claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>>
OK, thanks.
I really have to push that patch in.
On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di
gregorio
<silvio.digrego...@gmail.com
<mailto:silvio.digrego...@gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>
<mailto:silvio.digregorio@
<mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>>> wrote:
yes i do,
i have seen this in your post in:
http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html
<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html
<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>>
excuse me if i had checked in the mail-achive
first I would
have avoided the last post.
Now zk issue are resolved.
2013/10/7 Claudio Martella
<claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>>
Can you try applying this one first?
<<< <
http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>><<<<
On Mon, Oct 7, 2013 at 8:40 AM, Silvio
Di gregorio
<silvio.digrego...@gmail.com
<mailto:silvio.digrego...@gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>
<mailto:silvio.digregorio@
<mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>>> wrote:
*As i said i have builded*
/giraph-examples-1.0.0-for-____hadoop-2.0.0-cdh4.1.2-jar-____with-dependencies.jar/
*for cdh4, successfully. The job
start to
monitoring the success rate:*
/13/10/07 08:28:45 INFO
mapred.JobClient:
map 0% reduce 0%/
*but then*
/Error running child
java.lang.____IllegalStateException: run:
Caught an unrecoverable exception
java.io.FileNotFoundException:
File
_bsp/_defaultZkManagerDir/job_____201309181636_0678/_zkServer
does not exist.
/
/.../
/Caused by:
java.lang.RuntimeException:
java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_____201309181636_0678/_zkServer does
not exist./
2013/10/5 Silvio Di gregorio
<silvio.digrego...@gmail.com
<mailto:silvio.digrego...@gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>
<mailto:silvio.digregorio@
<mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>>>
I ha ve build w/
hadoop_cdh4.1.2 parameter.
Something is changed, monday i
report the
result. Now the farm is closed.
Il giorno 05/ott/2013 14:06,
"Claudio
Martella"
<claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>> ha scritto:
Oh, right, -vof is in
trunk. Anyway
it looks
like you built giraph for
the wrong
profile.
You mentioned you're
running on
2.0, but
your giraph is built for
0.20.203 <tel:0.20.203>
<tel:0.20.203 <tel:0.20.203>>
<tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>. try
building with a profile
for your hadoop version.
On Fri, Oct 4, 2013 at
2:35 PM,
Silvio Di
gregorio
<silvio.digrego...@gmail.com
<mailto:silvio.digrego...@gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>
<mailto:silvio.digregorio@
<mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>>> wrote:
org.apache.commons.cli.____UnrecognizedOptionException:
Unrecognized option: -vof
in 1.0 version is
-of,--outputFormat <arg>
Vertex output format
-op,--outputPath <arg>
Vertex output path
2013/10/4 Claudio Martella
<claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>>
did you try the
argument
(-vof) i
suggested?
On Fri, Oct 4,
2013 at 2:13 PM,
Silvio Di gregorio
<silvio.digrego...@gmail.com
<mailto:silvio.digrego...@gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>
<mailto:silvio.digregorio@
<mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>>> wrote:
i've specified
-of
org.apache.giraph.io.formats.____IdWithValueTextOutputFormat
but the same
error was
produced
Exception in
thread "main"
java.lang.____IncompatibleClassChangeError:
Found interface
org.apache.hadoop.mapreduce.____JobContext,
but class was
expected
at
org.apache.giraph.bsp.____BspOutputFormat.____checkOutputSpecs(____BspOutputFormat.java:43)
at
org.apache.hadoop.mapred.____JobClient$2.run(JobClient.____java:984)
at
org.apache.hadoop.mapred.____JobClient$2.run(JobClient.____java:945)
at
java.security.____AccessController.doPrivileged(____Native
Method)
at
javax.security.auth.Subject.____doAs(Subject.java:396)
at
org.apache.hadoop.security.____UserGroupInformation.doAs(____UserGroupInformation.java:____1408)
at
org.apache.hadoop.mapred.____JobClient.submitJobInternal(____JobClient.java:945)
at
org.apache.hadoop.mapreduce.____Job.submit(Job.java:566)
at
org.apache.hadoop.mapreduce.____Job.waitForCompletion(Job.____java:596)
at
org.apache.giraph.job.____GiraphJob.run(GiraphJob.java:____237)
at
org.apache.giraph.____GiraphRunner.run(GiraphRunner.____java:94)
at
org.apache.hadoop.util.____ToolRunner.run(ToolRunner.____java:70)
at
org.apache.hadoop.util.____ToolRunner.run(ToolRunner.____java:84)
at
org.apache.giraph.____GiraphRunner.main(____GiraphRunner.java:124)
at
sun.reflect.____NativeMethodAccessorImpl.____invoke0(Native
Method)
at
sun.reflect.____NativeMethodAccessorImpl.____invoke(____NativeMethodAccessorImpl.java:____39)
at
sun.reflect.____DelegatingMethodAccessorImpl.____invoke(____DelegatingMethodAccessorImpl.____java:25)
at
java.lang.reflect.Method.____invoke(Method.java:597)
at
org.apache.hadoop.util.RunJar.____main(RunJar.java:208)
2013/10/4
Claudio Martella
<claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>>
Hi,
you need
to specify the
vertex
outputformat
class
(-vof
option), e.g.
org.apache.giraph.io.formats.____IdWithValueTextOutputFormat.
On Fri,
Oct 4, 2013
at 1:06
PM, Silvio
Di gregorio
<silvio.digrego...@gmail.com
<mailto:silvio.digrego...@gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>
<mailto:silvio.digregorio@
<mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
<mailto:silvio.digregorio@__gmail.com
<mailto:silvio.digrego...@gmail.com>>>>
wrote:
Hi,
I hope
I have
sent to
the
right address.
i have
a graph
(directed
and
unweighted)
stored
in
hdfs like a
adjacency
list
(140Milions of
edges
6Milions
of vertex)
node<tab>neighbors
23 2
1 3
4 3
1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
<tel:1%20999%2099923>
909
.......
......
hadoop
version
Hadoop
2.0.0-cdh4.3.0
- java 1.6
I have
executed the
giraph-1.0
connected
components
example, in
this
fashion
> hadoop jar
/usr/local/giraph/giraph-____examples/target/giraph-____examples-1.0.0-for-hadoop-0.20.203.0
<tel:0.20.203.0>
<tel:0.20.203.0 <tel:0.20.203.0>>
<tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
<tel:0.20.203.0>>>-jar-with-dependencies.____jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.____ConnectedComponentsVertex
-vif
org.apache.giraph.io.formats.____IntIntNullTextInputFormat
-vip
/user/hdfs/lista_adj_txt
-op
connectedgiraph
--workers 4
and
then fail with:
13/10/04
09:28:29 INFO
utils.ConfigurationUtils: No
edge
input format
specified.
Ensure your
InputFormat
does not
require one.
13/10/04
09:28:29 INFO
utils.ConfigurationUtils: No
output
format
specified.
Ensure
your
OutputFormat
does not
require one.
13/10/04
09:28:30 INFO
job.GiraphJob: run:
Since
checkpointing is
disabled
(default), do
not
allow any task
retries (setting
mapred.map.max.attempts
= 0,
old value = 4)
13/10/04
09:28:31 WARN
mapred.JobClient: Use
GenericOptionsParser for
parsing the
arguments.
Applications should
implement Tool
for the same.
13/10/04
09:28:31 INFO
mapred.JobClient:
Cleaning up the
staging
area
hdfs://srv-bigdata-dev-01.int
<http://srv-bigdata-dev-01.int>.____sose.it:8020/user/hdfs/.____staging/job_201309181636_0535
<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
<http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>
<http://srv-bigdata-dev-01.__i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535
<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
<http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>__>
Exception in thread
"main"
java.lang.____IncompatibleClassChangeError:
Found
interface
org.apache.hadoop.mapreduce.____JobContext,
but
class was
expected
at
org.apache.giraph.bsp.____BspOutputFormat.____checkOutputSpecs(____BspOutputFormat.java:43)
......
Thanks
in advance
--
Claudio Martella
claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>
--
Claudio Martella
claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>
--
Claudio Martella
claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>
--
Claudio Martella
claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>
--
Claudio Martella
claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>
--
Claudio Martella
claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>
<mailto:claudio.martella@
<mailto:claudio.martella@>__gma__il.com <http://gmail.com>
<mailto:claudio.martella@__gmail.com
<mailto:claudio.marte...@gmail.com>>>
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
--
Claudio Martella
claudio.marte...@gmail.com <mailto:claudio.marte...@gmail.com>
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
