Hmm, I might do that, thanks. But I'm a little unclear, is that
actually a different version or do I just give a different profile? (ie.
-Dhadoop.version=0.20.203.0 or such) And will that work against 2.0.5
or do I need to install a different hadoop version as well?
I originally was following the quick start guide but ran in to a problem
with LocalJobRunner, I'm running a hadoop cluster with split master and
worker nodes, so trying to run the examples -w 1 errored about this
setup, but -w 4 errored that LocalJobRunner can't run multiple jobs or
some wording like that (it was about 8 hours ago, I've forgotten :)
So my next query, I couldn't find any answers about running the examples
on a split master-slave cluster (I hope that's the correct terminology,
I'm still getting the hang of hadoop, I'm trying to cluster a large set
of orthologous genes across about 2000 bacterial species).
Thanks.
On 06/10/13 19:16, Milinda Pathirage wrote:
I got the build working just by following above steps I mentioned.
May be best option is to try giraph version which uses map-reduce API
instead of pure YARN. Also there aren't any scripts which can use to
submit jobs to YARN cluster (for pure YARN based implementation). So
going with giraph version which is based on map-reduce API will be
easy for you.
Thanks
Milinda
On Sun, Oct 6, 2013 at 10:04 PM, Matthew Laird <lai...@sfu.ca> wrote:
Hmm, is there a recommendation on a particular version which might be good
for building/testing against Hadoop 2.0.5? I realize I'm playing with the
bleeding edge here...
I threw a try/catch around public static void printHaltInfo() just like
private static boolean waitForHaltInfo() has. However next I had issues
with the tests, I'm guessing because I'm running the hadoop head node on
this machine, it was a socket bind error, so I threw in a -DskipTests.
Next I ran in to a org.apache.rat issue, I added an exclude for *.txt,
*.xml, and *.py. Unfortunately next was:
[INFO] --- findbugs-maven-plugin:2.5.1:check (default) @ giraph-core ---
[INFO] BugInstance size is 2
[INFO] Error size is 0
[INFO] Total bugs: 2
[INFO]
org.apache.giraph.yarn.GiraphYarnTask$OverrideExceptionHandler.uncaughtException(Thread,
Throwable) invokes System.exit(...), which shuts down the entire virtual
machine ["org.apache.giraph.yarn.GiraphYarnTask$OverrideExceptionHandler"]
At GiraphYarnTask.java:[lines 171-178]
[INFO] exceptional return value of java.io.File.delete() ignored in
org.apache.giraph.yarn.YarnUtils.exportGiraphConfiguration(GiraphConfiguration,
ApplicationId) ["org.apache.giraph.yarn.YarnUtils"] At YarnUtils.java:[lines
47-240]
I guess this is what I get for playing with the trunk. :) But, what version
was it that is mentioned in the previous mailing list threads that you can
apply the patch to and get it compiled?
Thanks!
On 06/10/13 18:19, Matthew Laird wrote:
Thanks for the reply, so I just grabbed that I assume is the latest
version, the snapshot link from git:
https://git-wip-us.apache.org/repos/asf?p=giraph.git;a=tree;h=refs/heads/trunk;hb=trunk
And ran the command:
mvn -Dhadoop.version=2.0.5-alpha -Phadoop_yarn clean install
The error I receive in this situation is:
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Giraph Parent .............................. SUCCESS
[1.593s]
[INFO] Apache Giraph Core ................................ FAILURE
[23.522s]
[INFO] Apache Giraph Examples ............................ SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 25.574s
[INFO] Finished at: Sun Oct 06 18:17:21 PDT 2013
[INFO] Final Memory: 37M/692M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile)
on project giraph-core: Compilation failure: Compilation failure:
[ERROR]
/opt/giraph/giraph-core/target/munged/main/org/apache/giraph/job/HaltApplicationUtils.java:[78,49]
unreported exception java.lang.InterruptedException; must be caught or
declared to be thrown
[ERROR]
/opt/giraph/giraph-core/target/munged/main/org/apache/giraph/job/HaltApplicationUtils.java:[81,50]
unreported exception java.lang.InterruptedException; must be caught or
declared to be thrown
That looks like a bug in the code?
Thanks.
On 06/10/13 18:13, Milinda Pathirage wrote:
AFAIK, you need to specify hadoop.version system property. So command
to build giraph with YARN (2.0.5) should be something like following.
mvn -Phadoop_yarn -Dhadoop.version=2.0.5-alpha clean install
Hope this helps.
Thanks
Milinda
On Sun, Oct 6, 2013 at 9:04 PM, Matthew Laird <lai...@sfu.ca> wrote:
Further, I let it go and after two hours I see the file:
/giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphYarnTask.java.rej
is now 295GB in size. I'm not sure what it's stuck doing after applying
this patch...
Thanks.
On 06/10/13 16:13, Matthew Laird wrote:
Hello,
I'm trying to build giraph against Hadoop 2.0.5, and I've been browsing
the mailing list archives but am still having an issue. I found the
previous
thread where Eli points to
https://issues.apache.org/jira/browse/GIRAPH-688
I've grabbed the patch and applied it against a clone from git
(http://git-wip-us.apache.org/repos/asf/giraph.git, is that the correct
place to get the latest trunk?) Just like the other fellow asking a
few
months ago, patch complains the patch has been applied before.
Once applied I run the suggested build command:
mvn -U -Phadoop_yarn clean install
It reaches
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
And so far it's been sitting there about 20 minutes at this step, in
another window I can see it's working hard doing something, but I'm
assuming
it's stuck doing... I'm not sure what.
Without the patch I receive the same error the other fellow was having
about dependancies for hadoop not being satisfied.
Any advice on what might be going on, or is there a better place to get
a
snapshot that has the needed patches applied? (I see a lot of commits
since
the original thread)
Thanks!
