Hi Matthew, I am also new to giraph. I have only tried YARN version. If you want to follow exact steps as quick start you need to build hadoop_0.20.203 profile. Yarn profiles doesn't work with 0.20.203 due to differences in map reduce APIs. If your hadoop cluster is 2.0.5 you may need to try hadoop_2.0.3 profile. But I am not sure whether it will work with 2.0.5 cluster. But if there aren't any API changes in 2.0.5 which effects giraph implementation, there is a possibility of 2.0.3 version working with 2.0.5.
Others(those who have more experience with giraph) may have better alternative or solution for your scenario. Thanks Milinda On Sun, Oct 6, 2013 at 10:53 PM, Matthew Laird <lai...@sfu.ca> wrote: > Hmm, I might do that, thanks. But I'm a little unclear, is that actually a > different version or do I just give a different profile? (ie. > -Dhadoop.version=0.20.203.0 or such) And will that work against 2.0.5 or do > I need to install a different hadoop version as well? > > I originally was following the quick start guide but ran in to a problem > with LocalJobRunner, I'm running a hadoop cluster with split master and > worker nodes, so trying to run the examples -w 1 errored about this setup, > but -w 4 errored that LocalJobRunner can't run multiple jobs or some wording > like that (it was about 8 hours ago, I've forgotten :) > > So my next query, I couldn't find any answers about running the examples on > a split master-slave cluster (I hope that's the correct terminology, I'm > still getting the hang of hadoop, I'm trying to cluster a large set of > orthologous genes across about 2000 bacterial species). > > Thanks. > > > On 06/10/13 19:16, Milinda Pathirage wrote: >> >> I got the build working just by following above steps I mentioned. >> >> May be best option is to try giraph version which uses map-reduce API >> instead of pure YARN. Also there aren't any scripts which can use to >> submit jobs to YARN cluster (for pure YARN based implementation). So >> going with giraph version which is based on map-reduce API will be >> easy for you. >> >> Thanks >> Milinda >> >> On Sun, Oct 6, 2013 at 10:04 PM, Matthew Laird <lai...@sfu.ca> wrote: >>> >>> Hmm, is there a recommendation on a particular version which might be >>> good >>> for building/testing against Hadoop 2.0.5? I realize I'm playing with >>> the >>> bleeding edge here... >>> >>> I threw a try/catch around public static void printHaltInfo() just like >>> private static boolean waitForHaltInfo() has. However next I had issues >>> with the tests, I'm guessing because I'm running the hadoop head node on >>> this machine, it was a socket bind error, so I threw in a -DskipTests. >>> >>> Next I ran in to a org.apache.rat issue, I added an exclude for *.txt, >>> *.xml, and *.py. Unfortunately next was: >>> >>> [INFO] --- findbugs-maven-plugin:2.5.1:check (default) @ giraph-core --- >>> [INFO] BugInstance size is 2 >>> [INFO] Error size is 0 >>> [INFO] Total bugs: 2 >>> [INFO] >>> >>> org.apache.giraph.yarn.GiraphYarnTask$OverrideExceptionHandler.uncaughtException(Thread, >>> Throwable) invokes System.exit(...), which shuts down the entire virtual >>> machine >>> ["org.apache.giraph.yarn.GiraphYarnTask$OverrideExceptionHandler"] >>> At GiraphYarnTask.java:[lines 171-178] >>> [INFO] exceptional return value of java.io.File.delete() ignored in >>> >>> org.apache.giraph.yarn.YarnUtils.exportGiraphConfiguration(GiraphConfiguration, >>> ApplicationId) ["org.apache.giraph.yarn.YarnUtils"] At >>> YarnUtils.java:[lines >>> 47-240] >>> >>> I guess this is what I get for playing with the trunk. :) But, what >>> version >>> was it that is mentioned in the previous mailing list threads that you >>> can >>> apply the patch to and get it compiled? >>> >>> Thanks! >>> >>> >>> On 06/10/13 18:19, Matthew Laird wrote: >>>> >>>> Thanks for the reply, so I just grabbed that I assume is the latest >>>> version, the snapshot link from git: >>>> >>>> >>>> >>>> https://git-wip-us.apache.org/repos/asf?p=giraph.git;a=tree;h=refs/heads/trunk;hb=trunk >>>> >>>> And ran the command: >>>> >>>> mvn -Dhadoop.version=2.0.5-alpha -Phadoop_yarn clean install >>>> >>>> The error I receive in this situation is: >>>> >>>> [INFO] 2 errors >>>> [INFO] ------------------------------------------------------------- >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] Reactor Summary: >>>> [INFO] >>>> [INFO] Apache Giraph Parent .............................. SUCCESS >>>> [1.593s] >>>> [INFO] Apache Giraph Core ................................ FAILURE >>>> [23.522s] >>>> [INFO] Apache Giraph Examples ............................ SKIPPED >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] BUILD FAILURE >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] Total time: 25.574s >>>> [INFO] Finished at: Sun Oct 06 18:17:21 PDT 2013 >>>> [INFO] Final Memory: 37M/692M >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [ERROR] Failed to execute goal >>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile >>>> (default-compile) >>>> on project giraph-core: Compilation failure: Compilation failure: >>>> [ERROR] >>>> >>>> /opt/giraph/giraph-core/target/munged/main/org/apache/giraph/job/HaltApplicationUtils.java:[78,49] >>>> unreported exception java.lang.InterruptedException; must be caught or >>>> declared to be thrown >>>> [ERROR] >>>> >>>> /opt/giraph/giraph-core/target/munged/main/org/apache/giraph/job/HaltApplicationUtils.java:[81,50] >>>> unreported exception java.lang.InterruptedException; must be caught or >>>> declared to be thrown >>>> >>>> That looks like a bug in the code? >>>> >>>> Thanks. >>>> >>>> On 06/10/13 18:13, Milinda Pathirage wrote: >>>>> >>>>> AFAIK, you need to specify hadoop.version system property. So command >>>>> to build giraph with YARN (2.0.5) should be something like following. >>>>> >>>>> mvn -Phadoop_yarn -Dhadoop.version=2.0.5-alpha clean install >>>>> >>>>> Hope this helps. >>>>> >>>>> Thanks >>>>> Milinda >>>>> >>>>> >>>>> On Sun, Oct 6, 2013 at 9:04 PM, Matthew Laird <lai...@sfu.ca> wrote: >>>>>> >>>>>> Further, I let it go and after two hours I see the file: >>>>>> >>>>>> >>>>>> >>>>>> /giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphYarnTask.java.rej >>>>>> >>>>>> is now 295GB in size. I'm not sure what it's stuck doing after >>>>>> applying >>>>>> this patch... >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> On 06/10/13 16:13, Matthew Laird wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I'm trying to build giraph against Hadoop 2.0.5, and I've been >>>>>>> browsing >>>>>>> the mailing list archives but am still having an issue. I found the >>>>>>> previous >>>>>>> thread where Eli points to >>>>>>> >>>>>>> https://issues.apache.org/jira/browse/GIRAPH-688 >>>>>>> >>>>>>> I've grabbed the patch and applied it against a clone from git >>>>>>> (http://git-wip-us.apache.org/repos/asf/giraph.git, is that the >>>>>>> correct >>>>>>> place to get the latest trunk?) Just like the other fellow asking a >>>>>>> few >>>>>>> months ago, patch complains the patch has been applied before. >>>>>>> >>>>>>> Once applied I run the suggested build command: >>>>>>> >>>>>>> mvn -U -Phadoop_yarn clean install >>>>>>> >>>>>>> It reaches >>>>>>> >>>>>>> [INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core --- >>>>>>> >>>>>>> And so far it's been sitting there about 20 minutes at this step, in >>>>>>> another window I can see it's working hard doing something, but I'm >>>>>>> assuming >>>>>>> it's stuck doing... I'm not sure what. >>>>>>> >>>>>>> Without the patch I receive the same error the other fellow was >>>>>>> having >>>>>>> about dependancies for hadoop not being satisfied. >>>>>>> >>>>>>> Any advice on what might be going on, or is there a better place to >>>>>>> get >>>>>>> a >>>>>>> snapshot that has the needed patches applied? (I see a lot of commits >>>>>>> since >>>>>>> the original thread) >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>> >> >> > -- Milinda Pathirage twitter: milindalakmal skype: milinda.pathirage blog: http://milinda.pathirage.org
