We're running a HA job tracker (not deployed on top of mesos itself, though) with the mesos-hadoop framework being referenced here. This guide from Cloudera (CDH5) is pretty good for getting started: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_hag_jt_ha_intro_config.html
This also explains how ZooKeeper can be used as a failover controller to enable automatic failover if one job tracker goes down. -- Tom ArnfeldDeveloper // DueDil t...@duedil.com (+44) 7525940046 25 Christopher Street, London, EC2A 2BS Company Number: 06999618 On Fri, Oct 31, 2014 at 2:07 AM, Ankur Chauhan <an...@malloc64.com> wrote: > Thanks this is great. I actually ended up using the vanilla Hadoop > distribution and it worked just fine. I will try out your tutorial. Side > question, is there a solution around making the jobtracker ha? > -- ankur > Sent from my iPhone >> On Oct 30, 2014, at 5:05 PM, Stratos Dimopoulos >> <stratos.dimopou...@gmail.com> wrote: >> >> Hi Ankur, >> >> I recently went through the process of installing Hadoop on Mesos over >> cdh5.1.2 >> >> I created a post here - shouldn't be much different for your version: >> http://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installation-guide/ >> >> You can also find an other post about configuring CDH5.1.2 specifically to >> use with Mesos: useful: >> http://strat0sphere.wordpress.com/2014/10/30/cloudera-hdfs-cdh5-installation-to-use-with-mesos/ >> >> Have in mind that when using Mesos you don't need to start the jobtracker. >> Mesos will do this for you. >> You also mentioned that you are trying to start hadoop as ubuntu user. This >> is not the right thing to do. Either add root to cloudera's root list or >> (recommended) use the root user (mapred? hdfs?) that your cloudera version >> considers as root - you have to check the documentation for this. >> >> Regarding the error you are seeing "Does not contain a valid host:port >> authority: local" - I've seen this error when my worker version was >> different than the jobtracker version (happened because I was using a >> hadoop-on-mesos tar file compiled with a different version than the one my >> cluster was using). To fix this you can do the obvious, which is making sure >> the installed version is the same with the one you ship to the executors >> through HDFS or you can hack this by adding the property >> hadoop.skip.worker.version.check to True - In the later case I wish you good >> luck... Neverhteless, I am not sure if this error can also appear in other >> cases. >> >> Hope this helps. >> >> Stratos >> >> >>> On Tue, Oct 28, 2014 at 12:30 PM, Ankur Chauhan <an...@malloc64.com> wrote: >>> Anyone else have something to add on this? >>> -- Ankur Chauhan >>> >>>> On 28 Oct 2014, at 02:10, Ankur Chauhan <an...@malloc64.com> wrote: >>>> >>>> Hi tom, >>>> >>>> I was basically following the readme. This gist has the list of commands >>>> how i am setting up things >>>> https://gist.github.com/ankurcha/a9504b0e423b1a40d756 so first of all if >>>> possible if you could help me verify if my process of setting up >>>> core-site, hdfs-site and mapred-site is correct. I was starting the node >>>> with >>>> >>>> $ /opt/hadoop/bin/hadoop jobtracker >>>> >>>> There are two errors that i was working through. It seems that hadoop >>>> doesn't like running as root (which is good) but despite starting the >>>> process as ubuntu i kept getting >>>> >>>> Does not contain a valid host:port authority: local >>>> >>>> -- Ankur >>>> >>>>> On 28 Oct 2014, at 01:57, Tom Arnfeld <t...@duedil.com> wrote: >>>>> >>>>> Hi Ankur, >>>>> >>>>> There aren't any getting started resources other than the documention >>>>> there as far as I know. Could you share your hadoop configuration and >>>>> perhaps a description of the problems you're having? >>>>> >>>>> Tom. >>>>> >>>>> >>>>> >>>>>> On Tue, Oct 28, 2014 at 8:53 AM, Ankur Chauhan <an...@malloc64.com> >>>>>> wrote: >>>>>> H, >>>>>> >>>>>> >>>>>> I was trying to setup mesos/hadoop with the latest CDH version (MR1) and >>>>>> it seems like the instructions are sort of out of date and I also tried >>>>>> the suggestions in https://github.com/mesos/hadoop/issues/25 but after 4 >>>>>> hours of flailing around I am still kind of stuck :-/ >>>>>> >>>>>> It seems like the configuration/installation instructions aren't >>>>>> complete and I am just too new to hadoop to figure out what's missing or >>>>>> going wrong. Does anyone know of a good resource I can use to get going? >>>>>> >>>>>> -- Ankur >>
