Matei, I followed this document for hadoop setup, and the mesos's Running-Hadoop-on-Mesos for the mesos integration piece.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ Question on applying patch: 1. The hadoop in "mesos/frameworks/hadoop-0.20.2"; do I need to apply hadoop-mesos.patch first and then charles's patch? 2. If I want to run a different version of Hadoop (0.20.203), should I just edit the hadoop-mesos.patch and apply to the 0.20.203 hadoop? 3. Basically, do we have a documentation on editing hadoop-mesos.patch file to work with different version of hadoop? If we don't, I will document in my step-by-step guide. <https://github.com/mesos/mesos/wiki/Running-Mesos-On-Mac-OS-X-(Single-Node-Cluster)> On Wed, Jun 15, 2011 at 3:52 PM, Matei Zaharia <[email protected]> wrote: > By the way, please email [email protected] for stuff like this. > That's the main list we want to use for questions and development now. > > Matei > > On Jun 15, 2011, at 3:51 PM, Matei Zaharia wrote: > >> Hi Bill, >> >> This error means you haven't configured the mapred.job.tracker property for >> distributed operation. Check out >> https://github.com/mesos/mesos/wiki/Running-Hadoop-on-Mesos for how to run >> Hadoop on Mesos. (Note that this is part of the "set up Hadoop for >> distributed operation" piece we link to.) You'll need something like this: >> >> <configuration> >> <property> >> <name>mapred.job.tracker</name> >> <value>myhost:9001</value> >> </property> >> </configuration> >> >> Matei >> >> >> On Jun 15, 2011, at 3:25 PM, Bill Zhao wrote: >> >>> ---------- Forwarded message ---------- >>> From: Bill Zhao <[email protected]> >>> Date: Wed, Jun 15, 2011 at 3:20 PM >>> Subject: Re: hadoop on Mesos >>> To: Charles Reiss <[email protected]> >>> >>> >>> Charles, >>> >>> I have done the following: >>> >>> 1. build the mesos for github source >>> 2. copied "hadoop-for-new-iface.patch" to >>> mesos/frameworks/hadoop-0.20.2 directory >>> 3. applied the patch "git apply >>> mesos/frameworks/hadoop-0.20.2/hadoop-for-new-iface.patch" this works >>> >>> for some reason the "patch -p0 >>> mesos/frameworks/hadoop-0.20.2/hadoop-for-new-iface.patch" doesn't >>> work >>> >>> 4. build the jar with this command, and in mesos directory " ~/mesos$ ant" >>> 5. set the MESOS_HOME in .bashrc to ~/mesos/ >>> 6. copied the mesos.jar to the 'lib' directory of Hadoop directory >>> >>> However, I am getting this error message. Plus, I notice that I can >>> not copy the file from local file system to Hadoop’s HDFS. I do see >>> the file in HDFS (bin/hadoop dfs -ls), but it's size 0 byte. >>> >>> billz@dhcp-44-175: ~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop jobtracker >>> 11/06/15 15:12:18 INFO mapred.JobTracker: STARTUP_MSG: >>> /************************************************************ >>> STARTUP_MSG: Starting JobTracker >>> STARTUP_MSG: host = dhcp-44-175.eecs.berkeley.edu/128.32.44.175 >>> STARTUP_MSG: args = [] >>> STARTUP_MSG: version = 0.20.3-dev >>> STARTUP_MSG: build = -r ; compiled by 'billz' on Wed Jun 15 14:39:40 PDT >>> 2011 >>> ************************************************************/ >>> 11/06/15 15:12:18 INFO mapred.JobTracker: Scheduler configured with >>> (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, >>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) >>> 11/06/15 15:12:18 FATAL mapred.JobTracker: java.lang.RuntimeException: >>> Not a host:port pair: local >>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:136) >>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:123) >>> at >>> org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:1807) >>> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1579) >>> at >>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183) >>> at >>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175) >>> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702) >>> >>> 11/06/15 15:12:18 INFO mapred.JobTracker: SHUTDOWN_MSG: >>> /************************************************************ >>> SHUTDOWN_MSG: Shutting down JobTracker at >>> dhcp-44-175.eecs.berkeley.edu/128.32.44.175 >>> ************************************************************/ >>> >>> >>> Thanks, >>> >>> >>> >>> On Tue, Jun 14, 2011 at 6:21 PM, Charles Reiss >>> <[email protected]> wrote: >>>> On Tue, Jun 14, 2011 at 18:03, Andy Konwinski <[email protected]> >>>> wrote: >>>>> maybe charles, being our resident hadoop on mesos expert, has seen it >>>>> recently or can help. >>>>> if not, you should send the question in an email to >>>>> [email protected] >>>>> >>>>> On Tue, Jun 14, 2011 at 5:33 PM, Bill Zhao <[email protected]> wrote: >>>>>> >>>>>> Andy, >>>>>> >>>>>> Have you seen this error before? >>>> Looks like a missing/not-on-path mesos.jar. >>>> >>>> Make sure: >>>> (a) you've set MESOS_HOME >>>> (b) you've built mesos.jar >>>> If both of these things, then maybe try symlinking or copying >>>> mesos.jar to the 'lib' directory of Hadoop. >>>> >>>> -- >>>> I am following this doc: >>>>>> https://github.com/mesos/mesos/wiki/Running-Hadoop-on-Mesos >>>>>> >>>>>> I have tried this few times with different Hadoop's >>>>>> conf/mapred-site.xml configuration. However, I am not sure what the >>>>>> problem is. Do I need to run Charles Reiss's patch and rebuild >>>>>> hadoop? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> billz@dhcp-44-175: ~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop >>>>>> jobtracker >>>>>> 11/06/14 17:19:27 INFO mapred.JobTracker: STARTUP_MSG: >>>>>> /************************************************************ >>>>>> STARTUP_MSG: Starting JobTracker >>>>>> STARTUP_MSG: host = dhcp-44-175.eecs.berkeley.edu/128.32.44.175 >>>>>> STARTUP_MSG: args = [] >>>>>> STARTUP_MSG: version = 0.20.3-dev >>>>>> STARTUP_MSG: build = -r ; compiled by 'billz' on Sat Jun 4 11:49:29 >>>>>> PDT 2011 >>>>>> ************************************************************/ >>>>>> 11/06/14 17:19:27 INFO mapred.JobTracker: Scheduler configured with >>>>>> (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, >>>>>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) >>>>>> 11/06/14 17:19:27 FATAL mapred.JobTracker: >>>>>> java.lang.NoClassDefFoundError: org/apache/mesos/Scheduler >>>>>> at java.lang.Class.forName0(Native Method) >>>>>> at java.lang.Class.forName(Class.java:247) >>>>>> at >>>>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762) >>>>>> at >>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807) >>>>>> at >>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833) >>>>>> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1573) >>>>>> at >>>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183) >>>>>> at >>>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175) >>>>>> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702) >>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.mesos.Scheduler >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >>>>>> ... 9 more >>>>>> >>>>>> 11/06/14 17:19:27 INFO mapred.JobTracker: SHUTDOWN_MSG: >>>>>> /************************************************************ >>>>>> SHUTDOWN_MSG: Shutting down JobTracker at >>>>>> dhcp-44-175.eecs.berkeley.edu/128.32.44.175 >>>>>> ************************************************************/ >>>>>> >>>>>> -- >>>>>> Bill Zhao >>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> Bill Zhao >>> >>> >>> >>> -- >>> Bill Zhao >>> <mapred-site.xml>This email sent via mesos-users mailing list. >>> http://mesos.berkeley.edu. >> >> >> This email sent via mesos-users mailing list. http://mesos.berkeley.edu. > > -- Bill Zhao
