I will dig a little further as the behavior is inconsistent. On subsequent attempts I have seen the task-trackers go away with the job. They always go away when I shutdown the corresponding job-tracker.
The hadoop code I am using was included in the 0.13 tarball that I downloaded from here: http://mirror.nexcess.net/apache/mesos/0.13.0/ I built the jar by running hadoop/TUTORIAL.sh. I wound up integrating with hadoop manually since the tutorial script didn't work correctly for me. I mostly followed the instructions here: https://github.com/mesos/hadoop At one point I tried building it from https://github.com/mesos/hadoop but I had trouble getting it to build with 0.13. Should I be working off of a different version? Thanks, Paul On Thu, Sep 26, 2013 at 10:34 PM, Dan Colish <dcol...@urbanairship.com>wrote: > > > > On Thu, Sep 26, 2013 at 6:27 PM, Paul Mackles <pa...@loopr.com> wrote: > >> Hi - I am using mesos 0.13 with cdh4.2.0 in pseudo-distributed mode. >> While I am able to launch and run hadoop jobs through mesos successfully, I >> noticed in the Mesos UI (and through 'ps') that the task-trackers launched >> by mesos are sticking around long after my job is complete. Is that >> expected behavior? I am thinking the answer is no since they are tying up >> resources that could be used by other frameworks. On the other hand, mesos >> seems to know enough to reuse them when running subsequent hadoop jobs. >> Maybe there are using reservations or something by default? >> >> > Are you using the mesos-hadoop project found here, > https://github.com/mesos/hadoop? If so, you are correct that idle > tasktrackers should be torn down at the end. I wonder what the cluster > state when the JobInProgressListener is called with upon your job's > completion. Specifically, I would look into tracing this section [1] of code > * *where the task trackers job queue is checked for emptiness the tracker > is checked for being active. If the tracker was never activated I think it > would also be running but not killed. > > > [1] > https://github.com/mesos/hadoop/blob/master/src/main/java/org/apache/hadoop/mapred/MesosScheduler.java#L105 > > -- Thanks, Paul
