So, IIUC, there are two parts to what you need (1) How do I schedule containers in the first place asking for locality? (2) How do I keep my scheduling requirements the same even if some of my dependent service move / migrate.
For (1), YARN supports data-locality - you can give preferences for hosts / racks etc. More advanced ‘locality’ features are a work-in-progress. As of now, we deem (2) to be too custom that we leave it up to the applications to orchestrate. Once we start putting more of the advanced locality features that I mentioned above, we can consider having the platform move containers automatically to keep those locality promises when services move/migrate. HTH +Vinod > On Mar 31, 2016, at 1:44 AM, Zoltán Zvara <zoltan.zv...@gmail.com> wrote: > > The intent to move a container is to improve service-to-service locality, > co-locate services to reduce bottlenecks introduced by network. The idea that > a certain parallel process should be moved comes from an external service. > > Speculation might not be effective here as I see, since the intent to move a > container can be triggered hourly, daily, weekly or monthly - containers > contain long running services, data pipelines. Or am I on the wrong line > here? Can I simulate something like this with the current speculation? > > Thanks for help! > > On Wed, Mar 30, 2016 at 11:17 PM Eric Payne <eric.payne1...@yahoo.com > <mailto:eric.payne1...@yahoo.com>> wrote: > I think it would help if I knew what the criteria is for wanting to move the > container. In other words, was the container started on an undesirable node > in the first place? Or, did the node become undesirable after the container > started. > > Speculation could be considered a "move" operation for containers. If a > container isn't finishing fast enough, the default speculator will start > another container on a different node. Would it be possible to create a > specialized speculator that understood your criteria for needing to move > containers? If so, it could be done automatically / programatically. > > Thanks, > -Eric > > > From: Zoltán Zvara <zoltan.zv...@gmail.com <mailto:zoltan.zv...@gmail.com>> > To: Vinod Kumar Vavilapalli <vino...@apache.org <mailto:vino...@apache.org>> > Cc: user@hadoop.apache.org <mailto:user@hadoop.apache.org> > Sent: Wednesday, March 30, 2016 9:10 AM > Subject: Re: YARN re-locate container > > How is this achieved? As far as I see it now, after stopping a container, the > AM must reallocate the same container with the same resource vector but with > locality preferences pointed to the new, target node. After the new leash has > been acquired, then the AM can take it to the new node and initiate a > `startContainers` message. > Our use-case with Ericsson would require a more simple API, where (for > example) a `moveContainer` call from the AM would ask the RM or NM to move a > container from one node to another (or to any of the specified set of > preferred nodes). Move would simply kill the container and restart it on > another node at any given time whenever it is possible - I feel questions > around scheduling: how container moves should be handled? Probably not like > simple allocations. > > Am I understanding the architecture correctly here? > > On Tue, Mar 29, 2016 at 7:31 PM Vinod Kumar Vavilapalli <vino...@apache.org > <mailto:vino...@apache.org>> wrote: > Containers can be restarted on other machines already today - YARN just > leaves it up to the applications to do so. > > Are you looking for anything more specifically? > > +Vinod > > > On Mar 29, 2016, at 9:45 AM, Zoltán Zvara <zoltan.zv...@gmail.com > > <mailto:zoltan.zv...@gmail.com>> wrote: > > > > Dear Hadoop Community, > > > > Is there any feature available, or on the road map to support the > > relocation of containers? (Simply restart the container on another machine.) > > > > Thanks, > > Zoltán > > >
