Hi Michael, That's a very good question. Currently a TwillRunnable cannot request the AM to get more containers. In general, YARN application only has two type of tasks, Application Master (AM) and Container Task. There is no notion of local task in YARN. However, a TwillRunnable is like any regular Java program that it can further spawn new threads or even processes that it manages locally. Will this help on your use case?
Thanks, Terence On Sat, Nov 7, 2015 at 8:45 AM, Michael Coon <[email protected]> wrote: > I just found this framework last night and have been looking through the code > to understand how it all works. I like its elegance and abstraction away from > the complexities of creating a Yarn app; however, with simplicity comes > fidelity loss. > > I'd like to know if it's possible for a runnable to spawn another runnable in > its local container. > > Here is the scenario: > > - TwillApp kicks off 3 runnables, each of type "Manager", which will get > allocated their own containers. > - The Manager is responsible for coordinating Worker tasks and reporting > local status up to the TwillApp; but it needs to coordinate those workers > local to itself, not across the cluster's network. (The Manager might take > advantage of data locality in some cases where he may have multiple local > workers, etc) > > As I see it, the APIs do not offer a direct way for a runnable to spawn > another runnable in its own container. Should I just make the runnable > another TwillApp and have the original/master TwillApp just kick off another > app as if he's a client? That seems cumbersome and it would be nice if there > were just a way a runnable could use a context to say "run these runnables in > my local container". > > Ideas? >
