That's right - there would be one AppMaster per query. Hive server, afaik, does have some kind of round robin logic - as well as the capability to launch new AMs if required. The hive user group will likely be able to answer this better.
The number of AMs (applications) that can run on a YARN cluster is controlled via queue configuration, and we typically rely on that to throttle concurrent execution requests. Without Hive server, the number of concurrent Hive queries would be limited by this factor. Additional queries would just queue up as pending applications on YARN. Hive Server likely has it's own logic to limit this. In terms of making the AppMaster multi-threaded. This has been discussed, but I don't believe there's any immediate plans for this. This would run into scheduling decisions like how many queries to allow through simultaneously, priorities, which queue to run in etc. Currently, we rely on YARN scheduling for all of this. On Mon, Nov 3, 2014 at 7:26 PM, VJ Anand <[email protected]> wrote: > I have a follow-up question -- Bikas mentioned that the Tez App Master > submits one DAG at a time -- Now, for a Query engine like Hive, where there > would be multiple requests, how is this handled? Are we creating multiple > App Masters that round robins between them? Even then, when large number of > requests are submitted to the Hive server, if the App master can submit > only one DAG at a time, we would have situations where there would be many > outstanding requests. Is there a way we can make the App Master > multi-threaded? > > -- > *VJ Anand* > >
