Which scheduler is being used? Capacity/Fair/Something else?
From: Nicolae Marasoiu [mailto:nicolae.maras...@adswizz.com] Sent: Monday, November 23, 2015 7:59 AM To: user@hadoop.apache.org Subject: yarn does not allocate enough tasks/containers to my available node Hi, Tasks are allocated to my nodes by memory. Initially they are allocated ok across the cluster. After a while, one of the nodes does not receive new tasks fast enough: it gets to 0 tasks and from time to time I see it having 1 task which it finished in seconds. It is true that I currently have a problem of many small input files. And probably the fact that the nodes are oversubscribed in cpu by a factor of 2-3 (according to load average) is not helping. But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node at once (not one by one), and 2. why yarn is slow in allocating tasks? (I understand that allocating a new task/container in a few seconds may/may not be considered slow). Pls advise, Nicu